Jeffrey Gong

Introduction Laying out data is one of the oldest problems in computing. For over 15 years, since the advent of Hadoop and Hive, partitioning has been the standard way to physically organize data for processing and analysis. However, today’s Lakehouses serve agents, real-time pipelines, and query patterns that shift faster than any human can re-partition for.

Introduction The most performant, cost-effective lakehouse is one that optimizes itself as data volumes, query patterns, and organizational usage continue to evolve. Predictive Optimization (PO) in Unity Catalog enables this behavior by continuously analyzing how data is written and queried, then applying the appropriate maintenance actions automatically without requiring manual work from users or platform teams.