Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics

In one breath — the whole paper, compressed

By 2021 nearly every large enterprise ran a two-tier stack: raw data ETLed into a cloud data lake of Parquet or ORC files, then a curated subset ELTed again into a proprietary warehouse such as Redshift or Snowflake. The authors argue that this second copy is accidental complexity that buys reliability problems, days-old data, doubled storage cost, vendor lock-in, and an architecture that machine learning cannot read efficiently. Their alternative, the Lakehouse, keeps one copy of the data in open direct-access files on cheap object storage and adds the warehouse features above it: a transactional metadata layer such as Delta Lake that defines which objects form a given table version, plus caching, auxiliary statistics and indexes, and data layout optimization that recover SQL performance without altering the file format. ML and data science reach the same tables through declarative DataFrame APIs whose lazily built query plans are pushed into those same optimizations. On TPC-DS at scale factor 30,000 the Databricks Delta Engine ran all 99 queries in 3302 seconds for 104 dollars, against four cloud warehouses that took 2996 to 37283 seconds and cost 153 to 570 dollars.

Before this paper — the world it landed in

First-generation analytics platforms ETLed operational data straight into a schema-on-write warehouse, which worked until the appliance model, coupling compute and storage on premises, forced enterprises to buy for peak load, and until the data turned unstructured: video, audio and text that a SQL warehouse could not store or query at all. The second generation answered with data lakes, starting with Hadoop and HDFS, dumping all raw data cheaply into generic open formats such as Parquet and ORC under schema-on-read, and punting data quality and governance downstream. From 2015 cloud object stores such as S3, ADLS and GCS displaced HDFS with better than ten nines of durability, geo-replication and archival tiers, but the shape stayed the same: a small curated subset was ETLed onward into Teradata, Redshift or Snowflake for the BI that mattered. The authors report this two-tier lake plus warehouse pattern as dominant at virtually all Fortune 500 enterprises. Both halves of the industry were already leaning toward each other, with every major warehouse adding external tables over Parquet and ORC, and Spark SQL, Presto, Hive and Athena querying the lake directly, but external-table connectors ran slowly because the engines were tuned for their internal formats, and the lake engines still lacked ACID transactions and indexes.

The problem — what was actually breaking

Keeping a data lake and a downstream warehouse consistent requires continuous ETL engineering, and each additional step risks failures or bugs that quietly degrade data quality.
The two systems differ in supported data types, SQL dialects and even table schemas, so semantic mismatches between the lake and the warehouse become a permanent source of incorrect data.
Warehouse data is stale because incoming data sits in a separate staging area and is loaded by periodic jobs, frequently taking days, whereas first-generation platforms exposed freshly loaded operational data immediately.
Leading machine learning systems such as TensorFlow, PyTorch and XGBoost need to stream large datasets through complex non-SQL code, which is inefficient over ODBC or JDBC and impossible against a warehouse's proprietary internal format.
Warehouse vendors answer ML workloads by recommending an export to files, which adds a third ETL step and yet more staleness, while running ML on the raw lake instead forfeits ACID transactions, versioning and indexing.
Total cost of ownership is inflated by paying for continuous ETL and for a second copy of the storage, and by proprietary formats that make migrating data or workloads to another engine expensive.

Core ideas — the contributions, and why they work

The Lakehouse definition

A Lakehouse is defined as a data management system built on low-cost, directly accessible storage that nonetheless provides the traditional analytical DBMS management and performance features: ACID transactions, data versioning, auditing, indexing, caching and query optimization. It is the union of the two halves of the two-tier stack, cheap open storage readable by any engine from the lake, and management and optimization machinery from the warehouse. The paper is explicit about the price of that union: allowing direct access means giving up part of the data independence that has been a cornerstone of relational DBMS design, because the on-disk format is now a public API. The bet is that the features that made warehouses worth a second copy of the data can be reimplemented above an open format rather than inside a closed one.

A transactional metadata layer over open files

The first enabling idea is to keep bulk data in an object store in a standard format such as Parquet, and put a transactional metadata layer above it that defines which objects belong to which table version. Because table membership is now a transactionally updated set rather than whatever files happen to sit in a directory, atomicity, versioning, time travel and zero-copy cloning can all be implemented in the metadata layer while clients still read plain Parquet directly. This is the design already shipped by Delta Lake, Apache Iceberg and Apache Hudi. Crucially the abstraction is additive: an existing directory of Parquet files becomes a managed table simply by adding a log whose first entry references the files that are already there.

Format-independent SQL performance

Warehouses get their speed by co-designing storage format and engine, which a Lakehouse cannot do because the format is fixed and public. The paper's answer is that three classes of optimization leave the data files untouched and can therefore be applied to any format, present or future: caching hot files on SSD and RAM in a transcoded form, maintaining auxiliary data such as per-file statistics and Bloom filter indexes, and optimizing data layout by clustering records. The argument for why this suffices is a hot and cold split: hot data can be cached in exactly the structures a closed-world warehouse would use, while for cold data in the object store, performance is dominated by bytes read, which layout plus zone maps minimize just as a proprietary engine would.

Declarative DataFrame APIs as the ML interface

ML libraries are written in imperative code that cannot be expressed as SQL, yet they need bulk data access, so the naive integration is just to ask the metadata layer which Parquet files form the table and hand them to the library. The paper argues for a step further: make the DataFrame API declarative by evaluating transformations lazily, so the client library captures a relational operator plan instead of executing eagerly. That plan can then be optimized and pushed into the storage layer, letting ML data preparation inherit the same caching, data skipping and layout benefits as SQL. Because DataFrames are the common currency of the Spark analytics ecosystem, one optimization path accelerates MLlib, GraphFrames, SparkR and community libraries at once.

Data quality and governance in the metadata layer

Once a layer exists that mediates access to a table, it is the natural place to enforce correctness and policy rather than trusting every writer. Delta Lake implements schema enforcement so uploaded data must match the table schema, and a constraints API letting owners restrict ingested values, with the client libraries automatically rejecting or quarantining violating records. The same layer can act as a governance chokepoint: it checks whether a client may access a table before handing out credentials to read the raw objects from the cloud object store, and it can reliably log every access. This directly attacks the data swamp failure mode of lakes, where quality and governance were punted downstream and never solved.

Open formats as a governance and lock-in argument

Beyond performance and cost, the paper makes a regulatory and organizational case for direct-access open formats. Rising data management requirements mean organizations may need to search old datasets, delete specific data, or change processing infrastructure on short notice, and standardizing on an open format guarantees they can do so without waiting on a vendor. The same property makes Lakehouses fit distributed team structures such as the data mesh, since every dataset is reachable from the object store without onboarding consumers onto the producer's compute. The authors read the long-term direction of the software industry as being toward open data formats and expect enterprise data to follow.

Why the alternatives were rejected

Two rival routes to the same goals are considered and dismissed. Eliminating the lake and storing everything in a warehouse with separated compute and storage is called a straw man whose lack of adoption is itself evidence: it still handles video, audio and text poorly and still denies ML fast direct access. Building a massively parallel serving layer in front of the warehouse, in the manner of Hive LLAP, is judged more expensive to run, harder to manage and probably slower than direct object store access, and it degrades exactly the three properties that make object stores attractive, namely low cost, high bandwidth to elastic workloads and extremely high availability, while merely relocating the problem of choosing an efficient read format.

How it works — the mechanism, concretely

Table state as a transaction log in the object store

Delta Lake, developed at Databricks from 2016, records which objects are part of a table as a transaction log stored in Parquet format inside the same data lake, which lets it scale to billions of objects per table without a separate service. Apache Iceberg, started at Netflix, uses a similar design and supports both Parquet and ORC; Apache Hudi, started at Uber, targets streaming ingest but does not support concurrent writers. The lineage starts with Apache Hive ACID, which tracked the files of a table version in an OLTP DBMS and made updates to that set transactional. Adoption is cheap because conversion is metadata-only: a directory of Parquet files becomes a Delta table with zero data copies by writing a log whose first entry references every existing file.

Caching made safe by transactions

With a transactional metadata layer present, a Lakehouse can cache object store files on faster local devices, SSDs and RAM on the processing nodes, because a running transaction can determine from the log whether a cached file is still valid for its snapshot. This removes the usual invalidation hazard of caching over a mutable file directory. The cache need not hold bytes as written either: it can store a transcoded representation better suited to the execution engine, matching what a closed-world warehouse would keep internally. The Databricks implementation partially decompresses the Parquet data it loads into cache, trading space for decode work at query time.

Auxiliary data structures beside the base files

The base table format must remain readable for direct I/O, but the system retains full control of extra files it maintains itself. In Delta Lake and Delta Engine, per-file column min and max statistics are kept inside the same Parquet files that store the transaction log, so a query can consult them and skip entire data files whose ranges cannot match the predicate. This data skipping pays off in proportion to how well the base data is clustered on the filtered columns, which is why layout optimization is its partner. A Bloom filter based index was under implementation, and the authors note that a wide range of further structures, in the spirit of research on indexing raw data files, could live in the same place.

Data layout optimization within a fixed format

Even with the format frozen, the system chooses how records are distributed across files. The most important lever is record ordering, deciding which records cluster together and therefore become cheap to read together. Delta Lake supports ordering by individual dimensions and by space-filling curves, Z-order and Hilbert curves, to obtain locality across several dimensions at once rather than only the leading sort key. The paper sketches further layout freedoms that a future format could expose, such as placing columns in different orders within each file or choosing different compression strategies for different groups of records.

How the three optimizations compose

The techniques are designed for the skewed access pattern of analytics, where most queries hit a hot subset of the data. That hot subset is served from cache in warehouse-grade structures, so its performance need not depend on the open format at all. Cold data still lives in the object store, where the dominant cost is simply the volume read per query. There, clustering co-accessed records via layout, and using zone maps to decide which byte ranges of which files to touch, lets the engine minimize I/O in the same way a proprietary closed-world warehouse does, despite reading a standard open file.

The declarative DataFrame execution path

In the illustrated flow, a user writes ordinary DataFrame code, loading a users table, filtering to buyers, projecting a few columns and filling nulls with zero, before handing the result to a model fit call. None of it executes eagerly: Spark's lazy evaluation captures the whole data-loading computation as a query plan, which is handed to the Delta Lake client library. The planner pushes selections and projections down into the data source plugin class for each read, so the Delta Lake data source can apply caching, data skipping and layout-aware reads rather than scanning everything and filtering afterwards. The metadata layer is queried at the same time to decide which partitions and files are in the current table version.

Access control and ML lifecycle integration

Because clients ultimately read raw objects from the cloud store, the metadata layer acts as the point where policy is applied: it validates a client's right to a table before issuing the credentials that allow reading its files, and records the access in an audit log. On the ingest side, schema enforcement and the constraints API reject or quarantine records that violate the declared schema or value constraints. For data science, Delta Lake is integrated with MLflow's experiment tracking so a scientist can record which table version an experiment consumed and reconstruct that exact data later, which is the reproducibility feature that ad hoc file-based pipelines cannot offer.

What the paper showed — measurements and proofs

On the TPC-DS power test at scale factor 30,000, with comparable clusters of 960 vCPUs and local SSD storage on AWS, Azure and Google Cloud, running all 99 queries took 2996 s, 7143 s, 5793 s and 37283 s on the four anonymized cloud warehouses, versus 3302 s for Delta Engine on on-demand instances and 3252 s on spot instances.
Cost for the same power test under each service's own pricing model was 153, 286, 206 and 570 dollars for the four warehouses, against 104 dollars for Delta Engine on on-demand instances and 56 dollars on spot instances.
All systems were started with data cached on SSD because some of the compared warehouses only supported node-attached storage, but Delta Engine was only 18 percent slower when starting from a cold cache, which bounds how much of its result depended on warm local storage.
The metadata-layer approach is reported as generally providing similar or better performance than raw Parquet or ORC data lakes, while adding transactions, zero-copy cloning and time travel to past table versions.
Delta Lake was in use by thousands of customers and accounted for about half of Databricks' workload, growing to cover half the compute-hours on the platform within three years of its 2016 start, which the authors offer as evidence that the metadata-layer design is practical at scale.
The staleness claim is quantified from a Dimensional Research and Fivetran survey in which 86 percent of analysts reported using out-of-date data and 62 percent reported waiting on engineering resources numerous times per month.

Limits and trade-offs — conceded and discovered

Conceded by the paper: a Lakehouse gives up a significant portion of the data independence of a traditional DBMS, because the storage format becomes part of the public API, so the engine can never co-optimize format and execution the way a closed-world warehouse can.
Conceded: storing the transaction log in the same object store simplifies operations and gives high read bandwidth, but object store latency caps the transaction rate, and the authors suggest a faster metadata store may be preferable in some deployments.
Conceded: Delta Lake, Iceberg and Hudi all support transactions on only one table at a time, cross-table transactions remain future work, and Hudi additionally does not support concurrent writers.
Noted by the paper and worth stressing: the benchmark is not a clean sweep, since one warehouse finished the power test in 2996 s against Delta Engine's 3302 s, so the claim is comparable performance at lower cost rather than uniformly faster, and the authors themselves say there is plenty of room for further optimization.
Exposed by later scrutiny: the evidence is single-vendor, with competitors anonymized as DW1 to DW4 and the runs performed by the authors' own team, and the ML story assumes a Spark-shaped world, since APIs such as TensorFlow's tf.data push no query semantics into storage and instead care about overlapping loading with CPU-to-GPU transfer, a problem the paper flags as unaddressed.

What it became — the systems that inherited it

The paper named and legitimized a category: within a few years lakehouse became the standard label for analytics platforms built on open table formats over object storage. Its three named metadata layers all became load-bearing infrastructure, with Delta Lake and Apache Iceberg converging into an interchange standard supported by Snowflake, BigQuery, Trino, Athena, DuckDB and ClickHouse, and with Iceberg's catalog becoming a contested control point after Databricks acquired Tabular and Snowflake opened Polaris. The Delta Engine described here was published in detail as Photon, a vectorized C++ execution engine for lakehouse systems, at SIGMOD 2022, and Databricks Unity Catalog, Microsoft Fabric OneLake and AWS S3 Tables all shipped the paper's governance-at-the-metadata-layer argument as a product. The specific mechanisms travelled too: file-level min and max statistics for data skipping, Z-ordering and later liquid clustering for layout, and time travel by table version are now expected features of every table format. The declarative DataFrame argument was vindicated by the Pandas API on Spark, which absorbed Koalas, and by Polars and Ibis pushing lazy plans into storage. The strongest counter-evidence to the paper's own prediction is that the warehouses did not wither so much as converge, with Snowflake, BigQuery and Redshift all adding first-class Iceberg tables, so the industry arrived at the paper's open-format single-copy end state from both directions at once.

In the paper’s words — verbatim

“We define a Lakehouse as a data management system based on low-cost and directly-accessible storage that also provides traditional analytical DBMS management and performance features such as ACID transactions, data versioning, auditing, indexing, caching, and query optimization.”

§3 The Lakehouse Architecture

“Regardless of the exact design, however, the core challenge is that the data storage format becomes part of the system's public API to allow fast direct access, unlike in a traditional DBMS.”

§3.3 SQL Performance in a Lakehouse

“According to a survey by Dimensional Research and Fivetran, 86% of analysts use out-of-date data and 62% report waiting on engineering resources numerous times per month [47].”

§1 Introduction

Vocabulary — as this paper uses it

Lakehouse: A data management system based on low-cost, directly accessible storage that also provides traditional analytical DBMS management and performance features such as ACID transactions, data versioning, auditing, indexing, caching and query optimization.
Two-tier architecture: The dominant pattern the paper attacks, in which raw data is ETLed into a data lake and a curated subset is then ELTed again into a separate warehouse, producing two copies, extra pipelines, and the reliability and staleness problems that follow.
Metadata layer: A transactional layer above an object store that records which objects belong to which version of a table, thereby implementing ACID transactions, versioning and time travel without changing the underlying file format. Delta Lake, Apache Iceberg and Apache Hudi are the examples given.
Auxiliary data: Extra files the Lakehouse fully controls and maintains beside the immutable base data, such as per-file column min and max statistics or a Bloom filter index, used to speed up queries without touching the open storage format.
Data skipping: Using per-file statistics, also called zone maps, to determine that a data file cannot contain any row matching a predicate and to avoid reading it entirely. Its effectiveness depends on how well the base data is clustered on the filtered columns.
Z-ordering: A data layout technique that orders records along a space-filling curve, Z-order or Hilbert, so that records close in several dimensions at once land in the same files, giving multi-dimensional locality that a single sort key cannot.
Declarative DataFrame API: A DataFrame interface whose transformation operators are evaluated lazily, so the client library captures a relational operator plan and hands it to an optimizer instead of executing step by step, letting ML data preparation benefit from Lakehouse caching, statistics and layout.
Zero-copy cloning and time travel: Management features the metadata layer makes possible: creating a new logical table that references existing files without duplicating data, and querying a past version of a table by reading the object set recorded in an earlier log entry.
Data independence: The relational principle that clients see a logical schema while the system controls physical representation. A Lakehouse deliberately surrenders part of it, since the storage format becomes a public API that external readers depend on.

On the timeline — where this sits in the story

Year2016–2020, mainstream in 2020s

MilestoneApache Hudi, Apache Iceberg, Delta Lake, and lakehouse architecture

Problem solvedData lakes were often just files on object storage: weak transactions, poor schema evolution, hard concurrency, and fragile metadata.

How it solved itLakehouse table formats added ACID-like semantics, schema evolution, time travel, indexing/metadata, and table abstraction over object storage.

Why it matteredMerged warehouse-like reliability with lake-like open storage economics.

DDIACh. 4: Storage & Retrieval Ch. 12: Stream Processing

View on the timeline