Why do we need the Data-Lakehouse?

Introduction 🎯

#

Data, Data, Data… Everyone has data coming out their ears, and with so much of it, users keep reaching for a new system to store it in.

First it was the database. Then the warehouse. Then the lake. And now we’re up to the lakehouse.

Well my dear user, before I answer this question, we first need to understand the two jobs data systems are actually asked to do on a day-to-day basis…

OLTP vs OLAP ⚖️

#

OLTP (Online Transaction Processing)

#

OLTP systems are generally concerned with running an application.

Imagine an online shop called “Russells Widgets”. Someone places an order (for a widget), and the system needs to:

1. Create the order
2. Update the customer’s account
3. Reduce the inventory (subtract a widget)
4. Record the payment

And it needs to do all of that correctly and quickly, every single time. A typical OLTP database might look something like:

graph TD;
    A["Application"] --> B[("PostgreSQL")];
    B --> C["customers"];
    B --> D["orders"];
    B --> E["products"];
    B --> F["payments"];

This is where traditional relational databases shine: transactions, constraints, consistency, concurrent writes, fast lookups. They’re built to protect one order at a time.

OLAP (Online Analytical Processing)

#

OLAP has a completely different problem. Instead of asking “Which widget did a user just buy?”, we might ask:

“What were our total widget sales by product, region and month over the last five years?”

That’s a different animal, (and the same beast).

So, instead of lots of tiny, protected transactions, we’re interested in large analytical queries.

OLTP runs the business, (kind of like the workers of a business), and OLAP is about analysing it, (kind of like the C - Suite of the same business)

Now, if you didn’t find my explanation fulfilling, I’ve linked off to a video by techTFQ to further explain the differences between the two systems.

(btw @TechTFQ, your videos rock, but I don’t like your new AI thumbnails)

Database’s 🗄️

#

Databases are fundamentally a system for storing and retrieving data, and when people say “database” (in a Data Engineering context), they usually mean a relational one.

Generally, people are referring to one of the big four (listed below):

• PostgreSQL
• MySQL
• Microsoft SQL Server
• Oracle (yuck)

These systems are great at OLTP workloads (such as the example flow, I’ve featured below):

graph TD;
    A["Application"] --> B[("Database")];
    B --> C["Orders"];
    B --> D["Users"];
    B --> E["Products"];

The database is the system of record, it holds the data the application needs to actually function. And that’s exactly why it starts to “break” when the CEO wanders over and asks:

“Can you tell me how revenue has changed by customer segment over the last ten years?”

We could run that query against production.

But we probably shouldn’t, as our poor PostgreSQL instance has enough problems without a ten-year analytical scan running while 10,000 customers are mid-checkout (or some other production-esque situation).

So the database alone can’t be the whole answer, we instead need somewhere else to send those questions…

Data Warehouses 🏢

#

The Data Warehouse exists to run analytical workloads, without wrecking the application.

Instead of being the database that powers our application, it becomes the place we bring data together specifically to be analysed:

graph TD;
    A["Application (Widget Store)"] --> B[("Database")];
    B --> C["Ingestion"];
    C --> D[("Data Warehouse")];
    D --> E["BI / Analytics"];

Now the production database can concentrate on running the application, and the warehouse can concentrate on answering questions.

Because analytical workloads have different needs from transactional ones, the warehouse reshapes the data for that job:

graph TD;
    A["Sales"] --> B["Customer"];
    A --> C["Product"];
    A --> D["Date"];

This is where fact tables, dimension tables, star schemas and aggregations show up. And to be clear, this is a genuinely good solution — for structured, well-understood business data.

But it only solves half of the original problem. It assumes the data already looks like rows and columns. What happens when it doesn’t?

Data Lakes 🌊

#

Imagine Russell’s Widgets company starts collecting data from:

• APIs
• CSV files
• JSON
• Logs
• IoT devices
• Images
• Videos

Forcing all of that into a tidy relational schema before we’ve even stored it is a losing game.

So the Data Lake solves a different problem than the warehouse does: not “how do we analyse structured data”, but “where do we even put everything else, cheaply, before we know what we’ll do with it?”.

graph TD;
    subgraph Sources["Data Sources"]
        A["API"]
        B["CSV"]
        C["Logs"]
        D["IoT"]
    end
    A --> E[("Data Lake")];
    B --> E;
    C --> E;
    D --> E;
    E --> F["Processing"];
    F --> G["Analytics / ML"];

So, why would we use this particular paradigm? Well, it’s super useful when:

• We don’t know exactly how the data will be used yet
• We have very large datasets
• We need to retain raw data
• We work with semi-structured or unstructured data
• We want cheap, scalable storage

So now we’ve solved the flexibility problem the warehouse couldn’t.

Great.

Except the lake trades that flexibility for something else…

Remember our Reddit user from the beginning? Mr “That’s the neat thing, they all become data swamps.”

They weren’t capping, fr.

If you dump everything into object storage without:

• organisation structure
• metadata
• governance systems
• ownership or documentation

you lowkey end up with this:

data/
├── final.csv
├── final_final.csv
├── final_v2.csv
├── definitely_final.csv
├── raw/
├── raw_new/
├── raw_old/
├── stuff/
└── PLEASE_DO_NOT_DELETE/

Scary stuff huh? ^ 🫣

So, the lake’s greatest strength, the whole “put anything here, we’ll figure it out later” is also exactly what turns it into a swamp.

And that’s the second unsolved half of the problem: we now have a place that’s flexible enough to hold everything, but not reliable enough to actually trust for analytics.

So Why Do We Need the Lakehouse? 🏠

#

Take a look at where that leaves us:

Neither of these two systems is inherently wrong (neither is completely right either)…

Plenty of organisations and teams end up running both:

• a lake (for raw everything)
• a warehouse (for trusted BI)

And a bunch of pipelines copying data between them just to keep both sides happy.

However, that means there’s two systems to pay for, two systems to secure, and data duplicated (and drifting out of sync between them).

That duplication is the actual reason the Data-Lakehouse exists.

The Lakehouse keeps the lake’s cheap, flexible object storage as the foundation, but adds a layer on top that brings in the things that made warehouses trustworthy in the first place:

• ACID transactions
• Schema enforcement
• Schema evolution
• Versioning / time travel
• Table metadata
• Data governance
• Performance optimisation

Compute vs Storage ⚙️

#

There’s another important idea hiding underneath the whole Data Lakehouse architecture: storage and compute can be decoupled.

Traditionally, a database is responsible for both:

• Storage
• Compute

However, Cloud data platforms changed this relationship.

Instead, we can keep our data in relatively cheap object storage and bring compute to it when we actually need to do something with it:

graph TD;
    subgraph Compute
        A["Spark"]
        B["SQL"]
        C["BI / ML"]
    end

    A --> D[("Object Storage")];
    B --> D;
    C --> D;

    D["S3 / ADLS / GCS"];

This is one of the reasons data lakes became so attractive. We can store enormous amounts of data without needing a database engine sitting there processing it 24/7.

ETL vs ELT 🔄

#

We’ve talked a lot about moving data around, but there’s another question… When should we transform it?

There are two common approaches: ETL and ELT.

ETL — Extract, Transform, Load

#

The traditional approach is:

Source
  ↓
Extract
  ↓
Transform
  ↓
Load
  ↓
Warehouse

We take the data out of the source system, clean and reshape it, and then load the transformed result into the warehouse.

This made a lot of sense when storage and compute were relatively expensive and the warehouse was expected to contain mostly clean, structured data.

ELT — Extract, Load, Transform

#

Modern cloud platforms often flip that around:

Source
  ↓
Extract
  ↓
Load
  ↓
Data Lake / Lakehouse
  ↓
Transform
  ↓
Analytics

Instead of transforming everything before storing it, we can store the raw data first and transform it later.

This works particularly well with cheap, scalable object storage.

It also means we don’t necessarily have to decide exactly what the data will look like before we’ve even stored it.

And this is another reason the lakehouse architecture is interesting: the same underlying data can support raw ingestion, transformation, analytics and machine learning without necessarily requiring a separate copy of the data for every stage.

So, very roughly:

ETL: “Clean it, then store it.”

ELT: “Store it, then figure out what to do with it.”

The Mental Model 🧠

#

Wrapping up, if I had to boil this whole post down to why each system exists, not just what it does, I’d put it like this:

• Database = “I need to run an application, correctly, right now.”
• Data Warehouse = “I need to analyse structured business data, without breaking the application.”
• Data Lake = “I need somewhere cheap and flexible to store everything else.”
• Data Lakehouse = “I need the lake’s flexibility and the warehouse’s trust, without running two systems to get both.”

So no, you don’t always need a lakehouse.

Plenty of business’s and teams exist right now who are served perfectly well by just a database, or a warehouse, or a lake on its own.

But once you’ve felt the pain of maintaining two systems just to cover each other’s gaps, it becomes obvious why the lakehouse showed up.

Cheers!