2026 Data Engineering Roadmap

Introduction 🎯

#

One of the first things I did when I started studying to become a Data Analyst was try to answer a deceptively simple question:

After researching this question, I was left trying to make sense of a fairly broad field that was entirely new to me.

There were terms like:

• SQL
• Python
• Statistics
• BI / Data Visualisation
• Data modelling
• Databases, and an ever-growing list of tools and technologies.

Now, a few years later (and slightly more informed), I’m finding myself asking a similar question again:

Why do I care? ❓

#

I’m increasingly interested in moving into more technical work (not that cutting my teeth as a Data Analyst isn’t fun) using different technologies and new skills to solve harder problems.

As such, the path I’ve chosen is that of the “Data Engineer”! Therefore, I need to develop some kind of roadmap to understand what I need to learn to solve Data Engineering Problems.

TLDR 📋

#

If you really don’t want to read the article, and instead just want to see the resources I’ve recommended, here’s the condensed version:

Task Status Key:

• Not Started (❌)
• In Progress (🚧)
• Completed (✅)

Books 📚

#

Courses / Certifications / Misc 🎓

#

Preface 💬

#

Before I start, I’ll state two things:

1. This roadmap assumes competency in certain technologies, i.e. SQL, Python, Git, Linux etc - If you’re just starting out in the “tech-world”, I’d recommend developing foundational knowledge in some of these technologies before jumping straight into this Roadmap!

2. If you disagree with anything I’ve posted in this article, feel free to email me to discuss, given I’m not a Data Engineer, and only just learning about these topics, I’m in a position where mistakes could be made. As such, I’m happy to make amendments to this article as necessary.

How I’m going to learn 💡

#

Where possible, I recommend a blended approach when it comes to learning.

The most fatal mistake when learning something new is to never get your hands dirty (so to speak), and to just consume content adjacent to what you’re trying to learn.

So what do I mean when I say a “blended” approach? Well, instead of just watching the lectures or completing the readings, you also:

1. Create content sharing what you learned.
2. Talk to interested folk about what you learned and how you aim to apply it.
3. Actually apply what you learned, create a small (or big) project etc.

How this road-map might differ to other courses? 🆚

#

From my research, I see quite a few Data Engineering courses that approach learning Data Engineering as a list of technologies to learn, (which might look like something like the below):

graph TD
    A[Python]
    B[SQL]

    C[AWS]
    D[Docker]
    E[Kubernetes]
    F[Terraform]

    G[Spark]
    H[Kafka]
    I[Airflow]

    J[Snowflake]
    K[dbt]

    A --> G
    A --> I
    B --> J
    B --> K

    D --> E
    F --> C
    C --> D

    G --> C
    H --> G
    I --> C
    J --> C
    K --> J

This approach might make for a good CV keyword list, but it doesn’t necessarily produce a deep understanding of Data Engineering.

Instead, I’m approaching Data Engineering as a series of concepts, with the idea being that each layer builds on the one before it.

Starting out with… (drum-roll)

Computer Science Foundations

#

I’m starting here because I don’t actually have a Computer Science degree.

Although I’ve spent time “programming”, I’ve mostly learned by building things and messing around, rather than through formal computer science education.

This has worked reasonably well so far, but I increasingly want to understand why the things I’m building work. As such, I need to fill in some of those foundational gaps.

Now, while I’m not trying to become a Comp Sci Graduate, I’ll be utilizing a few classic University resources to learn about Computer Science Foundations, while utilizing programming as a thread to apply these learnings.

The resources that I’ve linked below will allow us to learn about concepts like:

• Algorithms
• Data structures
• Complexity
• Memory
• Processes
• Threads
• Concurrency
• Networking
• Operating systems

and how these concepts combine together to form the entire digital eco-system we know and enjoy today!

graph TD
    CS[Computer Science Foundations]

    CS --> A[Algorithms]
    CS --> DS[Data Structures]
    CS --> C[Complexity]
    CS --> M[Memory]
    CS --> P[Processes]
    CS --> T[Threads]
    CS --> CO[Concurrency]
    CS --> N[Networking]
    CS --> OS[Operating Systems]

    A --> C
    DS --> A
    M --> P
    P --> T
    T --> CO
    P --> OS
    T --> OS
    CO --> OS
    N --> OS

Harvard CS50

#

I’m going to be completing the CS-50 Introduction to Computer Science, which is a series of free lectures released by Harvard on an introduction to the intellectual enterprises of computer science and the art of programming.

(Which I actually made a start on a long time ago, but never finished…)

Grokking Algorithms

#

I’ve consistently seen Aditya Bhargava’s book, Grokking Algorithms recommended for Data Structures and Algorithm Theory.

If you’ve already read Grokking Algorithms and/or are slightly more competent with DSA (Data Structures and Algorithms) i.e., not me! - Thomas Cormen’s book, Algorithms - Theory and Practice may be a much better use of your time.

LeetCode / NeetCode

#

I recommend using LeetCode as a way to test whether you’re actually understanding and retaining the concepts you’re learning around Data Structures and Algorithms.

Now, if you’ve never heard of LeetCode (or NeetCode) it’s a platform for practising programming problems, particularly algorithms and data structures, commonly used in software engineering interviews.

I’m specifically going to be completing the NeetCode Blind 75.

The NeetCode Blind 75 centers on a curated list of 75 LeetCode-style problems that target common coding interview patterns and data structures encountered in technical interviews.

The benefit of this approach is that rather than grinding through thousands of LeetCode problems without direction, you’re able to work through this carefully selected subset, organized by difficulty.

Also, I like this list because I think that Navdeep passes the “Beer Test”.

OSTEP

#

OSTEP Operating Systems: Three Easy Pieces - For learning about Operating Systems (or the Comet Book) is centered around three conceptual pieces that are fundamental to operating systems:

• virtualization
• concurrency
• persistence

So, why OSTEP for learning about computer operating systems? Well to be blunt, OSTEP is widely considered one of the best computer science textbooks ever written. I haven’t even mentioned that it’s free (accessible via the link above!)

Off-topic, but I also enjoyed Remzi Arpaci-Dusseau’s article on why textbooks should be free - Well worth a read if you have the time!

Computer Networking: A Top-Down Approach

#

Computer Networking A Top-Down Approach by Jim Kurose and Keith W. Ross is another seminal text in the field of Computer Science, and is a widely recommended read for Computer Science students.

Also, if you don’t feel like reading the book, Jim has actually published videos onto YouTube, where he explains these concepts, through a series of free lectures!

Databases

#

Knowing SQL and understanding databases are two very different things.

I want to move beyond the basics and begin to understand what happens under the hood when I press enter on a SQL Query + answer some of the following questions (as a way to demonstrate understanding).

• How is the query parsed?
• How does the database decide how to execute it?
• What is an execution plan?
• How do indexes work?
• Why are some queries fast and others slow?
• What are transactions?
• What does ACID actually mean?
• How does concurrency work?
• What is MVCC?
• How does a database store data on disk?
• What happens when the database crashes?

While I don’t know what many of the terms in this mermaid chart mean, I was informed enough to put it together!

(However, if you asked me about each of these, I wouldn’t be able to give you a full answer)

flowchart TD

A["SQL Query"] --> B["Query Parsing"]

B --> C["Query Optimisation"]

C --> D["Execution Plan"]

D --> E["Query Execution"]

E --> F["Indexes & Access Paths"]

F --> G["Storage Engine"]

G --> H["Pages / Disk"]

E --> I["Transactions & ACID"]

I --> J["Concurrency"]

J --> K["MVCC"]

H --> L["Write-Ahead Log"]

L --> M{"Database Crash?"}

M -->|No| N["Commit"]

M -->|Yes| O["Recovery"]

O --> N

We’re going to use the resources below to learn more about each of these steps!

Use the Index Luke

#

Use The Index, Luke is the free web-edition of Markus Winand’s book SQL Performance Explained.

The websites biggest strength is that it teaches you to think about SQL as something executed by a database engine, rather than just a language you use to retrieve data.

Database Internals

#

Database Internals by Alex Petrov, is widely recommended in Data Engineering circles, however, the book is fairly dense, a couple of dudes on Reddit recommended taking like 6 months to read it, first giving it a skim, then coming back to it with a fine-tooth comb.

While researching, I also found a great series of lectures on Database Internals on YouTube from a bloke named Professor Andy Pavlo.

Designing Data Intensive Applications

#

Designing Data Intensive Applications by Martin Kleppmann is another one of these “dense” books, that is almost a rite of passage in the Data Engineering community.

Once again, a couple blokes on Reddit just recommended taking your time with the book and being methodical.

Data Modelling

#

Databricks defines Data Modelling as:

“The process of creating a visual blueprint or structure for how data is stored, organized, and related within an information system or database. It maps real-world objects—like customers, products, or orders—and defines how they connect to one another”.

Some of the Data Modelling concepts I’m hoping to learn more about include:

• Fact tables
• Dimension tables
• Star schemas
• Snowflake schemas
• Normalisation & Denormalisation
• Slowly Changing Dimensions
• Aggregate tables

I’d also like to be able to look at a collection of source tables and answer questions like:

• What should the model look like?
• Who is going to consume it?
• What should the grain be?
• What should be calculated / downstream?
• How will the model behave as the data grows?
• How should changes to the underlying data be handled over time?

So, how are we going to learn all of this?

Fundamentals of Data Engineering

#

The general consensus on Fundamentals of Data Engineering is that the book is exceptionally useful at providing a high level generalized overview of the current state of Data Engineering.

One bloke notes that it’s:

“Not going to make you better at your job but will be extremely helpful for interviews and being able to speak about the data engineering landscape and key concepts at a high/medium level.”

With this being said, a few people (in the reddit threads I read) mention that the book is a bit of a slog, so we’ve got that to look forward to.

LLM-Based Modelling Practice

#

As I mention throughout this article, I don’t want this roadmap to be all theory.

As such, I’d like to get to working with a few LLM based prompts along the lines of:

Heya, ChatGPT/Claude/Gemini etc,

Question:
I'm looking to practice my data-modelling skills, could you please throw me a messy set of data and then I'll return a possible data-model?

I'd like to work specifically on the following tasks:
* Grain Determination
* Entity Identification
* Relationship Identification i.e.:
  * One customer can have many ______.
  * One order can contain many ______.
  * One product can appear in many ______.
* Determine the Keys
* Identify problems with the raw data.

Output:
* Please output a "made-up" table and a list of questions designed to improve my data-modelling skills

Context:
* I'm looking to become a Data Engineer and wish to improve my skills
* Take the tone of a teacher / lecturer for this specific task.

This task will be beneficial as we’re actually beginning to get experience solving (potentially) real Data-Modelling business problems!

Distributed Systems

#

A lot of the problems in modern Data Engineering exist because one computer isn’t enough.

If I have a few thousand rows, almost anything will work, however, what happens when the data grows, think 1 Million, 1 Billion, 10 Trillion - What happens then?

At some point, I need multiple machines. And as soon as I have multiple machines, I inherit a whole new category of problems.

• Machines fail.
• Networks fail.
• Messages arrive late.
• Machines disagree.
• Data needs to be replicated.
• Work needs to be partitioned.
• Different machines need to coordinate.

These problems inevitably lead into concepts like:

• Partitioning
• Replication
• Fault tolerance
• Consistency
• Availability
• CAP theorem
• Consensus
• Distributed transactions
• Eventual consistency

So, how are we learning about Distributed Systems? - Well, to be honest, we’ve got a couple of options here, as Designing Data-Intensive Applications already does a fairly good job with Distributed systems, however, if you really wanted to punish yourself…

MIT 6.5840 — Distributed Systems

#

MIT’s 6.5840 is a graduate distributed-systems course covering fault tolerance, replication and consistency, with lectures, papers and programming labs.

This is considerably more academic than we may necessarily need for Data Engineering, but it’s still a great resource nonetheless.

With this being said, I don’t think that I’ll be completing it in the near future…

Data Systems

#

The first part of this roadmap has been spent building out foundations.

We’re now in a position where we understand:

• how computers work
• how databases store and retrieve data
• how data should be modelled
• what happens when our systems grow beyond a single machine.

At this point, I don’t think reading another ten books is going to make us better Data Engineers.

So, I proclaim to the world:

The goal of this section is to take everything we’ve learned so far and start putting the pieces together into actual data systems.

A modern data platform might look something like this:

API
 ↓
Ingestion
 ↓
Object Storage
 ↓
Transformation
 ↓
Warehouse
 ↓
Data Model
 ↓
Analytics / ML
 ↓
Application

On a whiteboard, this looks relatively straightforward, however in reality, this is where things get interesting.

• What happens when the API goes down?
• What happens when we receive the same record twice?
• What happens when yesterday’s data arrives tomorrow?

And perhaps most importantly:

How do we know whether the data we’re producing is actually correct?

These are the kinds of problems that start separating writing data pipelines from engineering data systems. This is where concepts (such as the following) start becoming much more than just theoretical concepts:

• Idempotency
• Retries
• Backfills
• Data quality
• Data contracts
• Schema evolution

As I said before, I don’t think another purely theoretical course is going to help us learn about Data Systems, so instead we need something that forces us to build.

This is where DataTalksClub’s Data Engineering Zoomcamp comes in.

Data Engineering Zoomcamp

#

ziritrion/dataeng-zoomcamp

Jupyter Notebook

376

189

The Data Engineering Zoomcamp is a practical, project-oriented course covering many of the technologies and workflows that show up in modern Data Engineering.

More importantly, it gives us something we’ve been missing so far:

A reason to actually put the theory into practice.

Rather than studying each technology in isolation, we’ll start building pipelines that have to deal with ingestion, storage, transformation, orchestration, infrastructure and deployment.

Production Engineering

#

Building something that works is one thing. Building something that continues working is another.

Production Engineering is where I want to learn about:

• Monitoring
• Logging
• Alerting
• Reliability
• Incident response
• Testing
• Deployment
• Rollbacks
• Performance
• Security
• Cost

Google SRE Books

#

I’ve heard the following books from Google recommended as resources for learning about Site Reliability Engineering and the underlying principles and learnings that guide this field of study:

• Site Reliability Engineering (2016)
• The Site Reliability Workbook (2018)
• Building Secure & Reliable Systems (2020)

I’m only going to read The Site Reliability Workbook, because I’m more interested in learning how to apply SRE principles than reading three books covering largely overlapping ideas.

Other useful Data Engineering Resources

#

Well, well well! You’ve now reached the end of the road-map, congratulations!

Now, before I leave you to your lonesome, I thought I’d drop off a few more useful resources that you can consume and research at your own pace:

Data Engineer Wiki

#

Data Engineering Wiki

Data Engineering Cookbook

#

andkret/Cookbook

The Data Engineering Cookbook

Python

15370

2753

Data Engineer Handbook

#

DataExpert-io/data-engineer-handbook

This is a repo with links to everything you’d ever want to learn about data engineering

Jupyter Notebook

43867

9254

Data Engineering for Beginners

#

Data Engineering for Beginners

Missing Semester

#

Missing Semester

Data Engineering Certifications

#

The only Data Engineering Certificates I’m going to go after are:

• Databricks - Data Engineer Associate
• Databricks - Data Engineer Professional

The reason for this is that I’ve already worked with Databricks and had exposure to some Data Engineering on the Azure Databricks platform.

With this being said, there are also the following other options for certification, and feel free to choose one that works best for you:

• Databricks - Certified Associate Developer for Apache Spark
• Google Cloud Certified - Professional Data Engineer
• Microsoft DP-600: Fabric Analytics Engineer Associate
• Microsoft DP-700: Fabric Data Engineer Associate
• AWS Certified Data Engineer - Associate

Useful Media Personalities in the Data Engineering Space

#

Benjamin Rogojan (Seattle Data Guy)

#

Benjamin Rogojan (Seattle Data Guy)

Alexey Grigorev (DataTalksClub)

#

Alexey Grigorev

Ben Dickens

#

Ben Dickens

Joseph Machado (Start Data Engineering)

#

Start Data Engineering

Data with Zach

#

Data with Zach

Conclusion 🏁

#

If you’ve made it this far, you’ll probably have noticed that this isn’t really a roadmap for learning a bunch of technologies.

I’m much more interested in developing a mental model for how data systems work, and then using that understanding to decide which tools are appropriate for a particular problem.

I’m also not in any rush. I’ll be learning, building things, writing about what I learn, and inevitably discovering things I’ve got wrong along the way.

And while this roadmap focuses heavily on the technical side of Data Engineering, being a good Data Engineer isn’t just about technical knowledge. Communication, presentation, business context, and domain knowledge are just as important.

Cheers!