The Flow, Not the Tools

The gap between an engineer who "knows Kafka" and one who can build a data platform is not tool count. It is how they reason about movement. A junior reaches for the right product. A senior reaches for the right guarantee. When you can name what a dataset actually needs, how fresh, how complete, how replayable, and then choose deliberately, the pipeline stops being a pile of connectors and becomes a sequence of decisions you can defend at 3 a.m. with the pager in your hand.

Every concept in this manual is an answer to a specific pressure. CDC answers "the nightly copy misses deletes and melts the source." A log answers "five consumers need the same events at five different speeds." Watermarks answer "events arrive out of order and I still have to close the window." Learn the pressure first and the technique is obvious. Learn the technique first and you are carrying a logo collection you will freeze on, because you will not know which one the incident in front of you is asking for.

So we build the way you actually build. Start with the smallest pipeline that works. Push on it until it breaks. Name the force that broke it. Apply the smallest move that fixes it. Repeat. By the end, the picture above is assembled, and you have watched every piece arrive for a reason you can state in one sentence.

00 / The FrameHow to think before you move a single byte

Before any component goes on the whiteboard, there is an order of operations, and it works because a pipeline is a product with consumers, not a chore with a schedule. The mistake under pressure is to start naming tools. Resist it. Tools first means you commit to machinery before you understand the workload, and then you spend a year operating machinery the workload never asked for.

Five questions, asked in order, decide everything downstream:

• Who consumes this, and what do they do with it? A dashboard refreshed each morning, a fraud model scoring in 50 milliseconds, and a finance report closed monthly are three different physics problems wearing the same word, "pipeline."
• How fresh must it be? Not "how fresh would be nice." How stale before someone is actually harmed. This single number, minutes versus seconds versus days, is the biggest cost driver in the whole design, and people set it by vibes.
• How correct must it be? Can a duplicated event or a missing hour be tolerated, detected later, or never? Billing and analytics give opposite answers, and mixing them in one pipeline gives you billing-grade cost for analytics-grade value.
• Will you ever need to recompute the past? New metric definition, bug in a transform, a model retrained over two years of history. If yes, and it is always yes, replayability must be designed in on day one, because it cannot be bolted on after the raw data is gone.
• Who gets paged? A pipeline without a named owner is not infrastructure. It is a future incident with a start date you have not learned yet.

Learn to count before you stream

You cannot reason about scale without estimating it, and estimation is a handful of numbers plus the courage to round hard. A day is about 100,000 seconds. So 100 million events per day is roughly 1,000 events per second on average, and with a peak factor of five, you are designing for about 5,000 per second. At one kilobyte per event that is 5 MB/s at peak and roughly 100 GB per day, call it 36 TB a year raw. Every one of those numbers is small for modern hardware, and knowing they are small is what stops you from deploying a twelve-node cluster to carry what a single well-fed Postgres and a cron job would honestly carry for years.

The second arithmetic habit: cost scales with how often you touch data, not just how much you have. Scanning a 10 TB table nightly is 3.6 PB of reads a year. Scanning yesterday's 30 GB partition nightly is 11 TB a year, three hundred times cheaper, for the same business answer. Most pipeline cost disasters are not volume problems. They are "we re-read everything, every run" problems, which is exactly where our story begins.

01 / The First ForkFreshness or cost: batch and streaming are one dial, not two religions

The first deep dive, almost every time, is a single question that decides the personality of the whole pipeline: how quickly must a change in the source become visible to the consumer? The industry stages this as a war, batch versus streaming, ancient versus modern. That framing is noise. It is one dial, and every notch buys freshness by spending money and operational complexity.

At one end, a nightly batch: simple, cheap, debuggable by rerunning it, and 24 hours stale. At the other end, event-at-a-time streaming: seconds fresh, and you have signed up for brokers, consumer lag, out-of-order time, and stateful processing that must survive restarts. In between sit hourly batches and micro-batches, which are respectable engineering, not cowardice. Drag the dial and watch what each notch costs.

Here is the part that separates the strong answer from the weak one: almost no real platform sets this dial once, globally. Different datasets sit at different notches. Fraud features and inventory counts earn streaming, because a stale answer costs money per second. The finance mart and the executive dashboard are perfectly happy at nightly, and moving them to streaming is a subsidy paid to nobody. The senior answer is never "we are a streaming shop." It is "these three datasets need seconds and here is the harm that justifies it; everything else runs on schedule and we sleep at night."

02 / One Cron JobThe smallest pipeline that works

Every planetary data platform started as this: a script, a schedule, a copy. At 02:00, SELECT * from the production database, write it into the warehouse, replace yesterday's copy. For a real company with a modest database, this is genuinely enough, and do not let anyone shame you out of shipping it. It is correct, cheap, and any engineer can debug it by reading forty lines.

# the ancestral pipeline, 02:00 every night
pg_dump --table=orders prod_db \
  | load --replace warehouse.orders

It also has exactly three weaknesses, and almost everything in the rest of this manual is a cure for one of them.

• It reads everything, every time. The cost of each run grows with the size of history, not with the size of change. At 10 GB this is invisible. At 2 TB the "nightly" job takes eleven hours, saturates the production database's I/O while real customers are using it, and finishes after the analysts arrive.
• It sees snapshots, not history. A row updated three times between runs shows up once, in its final state. A row deleted between runs simply vanishes, silently, and your warehouse now disagrees with reality in a way no error message will ever report. State tells you where things are. It cannot tell you what happened.
• It is one hop with no memory. If the job dies at 03:40, there is no queue holding the work, no record of partial progress, no second consumer with its own copy. There is only a missing table and a morning of questions.

Three weaknesses, three forces: cost proportional to history, loss of change information, and fragility of the single hop. We relieve them in that order, because that is the order in which they hurt.

03 / The DiffFull loads, incremental loads, and the tap on the log

The first fix is obvious once the force is named: stop moving data that did not change. Add an updated_at column, remember the high-water mark from the last run, and each night pull only the rows above it. This is incremental extraction, and it converts cost-per-history into cost-per-change, which is the single largest efficiency win available in all of data engineering. Take it early and take it everywhere.

But watch what incremental extraction still cannot see. A deleted row has no updated_at, because it has no anything; it is simply absent, and absence does not show up in a query for recent changes. A row updated twice between runs still collapses to its final state. And the whole scheme trusts every application to maintain the timestamp column honestly, which some ORM, someday, will not. The diff query is an inference about what changed. What you actually want is the record of what changed, and your database has been keeping one all along.

Every serious database persists a write-ahead log, the ordered record of every insert, update, and delete, written before the change is applied, because that is how the database itself survives a crash. Change data capture (CDC) taps that log and turns it into a stream of change events: before-image, after-image, operation, position. Debezium is the standard tap. The deletes appear, because the log records them. The intermediate states appear, because the log records everything. The source barely notices, because reading a log file is nothing like scanning a hot table.

The trap that makes CDC the principled choice, not the lazy one

There is a deeper reason the log is the right tap, and it is the most instructive bug in distributed systems. Suppose your service, wanting to be modern, updates its database and then publishes an event about the change:

// the dual-write trap
await db.orders.update(order);      // succeeds
await broker.publish(orderUpdated); // process dies here

Two systems, no shared transaction. Eventually the write lands and the publish does not, or the reverse, and now your stream and your database disagree about reality, permanently, with no error anywhere. At production volume this is not a rare edge case; it is a weekly certainty. The two honest cures both collapse the two writes into one: the transactional outbox, where the event is written into an outbox table inside the same database transaction and relayed out from there, or CDC itself, where the write-ahead log is treated as the one true record and events are derived from it. Either way the principle is identical, and it is worth engraving: the log is the only place where "what actually happened" is recorded atomically. Everything else is somebody's opinion about what happened.

So now change events flow out of the source cheaply and completely. They need somewhere to land, and "somewhere" turns out to be the most important data structure in this entire field.

04 / The LogWhy Kafka is a data structure, not a message queue

The cron job's third weakness was the single hop with no memory. The change events from CDC deserve better than being piped straight into one destination, because the moment they are useful, everyone wants them: the warehouse loader, the search indexer, the fraud detector, the cache invalidator. Five consumers, five speeds, five failure schedules. Point-to-point delivery to five destinations from one producer is five integrations that each break independently, which is the reliability arithmetic from every distributed systems lesson ever taught: chained things multiply their failures.

The move is to put a durable, ordered, append-only log in the middle, and this is what Kafka actually is. Not a queue that deletes messages once someone reads them; a log that keeps them. Producers append at the head. Each consumer keeps its own bookmark, an offset, and reads at its own pace. A slow consumer inconveniences nobody. A crashed consumer resumes from its bookmark. A brand new consumer, deployed a year later, can start from the beginning and read history it was never alive for. The log decouples producers from consumers in time, which is the decoupling that matters, and it is why the log sits at the center of the architecture diagram rather than off to one side.

One machine cannot hold a planetary firehose, so the log is split into partitions, each an independent ordered log on its own broker. And here is the sentence that explains half of all Kafka behavior, interviews, and incidents: ordering is guaranteed within a partition and nowhere else. The producer chooses a partition by hashing a key, so all events for user-42 land in the same partition and stay in order relative to each other, while events for different users interleave freely. Choose the key badly, say by country when one country is 60 percent of traffic, and one partition carries 60 percent of the load while its siblings idle. That is a hot partition, and no amount of adding brokers fixes it, because the skew is in your key, not in the cluster.

Consumers scale the mirror-image way: a consumer group divides the partitions among its members, each partition owned by exactly one member at a time. Three partitions can feed at most three active consumers in a group; a fourth stands idle, which is the simulator's quiet lesson that partition count is the ceiling on consumer parallelism, chosen at topic creation and expensive to regret. When a consumer dies, the group rebalances and survivors adopt its partitions, resuming from the committed offsets. Death, in this design, is an ordinary event with a rehearsed response, which is precisely what the cron job never had.

Two more sentences about the log that pay rent for years. Its memory is bounded by retention, and retention has three modes worth knowing by name: time-based, size-based, and compacted, where the log keeps only the newest event per key and a topic becomes a changelog you can rebuild an entire state store from. And tiered storage now lets brokers spill old segments to object storage, which quietly blurs the line between the log and the lake we build in section 08, and turns "replay from months ago" from an impossibility into a line item.

05 / The PromiseDelivery semantics, and the myth of exactly-once

The log made delivery durable. It did not make delivery clean, and here lives the most misunderstood promise in data engineering. Systems fail between doing work and recording that the work is done. A consumer reads an event, writes the result to the database, and crashes one millisecond before committing its offset. On restart, the log does exactly what it should: it re-delivers from the last committed bookmark. The work happens twice.

You get to choose which side of this cliff to fall off. Commit the offset before processing and a crash loses the event: at-most-once, acceptable for metrics nobody bills on. Commit after processing and a crash duplicates the event: at-least-once, the default everywhere that data matters. The thing everyone wants, exactly-once, is not purchasable as a global property, because it would require the delivery and the side effect to share one atomic transaction across systems that do not share anything. What is achievable is effectively-once: at-least-once delivery plus consumers that make duplicates harmless. That property has a name you already know.

This is why idempotency is not an optimization but the entry fee. Every consumer downstream of a log must be written as if every message will arrive twice, because eventually it will: keyed upserts instead of blind inserts, event IDs recorded and checked, MERGE instead of INSERT in the warehouse loader. Kafka's transactions can give true exactly-once within a Kafka-to-Kafka topology, which is genuinely useful and genuinely narrow; the moment your consumer touches Postgres or S3 or an external API, you are back to idempotency as the load-bearing wall. Design for duplicates and duplicates become boring. That is the entire promise on offer, and it is enough.

One more trick worth carrying, because it comes up constantly and almost nobody names it: when the sink is a transactional database, store the consumer's offset in that database, updated in the same transaction as the side effect. The work and the bookmark now share one commit, and the crash window between them simply does not exist for that path. Notice this is not a framework feature; it is the dual-write cure from section 03 wearing consumer clothes. The pattern is always the same, and once you see it you see it everywhere: find the two writes that can disagree, and make them one.

06 / TimeThe two clocks, and the hardest idea in streaming

Everything so far had one merciful property: it did not care when events happened, only that they arrived. The moment you compute anything over a time window, "orders per minute," "clicks per session," "spend in the last hour," you collide with the fact that every event carries two timestamps, and they disagree.

Event time is when the thing happened, stamped at the source. Processing time is when your pipeline got around to seeing it. In a healthy system they differ by milliseconds. Then a phone rides the metro under Sofia for six minutes and syncs its buffered clicks on emerging. A producer retries after a deploy. A partition recovers and floods an hour of backlog in forty seconds. Events arrive minutes or hours after they occurred, interleaved with fresh ones, and any window computed on processing time quietly assigns them to the wrong bucket. The graph looks plausible. It is wrong, and nobody will ever file a bug, because plausible-but-wrong is invisible.

So real windows are computed on event time, which creates the genuinely hard question: when is a window finished? The events for 14:00 to 14:05 can keep dribbling in until 14:11 or 14:40. Wait forever and you never emit a result. Close instantly and you drop the stragglers. Streaming engines resolve this with a watermark: a moving assertion that says "I believe I have now seen everything up to time T," typically trailing the newest observed event time by a chosen lateness budget. When the watermark passes a window's end, the window closes and emits. It is not knowledge. It is a bet, made explicit and tunable, and making the bet explicit is the entire contribution.

The lateness budget is a business decision wearing an engineering costume. A short budget gives fast, slightly-lossy answers; a long budget gives complete, slow answers; and "accept late data" gives fast answers followed by corrections, which pushes the burden onto every consumer that must now handle a result that changes after being emitted. There is no setting called "correct." There is only the question from the Frame, "how stale before someone is harmed," answered honestly, per dataset, and written down where the next engineer will find it. Batch pipelines, note, face the same physics and answer it silently: "we process yesterday's partition at 06:00" is just a watermark of six hours that nobody calls by its name.

07 / The ShapeETL, ELT, and the trust gradient

Data is flowing; now it must be shaped, because raw events answer no business question directly. For decades the shape was ETL: extract, transform on a dedicated engine, then load only the polished result into the expensive warehouse. The ordering was economics, not philosophy. Warehouse compute and storage were scarce and coupled, so you refined the ore before shipping it.

Cloud warehouses and object storage broke the coupling: storage became almost free and compute became rentable by the second, and the letters flipped to ELT. Load everything raw first, transform inside the warehouse afterward, with transformations written as SQL, versioned in git, tested in CI. This is the ground dbt stands on. The flip matters for one reason above all the others: when raw data is retained, transformations become reproducible and repairable. A bug in the transform is no longer a permanent scar on the data; it is a code fix plus a rerun. Hold that thought for section 10, because it is the seed of the whole replay story.

Inside the warehouse or lakehouse, transformations arrange themselves into layers, popularly called medallion: bronze, silver, gold. Treat the names as marketing for an old and load-bearing idea, a trust gradient. Bronze is raw, exactly as it arrived, immutable, the evidence locker. Silver is cleaned, deduplicated, typed, conformed, the working truth. Gold is modeled for consumption: facts and dimensions, metric definitions, feature tables. Each boundary is a contract: a consumer of silver may assume deduplication happened and need not re-check it. Trust becomes a property of the topology instead of a property of each analyst's diligence, and "is this table safe to use" stops being tribal knowledge.

-- silver: one honest row per order, from at-least-once bronze
select *
from bronze.order_events
qualify row_number() over (
  partition by event_id        -- the idempotency key, again
  order by ingested_at desc
) = 1

Notice what that query is: section 05's idempotency, expressed in SQL. The forces do not change between streaming and batch; only the syntax does. And one discipline binds the whole layer: a metric defined twice is a metric defined zero times. The moment "revenue" is computed one way in the gold model and another way in someone's dashboard formula, the Monday meeting becomes an argument about whose number is real. Definitions live in one place, in code, under review. Everything else references them.

08 / The LakeObject storage, the small files tax, and tables made of files

Everything raw and everything historical eventually lands in object storage, S3 and its siblings, because nothing else offers eleven nines of durability at a price that lets you keep everything forever. The lake is the platform's long-term memory, and the memory is what makes replay possible. But object storage is not a database. It is a bag of immutable files, and two consequences of that fact dominate lake engineering.

The first is layout is the index. There are no B-trees over a bag of files; the only way a query avoids reading everything is if the files are organized so most of them can be skipped. Columnar formats like Parquet let a query read only the columns it touches, and their internal statistics let engines skip row groups. Directory partitioning, /date=2026-07-12/, lets a query over yesterday touch one directory instead of three years. Every fast lake query is fast because of pruning, and every ruinously expensive one is a full scan someone did not notice they were writing.

The second consequence is quieter and eats streaming pipelines alive: the small files problem. A streaming writer flushing every thirty seconds across two hundred partitions produces hundreds of thousands of files a day. Each file costs a listing call, an open, a metadata fetch, a planning entry. The data volume is identical; the file count is the tax.

Now the harder problem. Files are immutable and there is no transaction manager, so the primitive operations a data platform needs are all unsafe by default. Update a row? Rewrite the file. Delete a user for a GDPR request? Find every file containing them, rewrite each. Have a reader list files while a writer is mid-job? The reader sees half a commit, and this is not hypothetical; it is the default behavior. The raw lake gives you durability and none of the semantics you actually need.

Open table formats, Iceberg and Delta Lake, close this gap, and they are the most important lake development of the past decade. The idea is small and mighty: keep a transaction log of metadata about the files, and make the log the truth. A table is no longer "whatever files sit in this directory"; it is "the set of files the log's current snapshot points to." A write prepares new files invisibly, then commits by atomically advancing the log one pointer. Readers see the previous complete snapshot or the new complete snapshot, never a torn middle. That one mechanism buys, in a single purchase: ACID commits on object storage, safe concurrent readers and writers, schema evolution tracked in metadata rather than in directory renames, and time travel, because old snapshots still exist and can simply be read.

-- the lakehouse party trick, and the audit team's favorite query
select count(*) from orders for timestamp as of '2026-07-01 00:00:00';

-- what did this table look like before Tuesday's bad deploy?
-- answered in one line, restored in one more

This is why "lakehouse" is not a buzzword but a specific claim: lake storage prices with warehouse semantics. The files are still cheap, still open format, still queryable by any engine. The log on top gives them the guarantees that previously forced you to copy everything into a proprietary warehouse just to get transactions. You will still likely run a warehouse for the hot, modeled, interactive tier. But the lake stopped being a swamp the day tables got a commit log, and it is no coincidence that the mechanism is the same one from section 04. It is logs, all the way down.

09 / TrustQuality is checked at two altitudes, and rejects need a graveyard with an alarm

A pipeline that moves garbage quickly is a garbage cannon. As the platform grows, quality stops being a virtue and becomes a subsystem, and the first thing to understand about it is that checks come in two mathematically different species, and no single tool or stage can host both.

Record checks are predicates on one row: the schema matches, the amount is non-negative, the currency is a real ISO code, the timestamp parses. They are pure, stateless, and instant, which makes them perfect for the stream. Enforce them at the earliest boundary, right where events enter the log, because the cost of a defect grows roughly an order of magnitude per stage it travels: a rejected write at the source is a log line, the same defect in the warehouse is a backfill, and in a trained model it is a quarter of quietly wrong decisions. Records that fail go to a dead letter queue with the reason attached, and here is what the diagrams never say: an unmonitored DLQ is where data goes to die politely. Without an alert on its volume and a named owner for replay, you have not built error handling; you have built a socially acceptable way to delete customer data. Crashing would at least get noticed.

Population checks are statistics over many rows: did at least 99.9 percent of expected orders arrive, is today's volume within three sigma of the seasonal baseline, is the null rate on this column drifting, does every order's customer exist in the customers table. No single record can violate these; a feed that silently died delivers zero rows, every one of them schematically perfect. Population checks need a closed window and a baseline, which makes them creatures of scheduled batch against the lake and warehouse, and which means they inherit the watermark problem from section 06: "is the data complete" always secretly means "complete as of when," and pretending otherwise is how teams end up with alerting nobody trusts.

And upstream of both sits the fix that prevents rather than detects: the data contract. Schema, semantics, freshness and completeness SLAs, and an owner, versioned in git, checked in CI, enforced against producers at the boundary. In practice the enforcement point is a schema registry checking compatibility on every publish, with the compatibility mode, backward, forward, or full, chosen per topic and treated as part of the contract rather than as a default nobody read. The uncomfortable truth contracts encode is organizational, not technical: the producing team ships daily and is not thinking about your pipeline, nor should they have to be. The contract is the treaty that lets them move fast without silently breaking you, the same role an API holds between services. When a schema change arrives as a reviewed pull request instead of a 3 a.m. parse error, the platform has grown up. Add lineage, the recorded graph of which datasets feed which, and the 3 a.m. question changes from "who do I even page" to "this gold table is wrong, its silver parent changed at 02:14, here is the commit."

10 / ReplayBackfills are the real test of a pipeline

Here is the question that separates a demo from a platform, and it is the question to ask of any pipeline design, including your own: the transform had a bug for three weeks; walk me through the fix. Not the code fix. The data fix. Three weeks of silver and gold tables, dashboards, exports, and model features are wrong, and they must become right without taking the platform down or double-counting a single event.

Everything built so far was secretly preparing for this moment. The lake retained raw history (07, 08), so the correct inputs still exist. The log keeps events replayable by offset and the table format can time-travel to any snapshot (04, 08), so both the stream and the batch worlds can be rewound. And every transform was written idempotently (05), so running it again produces the same result instead of a duplicate. Replayability is not a feature you add. It is a property that emerges if, and only if, three disciplines were kept: retain the raw, key every write, make every run idempotent. Miss any one and the backfill becomes archaeology with a deadline.

The mechanical pattern that makes backfills routine instead of terrifying is partition-scoped, deterministic runs. Each run of a transform owns exactly one partition of output, usually a day, computes it entirely from inputs, and overwrites it atomically. The run for July 3 produces July 3's rows, all of them, every time, no matter how many times it runs. A three-week backfill is then just twenty-one ordinary runs executed with old parameters, indistinguishable from the nightly runs except for the date they are handed. Contrast the anti-pattern, the append-only transform that adds "new" rows each run: it cannot be re-run without duplicating, cannot be repaired without manual surgery, and every backfill against it is a small novel written in DELETE statements.

One honest architectural note. The old answer to "streaming for freshness, batch for correctness" was the lambda architecture: run both, a speed layer and a batch layer, and merge at query time. It works and it doubles everything: two codebases computing each metric, which section 07 already told you is how a metric gets defined zero times. The kappa answer keeps one path, the stream, and handles reprocessing by replaying the log from an earlier offset through the same code. Modern platforms mostly land pragmatically in between: one set of definitions, executed by engines in both modes, with the lake as the shared replayable substrate. The principle to keep is not a Greek letter. It is single definition, replayable inputs, idempotent outputs.

11 / The ConductorOrchestration, retries, and the day the consumer falls behind

Dozens of transforms now depend on each other: silver waits on bronze, gold on silver, the ML features on three golds and an external feed. Something must know the graph, run things in order, retry the failures, and page a human when retrying stops helping. That something is the orchestrator, Airflow and Dagster being the usual names, and the essential thing it manages is not schedules but dependencies: the DAG, run the thing when its inputs are ready, not when the clock says so. The modern refinement is to make the orchestrator think in datasets rather than tasks, "materialize gold.revenue when silver.orders lands," which is the same shift the whole field keeps making: from imperative steps to declared outcomes.

Two of the conductor's habits do most of the protecting. Retries with exponential backoff and jitter, because most failures are transient and the worst response to a struggling dependency is a thousand synchronized retry hammers arriving in lockstep; the jitter is not decoration, it is what prevents your own recovery from being a self-inflicted stampede. And idempotent tasks, section 05 for the third time, because a retry is by definition a rerun, and reruns are only safe if running twice is boring. An orchestrator wrapped around non-idempotent tasks is an incident generator with a nice UI.

The streaming side has its own version of falling behind, and it deserves its own instrument. Consumer lag, how far behind the log's head each consumer sits, is the single most honest health metric a streaming platform has, because it is the integral of every other problem: slow code, undersized fleet, hot partition, downstream database groaning. Lag has a brutal arithmetic that the widget below makes visceral: if consumers process even slightly slower than producers produce, lag does not stabilize, it grows without bound, and freshness quietly transforms from seconds into hours while every individual component reports green.

When lag grows and consumers cannot be scaled past the partition ceiling from section 04, the remaining lever is backpressure: slow the producers, shed load explicitly, or degrade freshness deliberately for the datasets that tolerate it. The immature platform discovers backpressure as an outage. The mature one designed the choice in advance, per dataset, using the same freshness dial from section 01, because the forces never changed. They just arrived wearing operational clothes.

12 / The CapstoneWe build the clickstream platform, and every force shows up

Now we run the frame end to end on a real and famous problem: the data platform for a large e-commerce product. Clickstream, orders, catalog. Analytics wants dashboards, the ML team wants features for a real-time recommender, finance wants numbers that survive an audit. This is the payoff: no new tricks, just the forces we named arriving exactly when the pressures appear.

Requirements. Functional is short: capture user activity and order events, serve modeled analytics, and serve fresh features to the recommender. Then we ask the Frame's questions and get honest numbers to design against.

Notice the very first decision is the freshness dial from section 01, and it is set twice. The recommender is harmed by staleness measured in minutes, so its path earns streaming. Finance is harmed by incorrectness, not staleness, so its path earns nightly batch with population checks and an audit trail. One platform, two notches, each justified by a named harm. Anyone proposing a single global answer has not asked who consumes the data.

Core entities. A user, a session, an event, an order, a product. The events split immediately into two species that want different treatment: the clickstream, enormous, append-only, tolerant of tiny loss, and the order events, small, sacred, tolerating nothing. We keep them on separate topics with separate contracts from the first sketch, because pretending they are one stream is the mistake that haunts the rest of the design.

High level design, then the deep dives break it. Naive first, exactly as the method demands: apps write events to the log, one job loads the warehouse, dashboards read it. Correct, and fragile in every way this manual has named. Walk the path below and watch each earlier section arrive to relieve its pressure.

Now the deep dives, each one a callback. Ingestion: clicks are fired at the log directly with client-generated event IDs, because at-least-once retries from flaky mobile networks are a certainty and dedup needs a key (05). Orders leave the OLTP database through an outbox relayed by CDC, never through dual writes (03). The two topics get different partition counts sized from the arithmetic in the Frame: 5,000 events per second peak across 64 partitions is a comfortable 80 per second per partition, with parallelism headroom for the consumers (00, 04), keyed by user so per-user ordering holds where sessionization needs it.

The streaming tier validates every record against the contracts at the boundary and routes rejects to an alarmed, owned DLQ (09). It deduplicates on event ID, sessionizes clicks with event-time windows and a 90-second lateness budget chosen by measuring real arrival skew rather than by guessing (06), and maintains the recommender's features, "views in last 30 minutes," "category affinity," in a store the model reads at inference. The features are fresh within a minute, which is what the harm analysis demanded, and no fresher, because fresher costs more and helps nobody.

The lake and warehouse tier sinks every validated event to Iceberg tables, compacted on schedule because a 64-partition streaming writer is a small-files factory (08). Nightly, the orchestrator materializes the trust gradient, bronze to silver to gold, every transform partition-scoped and idempotent so any day can be recomputed by rerunning it (07, 10, 11). Population checks close each day's partition: volume against seasonal baseline, completeness against the order counts CDC reported, referential integrity between orders and products (09). Finance reads gold, and when the audit asks what the revenue table said on July 1, time travel answers in one query (08).

And the test that matters: three weeks from now, someone finds a sessionization bug. The fix is a code change, a replay of the affected offsets through the corrected job into a shadow table, twenty-one partition-scoped reruns of the downstream models, and an atomic swap. No archaeology, no DELETE novels, no double counting, because raw was retained, writes were keyed, and reruns were boring by construction (10). That sentence, more than any diagram, is what a production-grade pipeline means.

The level ladder, said out loud because it is what the room actually measures: a mid-level engineer reaches the log and the warehouse with some prompting. A senior separates the two freshness paths unprompted, insists on the outbox, and sizes partitions from arithmetic. A staff engineer designs the backfill story first, gets contracts and DLQ ownership into the first sketch, and can tell you which datasets are allowed to be wrong, for how long, and who is harmed when they are. The difference was never the tool list. It is how early, and how independently, the right guarantee surfaces.

13 / The Meta LessonWhat you actually carry out of here

You will not be asked to recite Kafka configuration, on the job or in the interview room. You will be handed a pressure, a lagging consumer, a wrong dashboard, a three-week-old bug, a new team that wants "real-time," and asked which valve relieves it, and why. Three habits separate the people who can from the people who memorized a logo wall.

First, they set the dials per dataset, not per platform. Freshness is chosen one consumer at a time. Delivery semantics are chosen per topic. Lateness budgets are chosen per window, from measured skew. The lazy answer picks one global setting and applies it everywhere; the strong answer says "this dataset needs seconds and here is the harm, this one needs correctness and here is the audit," and means both halves.

Second, they trace every component back to the force that justifies it. If you cannot say what pressure a box relieves, the box should not be on the board. CDC answers "the copy misses deletes and melts the source." The log answers "many readers, many speeds, decoupled in time." The watermark answers "the window must close while events still straggle." Compaction answers "streaming writers and batch readers want opposite file sizes." When you can narrate the why for every node, the design defends itself.

Third, they respect boring solutions, which is the rarest habit of the three. A cron job with incremental extraction serves an astonishing share of real businesses honestly. A single Postgres, well fed and well indexed, carries more companies than Kafka does. The instinct to reach for the streaming stack before a named freshness harm exists is the same instinct that builds a platform nobody can operate at 3 a.m., and every box you add is a box that can page someone. Add complexity when a pressure forces it, and not one component sooner.

The whole manual as a lookup table

Every force in this piece, paired with the smallest move that relieves it. If you keep nothing else, keep the shape: a pressure on the left, a deliberate move on the right, and a reason connecting them.

Good pipeline design is choosing the right guarantee for the dataset in front of you. Not reciting the catalog. Naming what the data actually needs, freshness, completeness, replayability, and then deciding deliberately, out loud, with the harm analysis attached. The vendors will rotate. The forces will not.

See you in the trenches.