Data Pipelines
Kitchen table
Water reaches your tap through pipes you never see, cleaned and pumped along the way. Information travels the same way inside a company, from the shop tills and the website to the place where the learning machines drink. The pipes have to run every night without anyone watching, and when one leaks or clogs, everything downstream goes wrong at breakfast. The machines get blamed, but it is usually the plumbing.
Whiteboard
Batch and streaming, orchestrators and retries, schema drift, late-arriving data, backfills that rewrite history, and idempotency so a rerun does not double-count the world. The unfashionable truth: pipeline failures cause more AI incidents than models do, and the worst ones are silent. The job that succeeds while quietly processing garbage. Data contracts and freshness checks at the boundaries buy more reliability per hour of effort than anything you will do to the model itself.
Boardroom
This is the plumbing, and it obeys plumbing economics: invisible when it works, catastrophic when it does not, and never celebrated at the launch party. Underfund it and every AI initiative downstream inherits the mess – stale dashboards, models trained on wrong numbers, decisions made on data that quietly stopped arriving in March. When AI budgets are set, the unglamorous line item for data engineering is the one protecting all the glamorous ones.
Related concepts