The status quo for building data pipelines is painful and cumbersome. Sure, building the first couple isn’t too bad…it’s the many that follow. And how they impact each other. And how they become harder and harder to maintain as the dependencies grow and the codebase becomes increasingly unruly. For us, this was a terminal path for our happiness as engineers.

To solve this, there are two options: Frameworks and Control Systems. Most other companies went the framework route. They are great at making it easier to build at a static point-in-time. But they lack a feedback loop, which is critical for dynamic systems.

Dynamic systems, like data pipelines, are where control planes shine. Instead of the “when X happens, do Y” framework mentality, a