Bringing clarity to complex data flows

Working with engineering, we introduced operations within stages. Stages represented the broader structure of the data flow (joining, combining or transforming inputs) while operations contained the detailed work.

This collapsed the pipeline into a manageable sequence. Users could see its overall structure, then open a stage to work on individual operations.

This was progressive disclosure, a principle I had learnt through wayfinding: help people orient themselves, then reveal the detail they need at the point they need it.

The model also made development practical. It allowed us to build the web application rapidly without getting caught up in complicated flowchart engineering, for which suitable web-based libraries were not available at the time.

Another pain point we identified was keeping track of data lineage through pipelines. I proposed connecting lines between stages to help users follow data through a pipeline. Initially, I envisioned user-defined colour groups to denote functions, but the feedback we got was that these would add configuration overhead. Instead, we automatically cycled colours through a spectrum to distinguish connections.

As clients chained pipelines together, we found ourselves explaining the connections through Miro flowcharts. This was a desire line: a view we repeatedly created outside the product.

In 2026, we built a flowchart view into Quantemplate, giving users an overview across pipelines alongside the detailed stage editor.

Pipelines glow as they run, along with their corresponding connecting lines in the flowchart. This makes execution visible in the context of the wider workflow. We recorded an uptick in engagement during demos and training sessions as the data flow through a pipeline became tangible.