Most data pipeline architecture diagrams show data moving. They rarely show whether that data can be trusted once it arrives. A pipeline that reliably moves a corrupted field from source to dashboard is not a success. It is a fast, well-orchestrated way to make a bad decision.

That gap is widely felt: in a 2025 Precisely report, 67% of respondents said they don't completely trust the data they use for decision-making (up from 55% a year earlier), with data quality named the top data-integrity challenge.

This guide covers five core patterns — ETL, ELT, streaming, Lambda, and data mesh — organized around the question most guides skip: for each one, where does quality break, and who has to catch it before it lands? You'll also find the components every pipeline shares, a diagram per pattern, the failure modes that recur in production, and the design practices that hold up at scale.

What Is Data Pipeline Architecture?

Data pipeline architecture is the structural design of how data moves from its source systems to the destinations where it's consumed, and everything that happens to it along the way: extraction, transformation, validation, storage, and delivery. It defines not just the path data takes, but the shape of that path: which stages run in sequence, which run in parallel, where data is stored at rest, and where quality gets enforced.

This matters because the architecture you choose determines where problems surface and how expensive they are to fix. A schema change in an ELT pipeline shows up differently from the same change in a streaming pipeline, and a design that looks clean on a whiteboard can still leave you with silent data loss at ingestion if quality isn't built into the architecture itself rather than bolted on afterwards.

Core Components of a Data Pipeline

Before comparing patterns, it helps to know the parts they're built from. Whatever shape a pipeline takes, the same core components show up in every one — and the quality/observability layer is the one most teams skip.

Data Sources

Databases, SaaS applications, event streams, flat files, and APIs. The diversity of source types is usually the first layer of complexity in any pipeline: each source has its own schema conventions, update frequency, and failure behavior.

In practice, that means relational databases like PostgreSQL, MySQL, or Oracle sitting next to SaaS APIs like Salesforce, HubSpot, and Stripe, plus event streams and flat-file exports — each with its own authentication, rate limits, update frequency, and format.

What can go wrong: a design that assumes uniformity across sources breaks the moment a new source type is added. A transactional database might support change data capture while a SaaS API throttles requests and returns paginated JSON whose field names drift between versions, and a CSV can arrive with no schema at all.

Ingestion Layer

The ingestion layer moves data out of source systems and into the pipeline, whether through full batch loads or change data capture (CDC), on a schedule or continuously via streaming.

Tools like Fivetran, Airbyte, and Kafka Connect handle much of this today.

Change data capture (CDC) sits between batch and streaming: instead of polling on a schedule, it reads the source database's transaction log and captures every insert, update, and delete as it happens — near-real-time replication without a full streaming stack.

The tooling splits along need: Fivetran for managed reliability and a broad connector library with minimal config, Airbyte when you want self-hosting and custom sources, and Kafka Connect with Debezium for high-volume, log-based CDC streamed into and out of Kafka topics.

What can go wrong: schema drift. A source system renaming or dropping a column upstream is invisible to the pipeline unless ingestion explicitly checks for it.

Processing & Transformation

This is where raw data becomes usable: cleaning, deduplication, joins, and business logic all happen here. It's also the highest-risk zone for hidden quality issues.

What can go wrong: a join that unintentionally drops rows or a transformation that miscalculates a metric produces output that looks structurally correct. Nothing fails. The numbers are just wrong.

Storage & Destinations

Warehouses, data lakes, feature stores, and serving databases all sit at the far end of the pipeline. Destination requirements should inform upstream design, not the other way around: if the destination is a feature store for a real-time model, that constrains how fresh and how structured the data needs to be well before it ever reaches storage.

Between the warehouse and the lake sits the lakehouse — Databricks Delta Lake or Apache Iceberg — pairing a lake's low-cost storage with warehouse-grade query performance. And for machine learning, feature stores like Feast, Tecton, and Hopsworks serve pre-computed features at low latency so a model sees the same values in training and in production.

What can go wrong: choosing the wrong destination is one of the most common reasons a pipeline gets rebuilt. A warehouse tuned for batch BI can't serve a real-time model, and a raw lake pushes the cleanup cost onto every downstream consumer.

Orchestration

Orchestration tools (Airflow, Dagster, Prefect) handle scheduling, dependency management, retries, and alerting across every other component.

Airflow is the default: its DAG-based model and large ecosystem made it the most widely adopted open-source orchestrator, with 320 million downloads in 2024 — roughly ten times its nearest competitor. Prefect takes a more cloud-native path built around dynamic flows and runtime control, while Dagster is asset-first, shaping pipelines around the data they produce rather than the tasks they run — a model that maps naturally onto dbt and gives lineage across assets.

What can go wrong: without solid orchestration, a pipeline doesn't fail cleanly. It fails inconsistently, with stages running out of order or retrying into a broken state.

Data Quality & Observability Layer

This is the component that catches the failures the others introduce. It determines whether everything upstream can actually be trusted. It validates data at each stage in the form of quality checks, detects anomalies, and traces lineage, so a downstream issue can be root-caused back to its source.

Read more about this layer below in Where Data Quality Fits in Your Pipeline Architecture.

The 5 Core Data Pipeline Architecture Patterns

Most data pipeline architecture in production today builds on one of five patterns. They aren't exhaustive — Kappa, micro-batch, and reverse ETL all have their place — but these five cover the decisions most teams actually face. Picking the right one is less about which is "best" and more about which failure modes you're willing to own.

If you're choosing between these, four axes decide it faster than any of the descriptions below:

• latency tolerance (seconds vs. hours)

• reprocessing cost (how painful is a backfill?)

• consistency requirements (exactly-once vs. good-enough)

• team surface area (how many codebases can you actually staff?)

Every pattern below is a different point on those four axes.

1. ETL (Extract, Transform, Load)

ETL is the classic pattern, with roots in on-premises data warehousing: data is extracted from source systems, transformed in a separate processing layer, and only then loaded into the destination warehouse.

A retail chain consolidating nightly point-of-sale data into an on-prem warehouse for next-morning reporting is a textbook ETL use case.

Use it when transformation logic is stable, compliance requires data to be cleaned before it lands, or your warehouse has limited compute.

The tradeoff is brittleness: because transformation happens before load, any upstream schema change or edge case forces a change to the transformation code itself, and reprocessing historical data means re-running the whole pipeline.

Where the checks belong: schema and freshness at extract, a row-count reconciliation after transform (output against source, net of any intended aggregation or filtering), and a blocking data contract scan before the load — so unclean or incomplete data never lands in the warehouse.

2. ELT (Extract, Load, Transform)

ELT flips the order above — raw data lands in the warehouse first, and transformation happens afterward using the warehouse's own compute.

This is the cloud-native default for a reason. It's why ELT dominates modern stacks: warehouses like Snowflake and BigQuery made compute cheap and elastic, so there's no reason to transform before load when you can transform after, in parallel, at warehouse scale.

A SaaS company loading raw event data from Segment into Snowflake, then modeling it with dbt, is a standard ELT setup.

The quality risk shifts to ingestion: because raw, unvalidated data lands directly in the warehouse, a schema change or malformed batch can sit there silently until a downstream model breaks.

Where the checks belong: at every handoff (ingestion, post-raw-load, and post-transform), not only at the end. A single check at the serving layer only tells you something is wrong after it has already propagated through every earlier stage.

3. Streaming (Event-Driven)

Streaming architectures process data continuously as events occur, rather than in scheduled batches, typically using tools like Kafka for event transport and Flink or Spark Streaming for processing.

This is the right pattern for fraud detection systems that need to flag a suspicious transaction within milliseconds, IoT platforms ingesting continuous sensor telemetry, or live operational dashboards.

The tradeoff is complexity: streaming systems are harder to test, harder to reprocess after a bug, and failure modes (a dropped event, a late-arriving record) often don't announce themselves the way a failed batch job does.

Where the checks belong: a schema registry on the topic, volume and distribution monitoring on the stream itself, and freshness or lag SLOs on the sink. The registry catches a structural break, but a producer that quietly changes what a field means (units, encoding, the definition behind an ID) passes the schema check and just produces increasingly wrong output until someone notices the dashboard looks off.

4. Lambda (Batch + Speed Layer)

Lambda architecture runs two parallel paths: a batch layer that recomputes complete, accurate results on a schedule, and a speed layer that processes the same data in real time for immediate (if slightly less precise) results. The two are merged at serving time.

This pattern shows up in ad-tech platforms that need both real-time bidding metrics and end-of-day reconciled totals.

The cost is operational: you're maintaining two codebases that need to produce reconcilable results, which roughly doubles the maintenance surface and creates its own reconciliation problem between the two paths.

Where the checks belong: reconciliation between the batch and speed outputs, at the point where the serving layer merges them. And before committing to two parallel paths, check whether a single stream-processing engine can also cover your reprocessing and accuracy needs — a Kappa architecture (streaming only, with reprocessing handled by replaying the event log) is often simpler than keeping a separate batch and speed layer in agreement.

5. Data Mesh

A data mesh distributes data ownership across domain teams instead of centralizing it in one platform team.

Ownership here is about the outcome — the data products other teams depend on — not just the pipeline. Each domain documents its data and guarantees its freshness, stability, and quality against standards set by both its consumers and a central governance team — responsibilities that, before data mesh, tended to fall to no one in particular.

It scales better than a central model because no central team can know every dataset intimately, whereas a domain owns a slice that maps to the part of the business it already understands.

Without shared quality standards enforced across domains, though, a data mesh becomes a data swamp: distributed ownership without distributed accountability just means more places for bad data to originate.

Where the checks belong: at every domain's output, not a central gate. Each domain enforces a data contract on the products it publishes (schema, freshness, and the quality thresholds its consumers agreed to), and federated governance keeps those standards consistent across domains, so autonomy stays accountable.

See how Soda approaches data quality across a data mesh.

Common Data Pipeline Challenges

Data volume spikes: a pipeline sized for average load breaks at peak.

→ Design for horizontal scalability from the outset, and test with realistic peak-load scenarios such as Black Friday traffic, month-end close, or a marketing campaign launch, rather than assuming average-day volume represents worst-case load.

Backfilling and late data: historical reprocessing is rarely considered in the initial design, but it's almost always required eventually.

→ In streaming pipelines specifically, late-arriving events corrupt aggregations if the pipeline doesn't explicitly account for out-of-order arrival.

Quality debt: teams that skip quality checks early end up maintaining an untrustworthy data layer indefinitely.

→ Retrofitting quality enforcement into a mature, already-in-production pipeline costs far more than building it in from the start, both in engineering time and in the accumulated damage from decisions made on bad data along the way.

Data Pipeline Design Best Practices

Design for failure from the start: build idempotent operations so reruns don't duplicate data, checkpoint recovery so a failed job can resume rather than restart, and retry logic with backoff so transient failures don't cascade into bigger ones.

Decouple pipeline stages: use message queues between stages, persistent intermediate storage, and clearly defined interfaces so a change in one stage doesn't ripple unpredictably into the next.

Validate data at every stage, not just at the end: the cost of an undetected issue compounds the further downstream it travels. Data contracts are the enforcement mechanism that makes this practical: instead of scattered, ad hoc checks, a contract defines what "good" means for a dataset and validates it automatically at each handoff.

Build in observability, not just monitoring: lineage, record-level diagnostics, and historical baselines let you catch the anomalies nobody thought to write an explicit check for. Monitoring tells you a job ran; observability tells you whether what it produced actually makes sense.

Treat pipeline infrastructure as code: version control your configs, dbt models, Soda Core checks, and Terraform infrastructure. This is what enables review, rollback, and audit, the same discipline you'd expect from application code.

Design for modularity and incremental change: build replaceable stages, use semantic versioning for pipeline components, and start simple. A pipeline you can change one stage of without touching the rest ages far better than a monolith.

Document lineage and ownership: automated lineage makes impact analysis possible when something breaks, and named owners for every pipeline stage and data product mean a failure has someone accountable for fixing it, rather than becoming everyone's problem and no one's job.

Where Data Quality Fits in Your Pipeline Architecture

The market already knows where the gap is: in the same Precisely report, the top barrier to high-quality data was inadequate tooling for automating it (49%) — exactly what a quality layer that runs inside the pipeline is built to close.

Soda sits in the Data Quality & Observability layer of your pipeline architecture. It works as two layers that catch different kinds of failures:

Data Contracts

A data contract defines what "good" means for a dataset (its schema, freshness, allowed values, and row-count expectations) in the form of machine-readable checks. It can be enforced at each handoff, so a producer can't quietly break a consumer downstream. These are known expectations, checked against landed data, that block or warn before bad data moves on.

See the example Soda contract for a transactions table below (YAML file, written in Soda contract language):

dataset: postgres_ds/finance_db/public/transactions

checks:
  - schema:
  - row_count:
      threshold:
        must_be_greater_than: 0
  - freshness:
      column: transaction_date
      threshold:
        unit: hour
        must_be_less_than: 24

columns:
  - name: transaction_id
    data_type: varchar
    checks:
      - missing:
      - duplicate:
  - name: amount
    data_type: decimal
    checks:
      - missing:
      - invalid:
          valid_min: 0
  - name: currency
    data_type: varchar
    checks:
      - invalid:
          valid_values: ['USD', 'EUR', 'GBP']
  - name: transaction_date
    data_type

Data Observability

Continuous observability watches volume, freshness, schema, and distribution for the anomalies no one thought to write a rule for. These monitors continuously assess the health of your data throughout its lifecycle. It focuses on analyzing metadata, metrics, and logs to detect issues as soon as they arise.

Soda integrates directly with Airflow, dbt, Databricks, and Snowflake, so contracts and monitors run as a step in the pipeline rather than alongside it as a separate process someone has to remember to run.

For a deeper walkthrough on how to implement this quality layer, watch the demo on How To Monitor Data Quality in a Databricks Unity Catalog or book a demo to walk through it on your own stack.

Frequently Asked Questions