How to decide where data quality belongs in the data value chain

A practical framework for placing data quality controls along the chain, from the point a record is created to the moment a model consumes it.

A contact record gets created in your CRM with the account region left blank. Within minutes it has synced to marketing automation, posted to the billing system, fed the customer 360 in the lakehouse, and landed in the feature store behind a churn model. By the time a data quality job in the warehouse flags the blank field that night, the same record is wrong in five places, the campaign has mis-targeted the account, and the model has scored on it.

That is the core problem with running data quality only at the end of the journey. The fix is no longer one update in the CRM. It is a correction in every system the record reached, plus an attempt to walk back the decisions made on it before anyone noticed. This is the sprawl that makes a late catch so expensive, and it widens with every hop the data takes.

Shifting data quality left addresses this by moving validation closer to where data is created. The harder question, and the one this piece is about, is how far left to move each check, because not every check belongs at the source.

What is the data value chain?

The data value chain is the full path a record travels, from the moment it enters a system of record to the moment someone reports on it or a model consumes it. Mature data teams map it deliberately. One useful framing runs from the second a contact is created in a CRM to the report that answers “who are our primary contacts across our key accounts,” counting every hop in between.

Each hop is a place where quality can degrade, and a place where you can validate it. So the real decision is rarely whether to run data quality checks. It is where along the value chain it makes sense to deploy them.

Prevention controls stop bad data before it spreads. Detection monitors what has already landed. The cost of a fix grows as data moves right.

What is shift-left data quality?

Shift-left data quality is the practice of validating data earlier in its lifecycle, closer to where it is created, rather than waiting until it has landed in a warehouse or lakehouse. The goal is to catch and stop errors before they propagate to downstream systems, models, and reports. Where traditional data quality watches data after it settles, a shift-left approach moves at least some of that validation upstream, toward the pipeline and the source.

Why did data quality traditionally run at rest?

For most of the past decade, data quality lived on the right side of the chain. You landed raw data, profiled it, wrote rules against the tables, and monitored for anomalies. For batch analytics that was a sound design, because a nightly load gave you hours to find and fix errors before the morning dashboards had to be right.

Two shifts changed the economics.

• Data estates fragmented. The same record now lives in more systems sooner, so a late catch is a more expensive catch. Every additional hop a bad value reaches is another system you have to clean up.
• AI consumption moved to real time. A feature pipeline that computes off a record the moment it is created gives you no overnight window. The model scores on the data it has available, so detection after the fact is detection too late.

That overnight window is the assumption the warehouse-based approach was built on, and real-time AI has taken it away. When a fraud model scores a transaction the instant it is created, or a recommendation engine reads a profile attribute the moment it changes, there is no batch cycle in which a nightly check could have caught the problem first. The model returns a decision on whatever the record contained at that moment, correct or not.

Where can you run data quality checks?

Validation can sit in three broad positions along the chain. It helps to be precise about where, because data comes to rest in two different places, at the source system where a record is created and in the warehouse where it lands, and it is in motion only while it moves between them through pipelines. Each is a place to put a rule.

• In the warehouse, on the right. Rules and monitoring run against tables after data has landed in the warehouse or lakehouse gold layer, where the data now sits at rest. This is where anomaly detection and trend monitoring do their best work, watching for drift and outliers across large volumes. For example, a nightly load lands the day’s orders and a monitor flags that total order value fell by a third, because an upstream change started sending the amount in cents instead of dollars. You catch the pattern across the whole table before the morning revenue dashboard refreshes, although the bad rows have already landed.
• In motion, in the middle. Validation runs inside the data pipeline before data reaches its destination, most often as part of a batch ETL load rather than a latency-sensitive streaming path, where an inline check would add delay. A record that fails can be filtered out, or the load can be stopped, which keeps bad data out of the warehouse in the first place. For example, a batch ETL job that loads the day’s account records checks each one as it moves, holds back the rows with a malformed tax identifier, and lets the clean rows land, so the gold tables and the feature pipelines that read from them never ingest the bad records.
• At the source, on the far left. Validation runs as close to creation as possible, checking a field on a record as it is written into the source system through an API. The data sits at rest in the source just as it does in the warehouse, but here the check fires at the moment of entry, before the record ever moves. Picture a sales rep who enters the country as “USA” when every downstream system expects “United States.” A check at that moment normalizes or rejects the value, so no warehouse job, no report, and no model ever has to reconcile the mismatch later. Caught here, an error never enters the chain at all. This is the most effective position and also the hardest to reach, since it means integrating quality checks into the source applications themselves, but for the highest-stakes data it is the position worth working toward.

Of these three, the middle position is where prevention happens. Monitoring in the warehouse catches problems after data lands, which is the right place for detection but too late to stop the spread. Validating data while it moves, before it lands, is the harder problem and the more valuable one when the goal is prevention. This is what Ataccama’s DQ Gates are for. DQ Gates is a Python SDK that pulls your governed data quality rules out of Ataccama’s control plane, packages them with their metadata, and embeds them in your pipeline code, most commonly a batch ETL job. From there the pipeline acts on the results, either stopping the load when a critical check fails or filtering the invalid records out so that only clean data lands. The rules are the same governed definitions you run as monitors against the landed tables, so you author a check once in Ataccama and enforce it in both places rather than rewriting it for each pipeline. Take a batch pipeline that loads third-party data feeding underwriting decisions. With the checks embedded, a record that fails validation either halts the load or is filtered out, so a bad value never reaches the model that prices the risk.

Should you always shift left?

No. The point of shifting left is to gate the checks that have to prevent or flag a bad record from moving forward, while monitoring the rest. A gate is a blocking control, and it is the right tool when a failed record must not reach a real-time model, a regulated process, or a downstream system that is hard to correct after the fact.

When you only need to know that something looks wrong so a steward can respond, a monitor on the landed data handles that without sitting in the pipeline’s path. Choosing which checks belong in a pipeline and which can run as monitors is as much a sign of a mature practice as knowing how to build them.

A simple way to decide.

• Prevent when propagation is the risk. If letting a bad record move forward would cause damage you cannot easily undo, put a blocking check in motion or at the source.
• Detect when a response is enough. If you mainly need to know that something looks wrong so a steward can act, keep the check in the warehouse.

Most mature environments run both, and match the control to the consequence rather than pushing every check as far left as it will go.

This maps onto how Ataccama splits the work. Data Quality Monitors are the default layer, running your governed checks against landed datasets, surfacing the results, and alerting stewards through Teams, Slack, email, or a Jira ticket when a rule or a table score drops below threshold, without removing any records. DQ Gates handle the smaller set of checks critical enough to enforce inside a pipeline. Most of your checks run as monitors, and only the ones that need to prevent rather than detect move into a pipeline.

How do you keep from maintaining the same rule everywhere?

Shifting left creates a real engineering problem. Once you validate data in three positions instead of one, you risk maintaining three implementations of the same rule. Take a “valid account region” check written in the ETL pipeline, rewritten against the warehouse, and reimplemented at the source. Someone adds a new region code to the warehouse rule for a market launch but never updates the pipeline rule, so the same record now passes one check and fails another, and your quality logic becomes its own source of inconsistency.

The way to avoid this is to author a rule once and enforce it wherever it needs to run, with the rule executing where the data is. For monitoring at rest, that can mean pushing execution down into the warehouse, where Ataccama translates the check into Snowflake’s or Databricks’ native language and runs it on their compute, which improves performance and avoids moving data out of the platform. It can also mean running on edge compute deployed inside your own VPC, so sensitive data stays under your control and never moves into the Ataccama cloud. Either way you keep one definition of what correct means and apply it both in motion and at rest. This is also where a platform that spans the full chain earns its place over a tool that only sees the warehouse, since the point of shifting left is to cover the systems upstream of the warehouse, not the warehouse alone.

Where should you start?

If you are rethinking where data quality belongs, three steps help.

• Map one chain. Pick one important dataset and trace it from the system of entry to every consumption point that matters, including AI and ML pipelines. Most teams find more hops than they expected.
• Rank by consequence. Mark each consumption point by what it costs to run on bad data. A real-time model and a quarterly report sit at very different ends of that scale, and they justify very different controls.
• Place controls to match. Put a blocking control upstream of the high-consequence points and monitoring everywhere else. Then check whether your tooling lets you write each rule once and run it everywhere you marked, or whether you are signing up to maintain the same logic in several places.

Shift-left works best as a targeted practice, placing the right control at the right point so that trusted data becomes the default by the time anyone, or any model, depends on it.

Take a closer look at the Ataccama data quality platform.