Shift-left data quality: Which data sources matter most

Key takeaways

• Warehouse-only data quality has a coverage problem. A large share of enterprise data never lands in Snowflake or Databricks, and it’s still doing work: people read it in the CRM, and agents increasingly reach it through operational APIs.
• The blocker on shifting left is whether the rule that runs at the edge is the same governed rule the business authored, or a reimplementation that quietly drifts.

Data quality moved to the cloud. Most of your data didn’t.

The last decade of modernization had one center of gravity: get everything into the platform. Data quality followed, and now most enterprise DQ investment sits on top of Snowflake or Databricks. A rational place to put it. It’s just not where all the data is. A great deal of it is created, read, and acted on in the CRM, the ERP, the on-prem database, and never makes the trip. Agents are hitting those systems directly now too, with no analyst in the loop to notice a field looks off.

The standard pitch for shift-left data quality is economic: the further downstream you find an error, the more expensive it is to fix. Run a check at the point of entry, and you can block the write: the bad record never exists. Run the same logic on a partner file at arrival, and you can refuse the load: the bad record exists, but not in your systems. Run it in the pipeline between bronze and silver, and you can quarantine: the bad record is in your estate, and you’re containing a blast radius. Run it in a BI dashboard, and you can raise a ticket.

The coverage gap is just as concrete. Sales reps read account records directly in the CRM. Service agents work open cases in ServiceNow. Procurement lives in SAP. None of that waits for a dbt run, and agents now call those same operational APIs at machine speed. If your data quality program lives exclusively in the warehouse, you have no coverage of the systems where most decisions get made.

1. Operational applications, where prevention is possible

Salesforce, SAP, Dynamics. This is the only tier where you can stop bad data from existing, and it’s the tier most DQ programs skip entirely.

What you can do here is narrow but powerful: single-record logic evaluated against the schema at the moment of entry. Format validation. Mandatory field enforcement. A reference data lookup: is this a real country code, a real industry classification, a real active supplier. Fast enough to sit inside a form submission, because nothing here needs a population scan.

The native platforms all offer some version of this. Salesforce has validation rules; SAP has its own configuration. The problem isn’t capability; it’s governance. Those rules live inside each application, authored by whoever administers that application, versioned separately if they’re versioned at all. The definition of an “active supplier” in your CRM validation rule and the definition in your warehouse quality check will diverge, and nobody will notice until they produce different numbers in the same meeting.

This is the specific gap Ataccama’s DQ Firewall is built for. You bundle rules from the central library, the same rules authored in Ataccama ONE by the people who own the definitions, and expose them as a REST service that front-end applications call before they commit a write. The rule library stops being documentation and starts being enforced. Where a REST call doesn’t fit the architecture, DQ Gates’ Python support gives you the same logic embeddable in an AWS Lambda or invoked from Apex.

Mastering belongs in this conversation too, and it’s usually treated as a separate discipline. MDM consolidates data from multiple operational sources into the record everything downstream depends on. That makes it a data quality decision point, not just a deduplication utility. Ataccama’s MDM evaluates quality first and then uses those results in matching and merging, so when two candidate records conflict, the survivorship logic favors the higher-quality one rather than the most recent or the alphabetically first. Whichever record wins that comparison becomes true for every consumer of the master record.

Prevention has a hard ceiling, though. Anything requiring population context, like profiling or distribution drift, cannot run at the point of entry. You don’t have the population in scope. Those checks belong elsewhere.

2. Transactional databases: MSSQL, PostgreSQL, Oracle, DB2

Connectors let you profile and catalog these systems in place, where you discover what nobody documented: the status column with fourteen distinct values in a field the data dictionary says is boolean, the date field that’s been storing ‘1900-01-01’ as a null sentinel since a migration in 2016.

These systems often live on-premises, so secured connections and edge processing matter. Rules run behind the firewall without exposing data over the public web. The same governed rule executes locally at the edge for the on-prem source and via pushdown or cloud processing for everything else, and the steward who authored it never needs to know which runtime picked it up.

The other approach is DQ Gates, which place checks in the pipeline between the transactional database and the cloud warehouse, wired into the orchestration tool you already run. That covers data on its way to the warehouse.

3. Files and partner feeds

You don’t control the producer. You can’t validate at entry because entry happened in someone else’s system, on someone else’s schedule, according to someone else’s interpretation of the spec. Arrival is the only point of control you will ever get.

So the checks are structural: schema conformance, expected column set, encoding, reference data conformance on the fields that matter. The real decision is a tolerance call. Reject the entire file, or accept it and quarantine the failing rows. 

4. Cloud data warehouses (Snowflake, Databricks) 

Snowflake and Databricks are where most organizations start, and there’s nothing wrong with the checks that run there. Bronze to silver is a good gate: the data has landed, you have full population context, and you can catch drift, distribution shifts, and cross-source inconsistencies that no point-of-entry check could see. DQ Gates exist precisely to make that gate enforceable inside the pipeline.

Enabling shift-left data quality 

Everything above is deployable today. Pull your most frequent data quality incidents. For each one, ask where it could have been stopped. You’ll likely discover that a meaningful share were single-record defects that a validation call at entry would have blocked outright, and that you’ve been investing in detection at the tier where the only remaining option was to write a ticket. That mapping is the whole planning exercise. It tells you which sources matter most for your specific failure modes.

Then ask the harder question. If you placed a check at each of those points tomorrow, would they be the same rule, governed, versioned, and owned by the person who understands what it means, or four copies that start drifting on day one? If it’s the latter, you don’t have a placement problem. You have an authorship problem, and placing more checks will only multiply it.

The only version of shift-left that survives contact with a real enterprise separates authorship from execution. Rules get defined once, by the people who own the meaning: stewards and business analysts. Then they package it and delegate it to engineers to place wherever the leverage is. A senior data engineer can implement any rule you can specify. Knowing that a claim with a reserve change in the last 30 days needs different completeness treatment than one without is a different job, and that knowledge lives in the claims team.

See how one governed data quality rule can be enforced from the point of entry through the cloud. Talk with our team about where shifting quality checks upstream could prevent problems before they spread.