Data environments are changing. Companies are foregoing centralized data landscapes in favor of decentralized teams working with new technologies and concepts like data fabric and mesh.
As more actors (users, processes, and systems) can now influence data flows (and data quality), causing data stacks to become more complex and fragile, data quality processes must update and evolve simultaneously. This new reality calls for more self-service features, better automation, and the ability to notice the most minor changes in tables and columns.
New-age, fully automated data quality monitoring – data observability – can help achieve all these goals. However, outdated and legacy systems might not have these capabilities. Manual efforts in data quality simply won't cut it in 2023. Here's why:
1. Slow and manual data quality monitoring
Traditional data quality management can be described in the following process:
- Determining standards that specific data needs to adhere to (by business department or by data domain or as required by analytics teams)
- Documenting requirements
- Coding rules for data sources and warehouses → manually mapping rules to sources/tables
- Fixing invalid data (either at the source when possible or at the consumption point)
On its own, this process is not scalable because it is:
- Centralized
- Requires technical knowledge, such as SQL or Python
Problems arise when business departments request custom checks, data transformations, or even adjusting data quality rules. This can create bottlenecks (slowing down processes and demanding additional effort) proportional to the company's size.
The biggest challenge of manual data quality is the issues it simply cannot prevent: the unknown unknowns. You may be able to write DQ rules for problems you're aware of, but noticing issues you didn't expect can be impossible. These are called "silent issues" and have their own problems.
2. Silent Issues
"Silent issues" is the term we use for problematic data that goes unnoticed in your organization. Silent issues are becoming more prevalent because of complex data landscapes and a lack of end-to-end responsibility for data pipeline health.
Some common types of silent issues in data are:
-
A bug in the ETL or data migration process results in duplicate or partially uploaded data or inconsistent data.
-
Changed schema: a new or deleted column, a change in data type, or a column name changed without being properly logged or communicated.
-
PII gets to systems where it should not be present.
Silent issues are especially problematic because they:
-
Take days or weeks to discover. Due to their unpredictable nature, it can be challenging to track these issues back to their source – or even recognize them to begin with. This can lead to them traveling to downstream systems and causing significant damage.
-
Cause people to lose trust in your data. Maintaining confidence in your data is hard enough when you know where the issues are coming from.
Once you spot a silent issue, you may think: "problem solved." Wrong. The problem with silent issues is that it’s nearly impossible to configure checks for them and anticipate what other problem arises in the future.
The next problem is that even if it is possible to create a check for a newly spotted issue, it might have already caused costly damage. This is called reactive monitoring, another scourge of traditional data quality management.
3. Reactive Monitoring
A company practices reactive monitoring when they only monitor issues they are aware of, either through domain knowledge or responding to data quality issues after they have occurred.
However, addressing data quality monitoring in this way can lead to many problems, such as:
- By the time unexpected issues occur, it may already be too late. Usually, companies notice these issues once the bad data reaches downstream systems, which can cost money (i.e., bad analytics) or cause other significant damage.
- Sometimes you don't know what to check for. You cannot write a data quality check for some issues because they are unpredictable. Proactively monitoring through AI and machine learning is the only effective solution for these issues. Even when you can write effective DQ rules manually, you will still need an expert with extensive knowledge and experience working with data in your business domain. Even then, you can find yourself in a repetitive and non-preventive cycle of discovering an issue, writing a rule, then discovering again.
Overall, the best way to deal with data quality issues is by addressing them before they occur. Something only possible if you have a fully automated solution – like data observability.
Data observability to the rescue: proactive monitoring of your data landscape
Data observability tools comprehensively monitor the state of your data systems and effectively deal with the common pain points of traditional data quality management. Let’s look at the main benefits that data observability brings:
1. Proactive, automated data monitoring
Data observability lets you monitor for issues that you don’t expect, so you don’t have to worry about silent issues with AI-powered monitoring and other automated processes that scan for:
- Volume changes
- Data freshness
- Schema changes
- Data domain monitoring (useful for monitoring new occurrences of sensitive data)
Even when you need to create a new rule to monitor for a newly discovered issue, it will be much easier to scale to other systems through automation.
2. Precise issue localization and faster issue resolution
Data observability makes it possible to find the root causes of issues faster thanks to two essential capabilities:
- Alerts. Instead of taking days or weeks to discover a silent issue, get alerted as soon as something unusual happens. The observability tool will regularly monitor your data and notify you as soon as an unexpected change occurs.
- Data lineage. Because data observability combines data lineage with metadata overlays displaying issues, data engineers can find the root cause of the issue faster.
3. Fast deployment and scaling
Unlike centralized setups, where one IT team takes care of change requests and rule deployments, data observability solutions help distributed help teams monitor their systems and data sources independently.
This is possible thanks to the automated data quality and AI-powered features that require a few clicks to configure.
Conclusion
The state of data management is clear: data leaders are dealing with complex, distributed data landscapes without clear ownership of the end-to-end data pipeline. This context calls for automated and broad monitoring of enterprise data systems, and data observability is the solution.
