The Situation

Organizations struggle with the quality of their data in many different ways, clearly to cover their own requirements as efficiently as possible while respecting the limitations and business processes of their environment. Still, there are some similarities to be identified, which involves starting with one-off / ad hoc activities, continuing with permanent but reactive measures, and then shifting to a more proactive approach.

Once the organization comes to the point where batch data cleansing is simply not sufficient, the time is at hand for DQ Firewall—a solution that extends the current cleansing and matching processes with a prevention component that will not allow users to create new data quality issues in the systems.

The Solution

A DQ Firewall solution simply prevents incorrect, incomplete, or duplicate data from entering the systems. The solution actually interacts with the business users, alerting them about data issues, giving them proposals, or suggesting data corrections. All of these are approved by the users before storing the values in the database.

From the technical perspective, DQ Firewall is a set of tailor-designed web services published at one location, using many systems and applications.

The most common online services include:

  • Online address validation. Checks and standardizes the whole input address in an external/official address register
  • Online address lookup. Proposes matching addresses from an external/official address register based on partial address information
  • Company information lookup. Enriches company-client information from external reference data
  • Contact details standardization. Transforms phone numbers, email addresses, websites, etc., to a standardized format, checks domains and phone number prefixes, etc.
  • Client identification. Checks existence of the new client in the current customer database to eliminate creating duplicates
  • WebLookup. Enriches records with data obtained from internet URLs, suitable for cases where a full reference data database is not available

In general, any data cleansing process can be easily turned into an online web service.

The major deliverable is a complex solution that impedes incorrect data at the point of entry. Given the services provided, a DQ Firewall typically acts as a crucial component of unified front-end applications and is also used by multiple existing applications. One of the major advantages is the centralized approach, i.e., all data cleansing, data validation, data identification, and data enrichment services are maintained in one location.

The solution is usually accompanied by a Manual Data Quality Issues Handling process (to deal with the exceptions) and a Data Quality Monitoring and Reporting component.

Ataccama supports the solution

Ataccama provides both the technology (specialized algorithms, predefined data cleansing & standardization plans ready to be adapted to customer requirements) and also the business logic (knowledge bases, country-specific cleansing and standardization rules, and the best practices in this area).

The technology is designed to handle high-performance requirements and deliver sub-second web services response time.

The Benefits

  • Prevention of major/most data quality issues at the point of entry
  • Streamlining (simplification and acceleration) of data entry processes
  • Online data validation and enrichment
  • Elimination of duplicates (identification services)
  • Integration of web services business logic at one location