Ataccama
  • Platform
    Enterprise Data Quality Fabric
    Enterprise Data Quality Fabric
    Arrow right
    How It Works
    Platform Overview
    Arrow right
    Data Quality
    Data Quality

    Automated DQ checks, monitoring, anomaly detection, and remediation

    Reference Data Management
    Reference Data Management

    Centralized RDM, authoring, hierarchies, and synchronization

    Master Data Management
    Master Data Management

    Multidomain mastering, stewardship, AI matching, flexible data providing

    Data Integration
    Data Integration

    Flexible data extraction, transformation, and providing

    Data Catalog
    Data Catalog

    Automated data discovery, business glossary, and data marketplace

    Data Stories
    Data Stories

    Tell engaging data stories with your data

    Deployment
    Deployment Options Platform as a Service On Prem & Hybrid Architecture & Integrations
  • Solutions
    Back
    Focused on
    Implementing Data Governance

    A tool stack for starting fast and sustaining data governance

    Data Fabric

    Activate metadata and automate data mapping, extraction, and providing

    Big Data Management

    Ingest reliable data, govern your data lake, and process data.

    Single View of Data

    Establish a single source of the truth and create a single view for everyone.

    See all industry solutions
    From
    Banking, Insurance, Finance

    Data validation on input, Customer 360, regulatory compliance

    Healthcare

    Patient 360, reliable data for tests and EHRs, HIPAA compliance

    Retail

    Data validation on input, Customer 360, data enrichment, reference data

    Government

    Citizen 360, data sharing and protection, smart cities

    Life sciences

    Product MDM, clean data for clinical studies, spend transparency

    Telecom

    Customer 360, data enrichment, equipment tracking, data privacy

    Transportation

    Equipment monitoring, Customer 360, reference data, data privacy

    Latest read
    Ataccama Receives $150 Million Growth Investment from Bain Capital
    Ataccama Receives $150 Million Growth Investment from Bain Capital

    Ataccama receives a $150 million growth investment from Bain Capital Tech Opportunities to enhance R&D and go-to-market, and enable data democratization.

  • Customers
  • Company
    Back
    Contact us
    Schedule a call Contact us Sign-up for newsletter Live chat
    Company
    About us

    All about us, who we are, vision, leadership, offices

    Media kit

    Download our brand assets, photos, and product screenshots

    Careers

    #NotYourAverageJob

  • Resources 1
    Back
    Resources

    Videos, articles, tips from our experts & thought leaders

    News Success Stories Blog Whitepapers Webinars Demos
    All resources
    Support

    Get answers to your technical questions

    Documentation Training Knowledge base User community Customer support
    Events

    Attend our live virtual and in-person events, coming up

    Future of Financial Services, Melbourne 2022

    Jul 20

    Innovate VIC 2022

    Jul 21

    Hand picked for you
    title
    What Is Data Quality and Why Is It Important?

    Learn what data quality is, why it is important, what costs and risks bad data carries, and how you can get started with data quality today for free.

  • Partners
    Back
    Partners
    Become a partner

    Get to know our partnership model, join us

    Ataccama Partner Portal

    Login to our partner portal to access all essential tools and resources.

    Register Opportunity

    Register the lead and get a partner reward

    Our partners

    See our technological partners, system integrators and delivery partners

  • Try now
    Back
    Meeting
    Book a meeting

    Discuss your needs and requirements with one of our sales representatives.

    Free tools
    Web profiling

    One-click profiling in your browser. Just drag and drop a file.

    Data Quality Analyzer

    Advanced profiling tool. Install in minutes on Windows.

    Data Stories

    Modern data visualization. Present complex facts and wow all stakeholders.

    See all free tools
  • Contact
Ataccama
Login
User
Log In or register
Contact
Logo with rockets
Announcing
$150 Million Growth Investment
BainCapital logo
Learn more
Blog

What is a Data Mesh? 

8 minutes read

Data mesh is a new data management architecture that has gained a lot of popularity in recent years. It can bring greater stability and efficiency to a company’s data. However, it’s crucial to fully understand it, its advantages, and the necessary considerations before deciding what is best for your company.

What is a data mesh

The data mesh is a decentralized architecture and governance concept that puts the responsibility for data on the teams that produce—and actually own—the data. For example, financial team members would be responsible for the data in the financial domain. There will be no centralized ownership of data. Instead, each team will monitor their data, analyze, and transform it.

While there won’t necessarily be centralized governance rules for the entire organization, each team will prepare its data in such a way that, if another branch or department needs it, it is ready for them to work with. This allows the data mesh to distribute ownership without having data become siloed.

The pillars of data mesh

Still not sure what it means to have a data mesh? Maybe it will be easier to understand if we break it down further. 

Domain oriented decentralization

The objective of this principle is to put the responsibility for data on teams that generate and own that data. Instead of the current status quo: giving the responsibility for data to “external teams” of data engineers and DWH teams that would ETL that data to a central data warehouse or data lake.

The people (data domain teams) working with the data (entering it, profiling it, sharing it with the organization, etc.) should take full responsibility for keeping their data up to company standards, usually by creating data products.

While the structure of these teams can vary depending on the domain, team, and organization, they will almost always have someone in the “data product owner” and “data product developer” roles. A data product owner is responsible for providing measures that guarantee data is delivered as a product. A data product developer is responsible for constructing, monitoring, and serving the data products — more on data products below.

Data as a product

Data as a product is a principle that forces domain teams to apply product thinking to the data they own, i.e., to ensure their data is usable. The result of applying this principle is creating, owning, and maintaining data products everyone will love. That, one could argue, is the alpha and omega of the data mesh, and not only… Here at Ataccama, we think companies should think about their data as a product regardless of whether they’re building a data mesh organization.

So what characteristics allow you to consider a regular data set a “data product?” According to data mesh founder Zhamak Dehghani, they are the following:

  • Sharable and discoverable: publish data products for the rest of the organization to use.
  • Self-describing: data products should be very well defined.
  • Addressable (same location, predictable in the future): data products should be able to be accessed.
  • Trustworthy: data products should have high data quality.
  • Interoperable: data products should work with other data products.
  • Secure: protected from unauthorized access.

To achieve all these characteristics, data products need to have all of the following built-in features:

  • A way to extract new data from sources
  • A way to output data in a well-defined interface
  • The code it needs to run to transform the data
  • Storage infrastructure
  • Control ports where you can call the data product, run transformations, and request the data

"[They should have the] ability to run independently without relying on other data products or infrastructure. This unit should be built, maintained, and deployed as one component.”

- Zhamak Dehghan founder of Data Mesh

To summarize, a data product is a way to describe a dataset that is entirely up to organizational usability and governance standards and ready for use with no additional work required. This way, regardless of where the information is stored or who’s responsible for it, everyone at the company will be able to use it. All of the company’s data will be accessible upon request without any additional effort or preparation.

Self-service data infrastructure

To maintain the data mesh’s decentralized nature, domain data teams must be autonomous in building data products. For that to work, data teams need to have data infrastructure as a service. This infrastructure should be easy to use, meaning instead of teams building their own, they have all the tools and technology they need to build a data product.

It goes against the data mesh concept to require teams to use anything they don’t want to. The mesh relies on most of them using the infrastructure by choice, so companies need something they enjoy, are familiar with, and can see the value in using the same standard as other teams in the organization.

Data mesh structures this infrastructure via three planes:

  • Data infrastructure plane
  • Data product developer experience plane
  • Data mesh experience plane

First layer: data infrastructure plane

Providing the underlying infrastructure needed to build, run, and monitor your data products.

Second Layer: data product developer experience plane

Abstraction of data infrastructure designed to support the data product development journey — provisioning things like output endpoints, data quality, versioning, and security for data products. This is supposed to work by data product developers being able to use easily — or request (again, in a self-service manner) —these capabilities and the underlying infrastructure will provide that functionality.

Third Layer: mesh experience plane

Browse existing data products, ensure documentation is in order, and look at how data products relate to each other — more convenient access to capabilities for data citizens.

Federated governance

Since this is a decentralized concept, the governance principles that apply to data products can vary from department to department. However, risks or “governance debts” can arise if teams have no common ground to stand on. Some governance problems that data mesh teams risk running into are:

  • Metadata and cataloging standards differ
  • Missing/incomplete documentation
  • Availability windows
  • Low quality/different quality standards
  • Incompatible formats
  • Different security standards and policies

To mitigate these risks, companies should adhere to the concept of global governance based on a few open standards. Teams will immediately understand the advantage of having some principles in common throughout the organization.

Data mesh, therefore, relies upon teams following a set of global principles for data governance, which should be documented as in any data governance program. However, to simplify the enforcement of these policies, data mesh commands to automate, automate, automate. And the way to do that is, of course, to embed them into processes. For example, automate data quality processing, data masking, or the assignment of workflows for data access requests.

Data mesh benefits and considerations 

Before you decide data mesh is suitable for your business, please weigh the following benefits and considerations.

Benefits

Produces motivated and efficient teams. Data domain teams focus on a specific domain and have both the knowledge and technical skillset to be autonomous. Freedom to use the tools of your choice and Independence from vigorous governance enforcement encourages them to take more ownership and become more motivated.

Accelerates delivery of data products. For all the reasons mentioned, these motivated and efficient teams are better at creating data products.

Reliable and extendable. As organizations grow, the self-service infrastructure makes it easier to onboard new users, and shared governance rules are clearly documented, allowing them to expand without having to constantly rework administrative principles.

Considerations

Governance debt (if standards are not enforced). If your global governance standards are too loose, you can end up with some of the governance debt problems we mentioned earlier.

Possibility to recreate silos. The worst-case scenario for a data mesh is that no one follows governance guidelines and you end up with data products that aren’t interoperable, discoverable, or addressable – data silos. You must maintain governance standards between all domains to avoid data siloing.

Data mesh vs. data fabric

Data fabric is another emerging data management framework that has made a lot of noise in the last 12 months. On the surface, it might seem that it is a competing concept for data mesh. But is it really so? The fact is these new blueprints are not mutually exclusive.

A data fabric is all about frictionless access to enterprise data for all consumers. It works by connecting all data sources, data management components, and data consumers into a unified system heavily driven by AI to automate data providing.

Data mesh, on the other hand (as we already discussed), is focused on producing high-quality data — data products.

If we combine these two we will get frictionless access to high-quality data. In essence, data fabric empowers data people to create great data products (while managing governance debt).

If you look closer at the self-service data infrastructure and federated governance pillars of the data mesh, you can see that a data fabric can be the layer that ensures them, with these specific characteristics:

  • Global Governance
  • Automatic Documentation
  • Discoverability & Shareability
  • Interoperability
  • Efficient Processing
  • Security
  • Quality

Having these principles in place will allow domain teams to create data products which are:

  • Easy to discover
  • Easy to use & combine
  • Trustworthy & documented
  • Unburdened by provisioning
  • Automatically enforced standards
  • Fast to create

The bottom line is:

Data fabric empowers data people to create great data products while managing governance debt.

Build your data mesh with Ataccama

If you feel data mesh is the right architecture for you, Ataccama can help. We provide a unified data management platform, Ataccama ONE, for teams to build, publish, and discover data products.

Ataccama ONE comes with a data catalog, automated data quality, MDM, and data governance seamlessly connected with shared metadata.

It easily integrates with your data landscape and can be deployed to any infrastructure. If you’d like to learn more, schedule a call with us.

Related articles

What is Data Fabric and Why You Should Care

What is Data Fabric and Why You Should Care

Blog
What Is Data Quality and Why Is It Important?

What Is Data Quality and Why Is It Important?

Blog
What is Data Profiling?

What is Data Profiling?

Blog
What is data governance?

What is data governance?

Blog
Privacy Policy Cookie Policy Terms of Use Ethics Hotline
English
English Deutsche Pусский Français Espanol
© Ataccama 2022
Cookies We value your privacy

We use cookies on our website to enhance your browsing experience. By using our website, you consent to the use of cookies. To understand more how we use cookies or how to change your preference and browser settings, please see our privacy policy.

Select cookies