On demand: Trust-driven medallion architecture: How to make your CDEs ready for AI
Enterprises are consolidating on cloud data platforms like Databricks to put AI agents into production. This session shows how Databricks and Ataccama help teams deploy agents in Genie that answer business questions using governed, validated data. Each answer comes with a clear view of how trustworthy the data behind it is.
What you’ll learn
In this webinar, Ataccama, Databricks and the New York Power Authority walk through the full path from source system to AI agent. You’ll see:
- A live demo where a business user asks Databricks Genie for business insights and recommendations, and gets answers backed by a data trust assessment from Ataccama through MCP.
- How to identify your critical data elements, and how criticality set once at the business process level is inherited by every physical table.
- How data quality gates work inside Databricks pipelines: stewards define a rule once, engineers call the same rule, and bad records never reach silver or gold.
- How the New York Power Authority is moving quality checks into its pipelines for meter data that arrives roughly every minute, and where it chose to place its gates.
How do you know when to trust an AI agent’s answer?
The live demo starts with the outcome everyone wants: a business user asking questions of real data in plain English. Eliza Pochinova, Sales Engineer at Ataccama, asks Databricks Genie what is happening in a product performance table in the gold layer, then which products to prioritize for reorder. Genie reasons across tables, metrics and metadata, respects the user’s table permissions, and returns recommendation in seconds.
So Eliza asks the question she would put to any analyst: can I trust the data behind this recommendation? Genie connects to Ataccama through MCP and returns a data trust assessment for the table.
That moment carries the whole argument. “There was no analyst sitting in between the numbers from the recommendations and me as a consumer,” Eliza says. “With agents, the sanity check for your responses has to be within the data itself.”
Watch the demo to see what the trust assessment covers, and how the same report can be scheduled to arrive by email every week with trust scores attached.
Do you need to govern every dataset before agents can use it?
No, and trying to is how governance programs stall. No governance team will ever be big enough to watch hundreds of systems and datasets with the same care, and the old 80/20 rule of thumb still applies: a small share of your data drives most of your outcomes. So start from the use cases with three questions: Which reports carry regulatory exposure? Which decisions move money? Which processes or agents would visibly break if their inputs were wrong? Whatever feeds those answers is a critical data element, or CDE, and it earns owners, stewards, a governed glossary definition and quality checks. This does not have to be a full-time tagging job.
In Ataccama ONE, criticality is set once at the business process level and inherited by business terms, data products and physical tables, and detection rules extend it to new tables as they are profiled. “Deciding what is critical is a business call because they know their processes and the use cases well,” Eliza says. “But finding where it lives is a machine job or a platform job.”
The webinar includes a live look at managing CDEs and inherited criticality in Ataccama ONE.
Why check data quality at every layer of the medallion architecture?
Trust has to hold at every layer: at the source, as data moves from bronze to silver to gold, and where people and agents consume it. Starting at the source is cheaper than fixing issues after several rounds of downstream reprocessing, and more AI now reads source systems directly. Once data moves, data quality gates put a checkpoint in the pipeline: good records pass and bad ones are set aside before they reach gold. “When we catch these records here at the gate, it’s going to cost us just one check,” Eliza explains. Catch them downstream and you pay in pipeline reruns and rework.
Gates also close the gap between business stewards, who understand the rules, and engineers, who implement them. With a central rule library in Ataccama, stewards define a rule once, engineers call the same definition in their Databricks pipelines, and rule changes reach the pipeline automatically. Join the webinar to see a data quality gate published from Ataccama, traced through lineage, and called inside a Databricks pipeline.
How is the New York Power Authority approaching it?
The New York Power Authority (NYPA) is one of the largest state public power organizations in the US, according to Saptaparna Gupta, Data Quality Lead at NYPA, whose role is to ensure high-quality, trusted data that supports NYPA’s operations, customers and energy goals.
NYPA uses Ataccama as its data quality tool, running regular technical and business validations on more than 10 highly confidential datasets. The push to move checks into pipelines came with its first data product on Databricks: high-volume meter interval data from NYPA locations and upstream utility providers, arriving roughly every minute. At that frequency, checking data only after it lands is not enough, so the team decided to validate data in motion as well
NYPA is now implementing data quality gates with Databricks and Ataccama, focused mainly on completeness, validity, accuracy and integrity. The goal: only good records reach the gold layer, so it holds certified, trustworthy data products that can be consumed enterprise-wide.
Asked what peers should weigh before starting a similar initiative, Saptaparna points to volume, frequency and purpose: “For us, it was a question that if we do not implement DQ gates and the volume keeps increasing, how do we handle the quality?”
In the session, Saptaparna explains where NYPA is placing its gates in the medallion architecture, and what the team expects to change.
See how trust gets built into every layer
Landing data in the lakehouse is the hard part most teams have already done. Making it AI-ready means knowing which data matters, keeping it clean as it moves, and letting agents see how far to trust it. Watch the webinar to see how Ataccama and Databricks make every answer trustworthy, from governed, validated data underneath to trust scores your agents can check.
Speakers
