Governing the AI Analyst: Preventing AI From Confidently Giving Your Team the Wrong Numbers
Making Large Language Models (LLMs) work on your data demands more than a clean data warehouse. It requires agreed-upon definitions for every metric your business measures.
One executive we’ve partnered with used Claude to cross-reference hundreds of sales calls against his forecast, creating a deal risk assessment to reprioritize over a million in pipeline in what would’ve previously taken his team a week to build manually.
That output was actionable only because his team partnered with us to do the unglamorous work of preparing their data platform for AI.
Technology was the foundation, but the harder work was defining the business by figuring out what to measure, how to structure the data, and what the metrics mean.
These three concepts separate AI analytics that work from AI analytics that confidently lie: a clean data model, an explicit semantic layer, and clear instructions for when definitions conflict. Most middle-market teams have none of these three.
LLMs punish ambiguity differently
Organizations run on institutional memory. People learn over time which definition of “pipeline” their team uses, which revenue number to trust, which analyst has the “right” spreadsheet, and what caveats to mention before presenting a number to leadership.
When this context isn’t on paper, the Large Language Model (LLM) and GenAI tool starts from scratch. It makes assumptions, collapses the data into a single answer, and presents it confidently. One head of operations put it to us directly:
"Someone's going to ask Claude for pipeline and it's going to give them a number they don't expect."
In other words, it’s very easy to build a machine that blasts bad information across your business.
A simplified data model: what clean data actually looks like
Companies have a common challenge: multiple datasets describe the same concepts in slightly different ways. "Active customer" means one thing in the product usage table but something else for finance.
One client's Salesforce admin told us directly: "We have a lot of duplicative fields, or fields that might suggest they're one thing, but are in fact something else." If we connect an LLM to this data and ask a question without documenting the ambiguity, we risk misinterpretation and erode trust in the data.
Clarifying this ambiguity begins by standardizing enterprise data within a data warehouse, a central environment where data from your CRM, ERP, and operational systems is consolidated. This centralized data has historically powered reporting and business intelligence teams. Today, it also powers AI tools.
Building data for AI to consistently operate with accuracy requires fewer tables, clear naming, and shared dimensions (customer, date, product) used identically across the warehouse.
In early interactions with one client’s data model, Claude returned different answers to the same general questions. Each answer was given confidently, looked correct on the surface, and had minor flaws. After cutting the data model from 20+ tables to 3, a non-technical operations leader easily recreated and tied out to their Power BI product analytics dashboard within Claude.
But schema design is only half of it. Business logic from BI tools and analysts must be stored in an explicit semantic layer, a documented set of definitions sitting between your data and your LLM.
An explicit semantic layer: what defined metrics actually look like
The semantic model defines each metric via a calculation and descriptions, providing the LLM with nuanced business context when engaging with the data.
Every metric definition, caveat, filter, and synonym must be written down in a form the model can reference. The AI needs to know that renewals aren’t in won pipeline, and that specific accounts must be excluded when adjusting EBITDA.
In practice, we write specific calculations for every metric, verify each with leadership, and carefully build guardrails into their AI platform. When a user asks something ambiguous, the LLM prompts them to clarify rather than picking a definition silently.
Like a healthy business, a healthy semantic model is always adapting. This is done by tracking and identifying patterns in the questions being asked. Three people asking for “churn” in three different ways reveals a definitional gap in the semantic model, and someone asking a question the model can’t answer exposes a data model gap. The system will get smarter as people use it, but only if you are tracking how they use it.
How to create consistent metrics
Metric alignment across departments is often impossible. Sales and customer success define pipeline differently not because someone is wrong but because they’re answering different questions. Forcing a single definition can destroy nuance that different teams legitimately need.
The solution: pick a default definition, document the alternatives, and make the AI transparent about which definition it’s using within each answer. The user must know what metric they’re looking at.
This is done through skills that provide data agents with clear instructions:
Claude skills are reusable instructions that tell the model how to respond to questions and behave in specific contexts, ensuring consistency of communication with users.
As metrics are defined and users interact with data through AI, the role of the analytics organization begins shifting from report building to semantic model optimization.
What separates the strongest implementations
Our best results come from companies where senior leaders use the tools themselves, find where the numbers feel wrong, and push their organization to fix it. These leaders become evangelists and drive adoption across the business.
The second common factor is a tight feedback loop. Businesses must track every question asked and enable non-technical users to create and edit metric definitions directly. The result is a system that improves with use.
The third is a focused starting point. Businesses can demonstrate value fast by delivering an end-to-end solution for a single problem. Select a narrow use case, deliver immediate and measurable time savings, and prove ROI to build internal credibility from the start.
The companies getting value today don’t wait for perfection. They move fast while also simplifying their models, writing down their metric definitions, and putting executives in front of the tools.
Skipping the data organization and governance work enables AI to confidently provide numbers that nobody has validated. The difference is whether your team gets a number they can act on today or a number they must spend the next three meetings explaining away.
Click here to learn more about how MERU can help your organization harness the full potential of your data. Our team of experts is ready to partner with you to design and implement a data strategy that meets your specific needs.
Authored by: Mike Rasmussen, Solutions Architect; Jasmine Xu, Analytics Engineer; Preston Howell, Senior Principal