The definitive guide to setting up agentic analytics

Analytics agents are the most exciting development in data analytics in years. But as with new technologies, data teams are still learning the right playbook as they build. Here we’ll give you an overview of how they work, how to think about making a buying decision, and strategies for how to roll out analytics agents successfully in your company.

Understanding agentic analytics

What is agentic analytics?

Agentic analytics is the use of AI agents to perform data analysis tasks independently with little human oversight. This extends beyond simple text-to-SQL capabilities and can entail complicated, multi-step workflows.

Analytics agents use a harness specifically designed for data work. A harness is a set of instructions and code that wrap around a LLM. Claude Code is a harness for Anthropic’s Claude models that is specifically engineered for coding tasks, for example.

Analytics agents take in the user’s question, read through the business context, search through the data warehouse for the right tables, run as many queries as needed to determine an answer, and present back to the user. The harness provides tools and guidance about how to execute this process.

Why do companies want agentic analytics?

  1. The data team is the bottleneck

    Katie Bauer from Hex wrote Your data team is too slow (and accuracy won’t save you), in which she argues that stakeholders prefer asking data questions in natural language (rather than using a BI tool) and prefer a quick answer to a perfect one. Agents are an improvement in that world.

  2. Self-service analytics

    This was promised in the age of the Modern Data Stack. AI agents are better equipped to handle the long tail of questions as well as the rounds of follow-ups that stakeholders ask in many analyses. And importantly, the business users can do this themselves rather than ask someone from the data team.

  3. Reduce ad hoc requests to the data team

    Agents handling inbound questions saves costly hours from the company’s data practitioners. This means more time can go into ambitious projects pushing the business forward.

The way we interact with data is changing

The practice of going into a dashboard every day and reviewing yesterday’s numbers is one of the past.

The first thing you’d probably do in the AI age is have your agent monitor the dashboard, then tell you if anything needs your attention.

When someone needs to review last quarter’s numbers, they ask Claude to grab them.

Point-and-click BI tools were a great technology but not the way we consume and interact with data. We need to design our data reporting in ways that are just as friendly to agents as they are to humans.

Will agentic analytics replace data teams?

We don’t think agentic analytics replaces data teams in the near future. But the role of the data team absolutely changes. Companies need internal owners of agentic analytics. Data teams need to shift priorities as needed until agentic analytics is a focus.

Data teams that do not own this function, and instead let it go to an org like engineering, risk being cut out of the way the business is interacting with data. Those teams then become at risk of the chopping block.

Can we trust analytics agents?

Is AI accurate enough for data analysis?

Yes, AI is accurate enough for data analysis when you provide the right infrastructure around it. That means the right model, right harness, the right context, and the right interaction from the user.

It’s a classic “garbage in, garbage out” scenario. Ignore any one of these categories and it’s easy to see results degrade.

Is AI as good or better than a data analyst?

This is the wrong framing. The best data analysts now use AI, and get better results in less time. So you need to ask who will get better results: a data analyst using AI or a business user using AI?

Data analysts likely have better data literacy skills and can trace more effectively through the agent’s thinking. The business user may have better domain knowledge.

Companies will have both of these archetypes. It’s the data team’s responsibility to ensure both are set up for success.

Benchmarks and evals

You may use benchmarks (standardized tests across different tools) and evals (tests on your systems) to measure the accuracy of analytics agents. AI agents are not hitting 100% accuracy anytime soon, so it’s important to know where you stand and ensure that performance improves over time as you refine your context suite.

Can you plug ChatGPT or Claude into your data warehouse?

You can plug general purpose agents like ChatGPT and Claude into your data warehouse, but without the right context they will deliver subpar results.

If you really prefer to use a general purpose agent, we’d nudge you towards Claude Data — its dashboards use consistent metric definitions, so you’ll use the same methodology between sessions. Those definitions may not be the right ones, though.

Setting up agentic analytics

How much effort does it take to set up agentic analytics?

The effort required for agentic analytics varies based on your starting point, budget, security/compliance requirements, and desired outcomes.

Starting point

If you use a data warehouse, dbt for data transformations, an existing data visualization tool, and a semantic layer — you have a great start. Your company keeps good documentation in Notion? Even better.

If you haven’t set any of these up yet, that’s okay. It just adds a few steps to the process. We’d recommend at least getting a data warehouse and a data pipeline tool to move your data from your database and SaaS tools into your new warehouse.

Budget

A limited budget may alter the decisions you make, towards a stack that takes more time to set up.

Security and compliance requirements

Companies in regulated industries such as GovTech or healthcare will have different requirements than your average early-stage B2B SaaS startup. This will often impact choices in tooling, budgets, and workflows.

Desired outcomes

The more ambitious the outcomes you expect of your analytics agents upfront, the longer it will take to achieve those goals.

Generally, we’d recommend “crawl, walk, run”. Get something working quickly, build the habit and learnings, and then compound.

Is your data AI-ready?

Your data is probably AI ready. Sorry to all the data consultants who hinge their pitches on this fear.

All companies think their data is messy. Using that as an excuse to procrastinate AI adoption is silly.

You need a data warehouse, and a data pipeline tool to get data into your warehouse. dbt data transformations, semantic layers, and example queries are nice to haves but can be done in parallel. They’re also easier than ever to spin up with AI.

Should you buy vs. build agentic analytics?

Whether to buy vs. build agentic analytics varies based on the company. The cliche is “buy for parity, build for advantage”.

That holds well in agentic analytics. Many big tech companies have opted to build their own solutions and done quite well. However, we’d generally recommend buying if you’re not able to pour in enough investment to ensure that your build gives you an advantage.

Reasons we default to buying:

  1. Vendors are incentivized to develop more robust observability and evals tooling than someone in-house whose attention is split.

  2. Vendors have many customers needing enterprise-level features (including for security). Data analysts and data engineers trying to set up a homegrown agentic analytics platform may not have enough expertise in these areas.

Other factors companies may need to consider in their decision:

  1. Desired data handling and retention policies. Many companies do not want AI trained on their data.

  2. Tolerance of vendor lock-in. Ideally, context is portable and team members can get the same answers regardless of which agent or app a user asks a question to. Some vendors do not support this. Others do to varying degrees.

How much does agentic analytics cost?

Components of agentic analytics cost include platform/SaaS fees, compute for running the agents if they’re in the cloud, and tokens for agents (likely at API prices).

Many businesses want cost predictability, but that’s generally a problem still being worked out. Some vendors enable orgs to cut off AI usage at the user or company level. If complete cost predictability is a must-have, decision-makers may need to consider self-hosting (including using cheaper or open-weight models) or work with their vendors on contingencies once they reach their limits.

Who owns agentic analytics internally?

Data teams should be volunteering to own agentic analytics. This is the way that business users will interact with data, and data teams should want to own that channel.

If you decide to buy instead of build, any role on the data team should be able to maintain the context needed for a successful pilot.

Who sets up agentic analytics?

Companies have three different options for implementing agentic analytics: in-house, consultants, or forward-deployed engineers.

In-house

If your company decides to buy, your data team will often be able to get your initial agents up and running. They’ll focus on providing basic context and skills to your agents.

If your company decides to build, in-house teams may include combinations of data analysts, data engineers, analytics engineers, and software engineers. Hosting, security, auth, and harness engineering are added to the context and skills responsibilities.

Consultants

Data and AI consultants do more of these zero-to-one deployments and can fill in skill gaps (or bandwidth gaps) of internal data teams. The internal data team remains the context expert, while the consultants know how to adapt the tools to the clients’ requirements.

Forward-deployed engineers

A forward-deployed engineer (FDE) works for a vendor and embeds directly with the customer team to implement the vendor’s product. They will have very strong expertise on their agentic analytics product and the best FDEs pick up organizational context quickly. Many consultants also work forward-deployed.

How do you connect agents to your data?

How you connect agents to your data depends on your tooling.

The simplest with general purpose agents like ChatGPT and Claude is just to give the agents your data warehouse credentials, or to set up the warehouse’s official CLI or MCP server. If you’re doing this on your own, the agent is probably acting as you. You may not want that.

If you are working with a vendor like Hex, you may create a dedicated user or service account for that tool so that agents have exactly the right level of permissions, and it’s easier to track how much activity in your warehouse comes from agents.

You should make sure you adhere to your organization’s data handling and privacy policies when giving AI agents access to your data.

How do you provide context to your analytics agents?

It’s crucial for both speed and accuracy that your agent has access to key context around your data. AI agents have context windows, which is the amount of information they can handle in one session. As you manage your agent’s context, you must think about 1) what gets loaded into every chat 2) how to effectively invoke skills or specific knowledge 3) how the agent searches through your environment to get the right details.

Here are some of the key pieces of context we recommend giving to your agent as soon as you can.

Global context file

Your global context file helps the agent orient itself and eliminates the need to re-introduce and explain yourself every new chat. It also prevents the agent from needing to inspect your data warehouse schema each chat.

Think of this as your AGENTS.md or CLAUDE.md. Your agent reads this in every chat that’s created. This means you want to limit information that is relevant a fraction of the time. Background information about your company, data warehouse architecture, and basic preferences about how the data team wants the agent to work with data are what should go here.

Generally, once this gets past a few hundred lines, you’ll find the agent has a harder time keeping track of all the instructions. Be careful of sprawl over time.

Skills and guides

Skill files give your agent the domain expertise to answer common questions effectively. Agents are not deterministic, meaning you aren’t guaranteed to get the same approach to a data question each time. But if you give an agent a skill file showing how you want it to handle questions about ARR, marketing analytics, or support ticket volume, it’s much more likely to follow the procedure you want it to.

If you’re a B2B SaaS company, you have several domain areas of the business using data. You should have at least one skill file for each of these teams. We’ve had success using the basic structure that Hex recommends.

  1. Description so that the agent knows when to activate the skill

  2. Key metrics definitions

  3. Entity relationships and join patterns

  4. Preferred tables

  5. Risk areas

and adding example queries where possible. These should again be a few hundred lines. If you’re cracking 1,000 lines or more, that’s a sign that you need to be separating into multiple skill files.

Semantic layer

Skills tell the agent how to approach a problem. The semantic layer tells the agent how the data in your warehouse fits together.

Don’t be intimidated by the jargon. A semantic layer is just you defining data types, metric calculations, joins, and other basic info about your data.

If you’re starting from scratch, this is super quick. Ask your agent to write the first draft, and then you review and correct where necessary.

Metadata and data documentation

Beyond the semantic layer, your agent should know basic metadata about tables in your data warehouse.

This includes knowing when tables were created/updated, table descriptions, and column descriptions. If you use dbt or a similar data transformation tool, see if you can also integrate your dbt repo so that agents know the code and tests behind the data they’re working with.

Showing the agent where to look

Regardless of how many tables are in your data warehouse, you want your agent to find the right data tables as fast as it can. Two levers help:

  1. Configure your database connection so the agent only sees data it will actually use

    The agents likely don’t need to see raw data, dev environments, or intermediate tables. You can tell the agent not to use these, but it’s better to just make sure the agent never sees it. This can greatly improve search speed.

  2. Certifications

    Many vendors let you certify, endorse, or verify tables that the data team recommends agents use in their work. Agent harnesses tend to look through certified tables first when searching for what data answers the user’s question.

How do you run a pilot program?

Run a pilot program for agentic analytics similarly to how you’d do it for any other tool rollout in your company.

We recommend getting a small group of data- and AI-forward people from your company to be your test users. Ideally, they’re plugged into day-to-day operations, so you’ll get real feedback about what workflows your analytics agents actually help them in.

Work with your small group, as you start building excitement within the rest of the company. Once the data team is satisfied with initial agent performance and your testers give the green light, expand access and be sure to monitor usage.

Optimizing agentic analytics

Evals

You can’t expect 100% accuracy from your analytics agents, so it’s important to know how accurate they are. This is not a one-time check.

Evals are a set of example questions that test the performance of the agents using your context. You can choose the balance of simpler questions vs. trick/pitfall questions.

Ideally, you run these on a schedule to see how change tracks over time. And if you make a change to your context files, you should run the evals over the updated version to see if there’s any regressions.

This takes extra work but is worth it. Some vendors are building this into their product, and some new vendors are popping up for this exact use case, benchouse.ai being one of the first.

Observability

It’s critical for data teams to actively follow how business users are interacting with analytics agents. If you use a vendor, there should be a feature to view the chats of users. Even better if you can feed this directly into an AI agent to analyze for you.

Your first few times reviewing old chats will likely be very stress-inducing. Bad prompting, asking the agent things it has no way of knowing, accepting a wrong conclusion, and more.

Whether it’s self-service BI tools or agentic analytics, the data team will ultimately be responsible for how the business uses the data. Having a dataset of past chats is an incredible resource to catch anti-patterns and correct them.

You can see where agents trip up or lack context, then close the feedback loop by patching those issues

What models should you use for agentic analytics?

Which models to use for your analytics agents depends on your preferences for cost, speed, and accuracy.

Izzy Miller from Hex runs the DataBench benchmark, which has done some very insightful testing of various AI models against a fake company’s data. As of November 2026, Claude Opus 5.5 is the clear leader in accuracy. But it’s slow and expensive.

In Izzy’s workshop at the Hex Prompt conference, he talked about model routing — the process of selecting the right model for the specific request. He found that teaming up Opus 5.5 with GPT-6 Luna produced similar accuracy but better cost and speed than just Opus 5.5.

Depending on whether you’ve built your own solution or use a vendor, we recommend running your own benchmarks when possible so that you can understand which models work best on your data.

How do you achieve consensus?

Ian Macomber from Ramp wrote an excellent post called The Shape and Feel of the Post-AI Data Stack. In it, he calls consensus the scarce resource.

He describes the problem: “Every person can ask a slightly different question, look at a different slice of customer, use a different metric definition, end up with a different number, and draw a different conclusion.”

His article recommends testing consensus through code, by testing each interface (app, agent, etc.) + model combination on the same questions and testing how many different answers they get back.

This is very impressive but probably not feasible for small data teams. Don’t completely ignore consensus, though. Reduce this down to whatever testing your team can handle. You want to get a hold of this problem before it gets out of hand.

How do you turn insights into action?

Companies are increasingly trying to become AI-native. When you’re making the conversion, you need to remove as many bottlenecks as possible to reduce the human intervention necessary.

Agentic analytics can speed up the process of getting insights and findings from our data. But now we’re producing more output than humans can actually read and take action on.

Where possible, you should think about where to safely let the agent act on its findings, once it’s passed your verification process and safeguards. Previous Riverboat clients have added analytics agents to custom internal applications in order to reduce the friction between data question and business action.

What are the risk areas with implementing agentic analytics?

There are several things that can go wrong with deploying analytics agents into your business. Some of these are on the data team, and some are more dependent on business users and company culture.

Not looking at chat usage

Data teams that aren’t monitoring where agents go off the rails will not learn how to improve their context setups and eventually lose trust from stakeholders when answers are frequently wrong.

Not having the right context

AI agents will often spit out a confident but wrong answer when they don’t have the right context, and take longer to deliver that answer. Data teams must stay on top of addressing frequently-asked or high-leverage topic areas, and add the right guardrails when the agent doesn’t have enough information to give a correct answer.

Bad prompting

The average AI user is very bad at prompting. Provide some basic best practices and training to your stakeholders. Often they just need to see a few examples to make a decent jump in prompt quality.

Blindly trusting the agent

AI agents will always make mistakes. So much of your success with agents will depend on whether or not and how quickly you catch those mistakes. Data practitioners and others with good data literacy skills have an advantage here, but you need to find a solution that meets everybody where they’re at.

AI dashboard and analysis sprawl

It’s easier than ever to create dashboards and analyses. Too much of them at low quality overwhelms organizations and erodes trust. Data teams must own the data culture and governance that prevents this.

Conclusion

Everything is moving quickly, but this is a great time to start on your agentic analytics journey. Let us know how it goes, and reach out if you’d like Riverboat to help.