AI billing reconciliation starts with a simple problem: your invoice tells you what you were charged, not what you actually used. That gap is what led Vaudit to build TokenAudit, verifying what companies actually used against what they were charged. So far, Vaudit has reviewed $34M in AI spend and found about $1.7M in billing errors.
‍
The following is an edited Q&A with Deepak Kumar, Head of Product at Vaudit, on how TokenAudit was built and what it found.

Q. What's your background, and what drew you to building an audit product rather than another AI tool?‍

I started in engineering, then moved into product at Flipkart and later Agoda. After that I founded a fintech company. Now I run product at Vaudit, along with our engineering team.
‍

What drew me to audit is that it's the unglamorous half of every tech wave, and it's where the real money sits. We started in ad tech, verifying what advertisers were billed against what they actually got. The pattern was simple: every vendor grades its own homework, and the buyer just accepts the grade because they have no way to verify it themselves. That's not just an ad tech problem, it's structural, and it shows up in any spend category where the seller controls the meter. AI is the newest and fastest growing version of that same pattern. Building another AI tool means competing with everyone else building AI tools. Building the record that verifies them, real billing reconciliation and real AI cost observability, means the bigger AI gets, the more this work matters.

‍

Q. Was there a specific bill, client, or moment that made you think "we have to build this"?

‍

Our own bill. We were auditing cloud spend for customers while our own AI consumption grew, and at some point it was obvious we should run the same reconciliation on ourselves. What came back was worse than what we were finding in cloud. That's uncomfortable when it's your own bill and you're the company that does this for a living. We took it to design partners in H2 2025 and the same patterns showed up in theirs.


Q. What was the moment you realized AI spend needed its own audit category, separate from general B2B spend auditing?

We found it in our own bill first. Before FinOps for AI was its own category, we ran standard cloud audits for customers. As our AI use grew, we ran that same billing reconciliation on ourselves. We verified what we had used against what we were charged. The gaps were bigger than anything we had seen in cloud.

If you have looked closely at your own AI invoice, you already know part of this problem. Cloud metering has had fifteen years to work out its bugs. AI billing has had none of that time. There is no independent AI cost observability. There is no LLM cost monitoring that was not built by the vendor sending you the bill. And AI billing breaks in more ways than cloud ever did. We mapped over thirty.

We took this to design partners in H2 2025. Their bills showed the same patterns. That is when it became its own category: FinOps for AI, on its own.

‍

Q. Companies already get invoices from their AI vendors. What are they not seeing in those invoices?

Your invoice tells you what you were charged, not what you consumed. It doesn't show which model served your request, how cached tokens were billed, or whether the rate matches your contract. There is no second record. The vendor counts your usage, prices it, and sends the bill.

That is what breaks billing reconciliation here. It takes two records, built separately, verified against each other. AI spend has only one number, from the party you are paying. Most AI cost observability tools have the same problem. They run on the vendor's own numbers.

Q. Why did AI billing specifically become such a mess? Is this a usage-based pricing problem, a multi-vendor problem, or something else?

‍

Usage-based pricing and multiple vendors make it worse, but neither one caused it. The real problem is simple: AI billing was never built in from the start. It was added on later.

‍

Token counting began small and simple. One model, one rate, one calculation. Then new features kept shipping: caching, tool use, vision, batch processing, extended thinking. Each one gave metering a new case to handle. Every new case is a place where a bug can hide.

‍

The system that runs the AI and the system that counts what it costs are two separate things. There is no second record to verify one against the other. That is exactly what billing reconciliation needs, and it is missing here. Billing also gets built after the product ships, not alongside it.

‍

Stripe took ten years to make its billing trustworthy. These labs have had two to four years, while shipping faster than any industry before them. That is why your LLM cost monitoring still feels unreliable.

‍

‍Q. Walk us through what actually happens when TokenAudit reviews a company's AI spend. What is it looking for?

We build a second record. We pull request level data straight from your own systems and recount it using the same tokenizers the vendor uses. So the numbers we start with are not our guess. They are the same math the vendor should have already run.

From there, we verify that count against two things: the vendor's billing data and your contracted rate card. We look across more than thirty known ways AI billing goes wrong, but it really comes down to three questions. Did you use what they say you used? Were you charged the rate you agreed to? Were you charged for things that should never be billed at all?

What we find gets sorted into two groups. Small mismatches, the kind of billing reconciliation drift you would expect in any complex system, go into one queue. Rate card breaches and confirmed vendor bugs go into another, and that is the queue we file claims on.

‍

Q. What's the anomaly actually catching? Can you give a concrete (even anonymized) example of a billing violation it flagged?

‍

These are patterns you only see at the request level. Once your invoice adds everything up, they vanish into the total. Here's what that includes: retries that keep failing but still get billed, with no result to show for it.  Model routing errors where the billed model isn't the one that answered. Responses that stop partway through but still get charged as if they finished. Cached data charged as if it were brand new. And prices that quietly change without you noticing. 

Retry storms are the most common. An agent fails, retries, and fails again the same way. Every attempt gets billed, none produce anything usable. On your invoice, that looks like normal volume. At the request level, it is obvious: the same call fires over and over with no result. Good LLM cost monitoring should catch this. Most tools do not, since they only show totals.

These are not hypothetical. A developer documented a 1000x token accounting error on Amazon Bedrock in March 2026, turning a $50 test into a $58,000 bill. It got fixed once he brought AWS the data. We saw a cluster of routing problems in April 2026, and Anthropic later confirmed a routing issue in that window. Routing made up roughly a quarter to a third of the overcharges we flagged that month, exactly the gap real AI cost observability is meant to close.

‍

Q. How is this different from a finance team just reviewing invoices manually, or from what the AI vendors' own dashboards show?

Your finance team can only verify the invoice against itself: the math, the totals, whether this month differs from last month. They can't verify it against what you actually used, since they have no record of that. That's not a diligence gap. It's a missing input.

‍

The dashboards you get from AI vendors are the bigger problem. They're built by the same company sending you the bill, from the same meter that produced it. If that meter is wrong, the dashboard just repeats the error instead of catching it. This is why most AI cost observability and LLM cost monitoring tools can only show you the vendor's own numbers

July's AWS incident made this clear. A pricing error pushed estimated charges into the trillions and ran for over 24 hours. AWS's own alarms left FinOps for AI teams guessing whether a 55 billion percent alert was even real. Real billing reconciliation needs two records built separately. A vendor can't give you both at once. That's not a knock on any provider. It's structural, and it's the same reason no other spend category lets the vendor grade its own homework.

Q. How fast can a company get a first estimate of what they might be overpaying?

‍

The analysis isn't what slows this down. Getting the data is. Once we have your request-level logs and the invoice for the same period, billing reconciliation is quick. We are just rerunning the same math the provider already ran.

What actually varies is how fast you can pull together your own request-level data. If your team already logs calls in one place, this moves fast. If your AI usage is spread across different product teams with no central logging, most of the work is just gathering your own side of the story first.

That work is worth doing either way, audit or not. Without it, you have no real AI cost observability. You have no LLM cost monitoring that is actually yours. You are just trusting the vendor's version of your own spend.


Q. We've said publicly that Vaudit has reviewed $34M in AI spend and identified about $1.7M in billing errors. What surprised you most in that data?‍

Two things stood out. The rate, first.  About five percent of the spend we audit has something wrong in it.  In a mature billing category, you'd expect a fraction of a percent. Anything higher would normally trigger a full control review.

The second is where it came from. Most people assume AI overspend is their own fault: bad prompting, sloppy agent design, no discipline. Some of it is. But the biggest single chunk we found actually came from the provider, not the customer. In April we saw a cluster of routing errors, and Anthropic later published its own report confirming a routing issue in that same window.

That changes how you should think about this. Without real AI cost observability, you can't tell your own waste apart from the provider's mistake. If you assume it's all on you, you'll spend your time optimizing your own code and never think to verify the bill.

‍

Q. About 80% of identified errors get credited back. What determines whether something is refunded vs. it stays a dispute?

‍

It comes down to whether we can show the provider its own data contradicting itself. If the finding is clear cut, meaning the vendor's own records prove the charge was wrong, it gets credited, usually fast. A rate that doesn't match your contract. A completion billed for a response that never finished. There's nothing to argue about, and once you put the evidence in front of them, most providers fix these without a fight.

‍

What stays disputed is anything that comes down to a judgment call rather than a plain fact. Waste is the clearest example. If your agent burned tokens in a retry loop, the provider still served those requests and still paid to run them. That's your architecture, not their mistake. We flag it because it's money you shouldn't have spent, but it isn't a claim, so we don't file it as one. That's the whole reason our billing reconciliation process runs two separate queues instead of one.

‍

Q. As AI spend keeps growing, what's the next blind spot you think teams aren't watching yet?

Agents making spending decisions without a person in the loop. Until now, someone wrote the prompt and had a rough sense of the cost. Now an agent decides how many calls to make, which model to use, how many times to retry, and how much context to carry. That spending decision has moved from a person to a system, and most companies have no AI cost observability at the moment the decision gets made. You find out on the invoice, after the fact.

The second blind spot is the resale layer. A lot of enterprise AI spend doesn't come straight from the model provider anymore. It arrives through AWS, Azure, or GCP billing instead. That adds another party between what you used and what you get charged, and it means your AI costs end up buried inside a cloud bill that a completely different team owns. 

‍

Q. If a CFO or head of finance does nothing else this quarter, what's the one thing you'd tell them to check first?

Ask one question: what percentage of the AI requests you paid for last month actually produced a usable result? Almost nobody can answer that. If your team can't either, you don't have a cost problem. You have an AI cost observability problem, and the cost comes after that.

Second question: does the rate on last month's invoice match your contract? That's basic billing reconciliation, and it needs no special tooling. Those two checks alone will tell you if this is worth a closer look.

‍