← Back to blog

Automated agent optimization with Claude Code and AgentHippo: an accounts-payable matching case study

When an agent is not accurate enough, the usual fix is a bigger model. We found a cheaper and more effective one: tune the whole system around the model, including its engine, prompts, number of agents and memory. On our benchmark, that let Haiku 5.5, the smallest model in its family, outperform Opus 5.5 at a fraction of the cost.

Tuning a system this way takes hundreds of experiments. Claude Code ran them for us on AgentHippo, and that is how we built Slipcatcher, which matches a business’s records against the contracts, policies and orders behind them. Slipcatcher is now in soft launch. This post explains how we built it, with accounts payable as the case study.

The result

The job was accounts-payable recovery, a matching problem: read a company’s contracts, invoices and ERP exports, match them against each other, and report every amount it is owed back, exact to the cent, with no false claims. On a business with 28 real cases, Opus 5.5 on its own found 16 to 18 and made as many as 46 false claims per batch of documents.

On its first batch, Haiku 5.5 made about 40 false claims. A reviewer then marked each case real or not, and the agent kept those decisions in a casebook it had to reproduce before reporting anything new. On every batch after that, Haiku found 21 to 23 cases and made no more than 3 false claims, at a fraction of Opus’s cost.

The difference was the harness. With standard AP audit rules added to its memory as well, Haiku found a median of 27 of the 28 cases across 18 runs, with no false claims in 15 of them.

The accuracy holds as documents keep arriving. On a car dealer’s books, we added nine more batches, from a refund to a newly signed supplier agreement, and checked again after each one. Haiku ended with all 39 cases and no false claims. When a refund cancelled an earlier claim, the agent asked the reviewer before dropping it. Our previous design, which checked each batch on its own, ended with 28 cases and counted old ones again, inflating the total by more than $10,000.

Why every choice has to be tested

No single setting is best everywhere. The engine is a clear example. With Sonnet 5.5, Codex made fewer false claims than Claude Code at about 20% lower cost. With the Haiku casebook agent, Claude Code found all 28 cases in one session, while Codex found 23 with 4 false claims at more than twice the cost and failed to finish the next batch. Each of these is a single run, but they show that the right engine depends on the model and the job. That means it has to be measured, which is why the number of experiments grows quickly.

How AgentHippo and Claude Code ran more than 1,000 workflow runs

Over more than 300 benchmark campaigns, the agents completed more than 1,000 workflow runs and 46,000 workflow steps, across more than 50 pre-registered tests. Claude Code acted as our research assistant. It edited the agents (each one an agent pack), deployed them, ran the benchmarks, read the logs behind every missed or false finding, and proposed the next change. Before each run it recorded the expected result and a pass bar, and a change was kept only if it held across several businesses. Our team chose the directions and decided what to keep. The loop is fast: one night, it took a redesign from first run to verdict in under four hours, across 71 runs on three machines.

AgentHippo made it safe to let Claude Code run that loop unattended. Each agent is a signed, versioned pack, so any configuration can be rerun exactly a month later. Packs ran in parallel as separate deployments across several VMs. Every deployment had a budget cap set outside the agent, so nothing being tuned could raise its own limit, and the loop ran overnight without an unexpected bill. The evaluation data stayed isolated from the agents, with hidden answer keys and a leak check that failed the build if an agent could see one. Every model call was logged, so Claude Code could explain each miss from evidence instead of guessing.

Our test businesses are synthetic, and the reviewer in these runs read from the answer key, so the learning curves are optimistic. Sealed, freshly generated data is our next test.

If your agents are not accurate enough, try tuning the system before you pay for a larger model. AgentHippo gives Claude Code the versioned agents, parallel deployments, budget caps and logs it needs to run those experiments for you.

Slipcatcher

The agent from this post now runs Slipcatcher. Slipcatcher is built for matching problems: it checks a business’s records against the contracts, policies and orders behind them, and shows each mismatch with the lines that prove it. Supplier invoices against contracts and payments, as in this case study. Loan files against a credit policy. Payment records against court orders. Nothing counts until someone on the team confirms it. Slipcatcher is in soft launch at slipcatcher.agenthippo.ai, with access codes going to a first group of businesses.

Ready to ship agent packs with fewer surprises?

AgentHippo gives you one IDE, versioned packs, full traces, and rollout visibility across OpenClaw, Claude Code, Codex, and custom engines.

Download AgentHippo →