Eval your agent's tool rounds: Promptfoo + LLM4Agents, end to end

A plain-prompt eval stops measuring your agent at the exact moment the agent starts doing its job: the first tool call. This is the follow-up we promised in the migration runbook — about forty lines of glue that make Promptfoo evaluate the whole conversation, MCP tool rounds included.

The setup we are testing is the RAG agent from the 50-lines tutorial: a system prompt, a one-tool allowlist (vector_query), and maxToolRounds: 2. That post shipped with a three-query golden eval — a loop that checks for expected substrings in the answer. It catches the loudest failures. It says nothing about how the agent got the answer: whether it actually retrieved, how many rounds it burned, what the run cost. Those are the properties that regress silently when you edit a system prompt, and they are exactly what a plain string check cannot see.

Why plain-prompt eval is not enough

Point Promptfoo at the LLM4Agents OpenAI-compatible endpoint — as the migration post showed — and you get a solid eval of a single completions call. One prompt in, one response out, asserts on the text. That mode has a structural blind spot for agents: the interesting behavior lives between the request and the final answer. The model decides to call a tool, the platform executes it, the result goes back in, and the loop repeats until the model answers or hits its round budget.

Three regressions we have seen in practice, none of which a substring assert can catch on its own:

The agent stops retrieving. A well-meaning system prompt edit ("be concise") makes the model answer handbook questions from its own weights. The answer often still contains the right-looking string. It is no longer grounded in your corpus, and the first policy update will prove it.

The agent burns its round budget. A vague tool description makes the model retry vector_query with near-identical queries. With maxToolRounds: 2 the symptom is a thin answer; with a bigger budget it is a bill. Either way, the string assert passes.

Cost drifts. Every tool round is a billed call plus more tokens in context. A change that doubles average rounds doubles per-question cost without changing pass rates. On a platform where every response carries its settled cost, not asserting on cost is leaving information on the table.

A note on the runner

There is an irony to address before the code. The migration post recommended Promptfoo as the destination for eval suites leaving OpenAI's hosted Evals — and in March, OpenAI announced it was acquiring Promptfoo. The Promptfoo announcement commits to keeping the project open source and to "support a diverse range of providers and models." Those are words; the structural facts matter more. Promptfoo runs locally, on your machine or your CI, against any endpoint you configure. Your eval cases live in YAML files in your repo, not in a hosted console that can go read-only — which is precisely the failure mode that forced the Agent Builder migration in the first place. If the project's direction ever changes, the MIT license means the version you depend on keeps working and can be forked. Portable assets, local execution, permissive license: the checklist from the evaluation post survives the acquisition. We keep recommending it, with eyes open.

The custom provider pattern

Promptfoo's extension point is the custom provider: a JavaScript or TypeScript class with two methods. id() returns a label. callApi(prompt, context) receives the rendered prompt and returns an object whose only required field is output — plus optional fields that are the whole point of this exercise: cost, error, and a free-form metadata object that your asserts can read later. You reference the file from the config with file://, and whatever you put under config: in the YAML arrives in the constructor.

The pattern, then: instead of letting Promptfoo make an HTTP call, the provider runs your real conversation — same SDK, same system prompt, same allowlist, same round budget as production — and reports what happened as metadata.

The glue code

Here is the provider, complete. Save it as llm4agents-provider.ts next to your config:

// llm4agents-provider.ts — Promptfoo custom provider for agent evals
import { LLM4AgentsClient } from '@llm4agents/sdk';

const client = new LLM4AgentsClient({ apiKey: process.env.LLM4AGENTS_API_KEY! });

type AgentConfig = {
  model: string;
  system: string;
  allow: readonly string[];
  maxToolRounds?: number;
};

export default class LLM4AgentsAgentProvider {
  constructor(private readonly options: { id?: string; config: AgentConfig }) {}

  id() { return this.options.id ?? 'llm4agents-agent'; }

  async callApi(prompt: string) {
    const { model, system, allow, maxToolRounds = 4 } = this.options.config;

    // fresh conversation per test case — no state bleed between rows
    const conv = client.chat.conversation({
      model, system, maxToolRounds,
      tools: { mcp: { url: 'https://mcp.llm4agents.com/mcp', allow: [...allow] } },
    });

    try {
      const reply = await conv.say(prompt);
      const toolNames = reply.toolCalls.map((t) => t.name);
      return {
        output: reply.content,
        cost: reply.costUsdCents / 100,  // the X-Cost-Usd-Cents header, surfaced by the SDK
        metadata: {
          toolNames,
          toolCallCount: toolNames.length,
          toolArgs: reply.toolCalls.map((t) => JSON.stringify(t.arguments)),
        },
      };
    } catch (e) {
      return { error: String(e) };
    }
  }
}

Two design choices worth naming. A fresh conversation per callApi means every test row starts clean — no history bleeding between cases, which is what you want for regression testing (multi-turn evals are a different config, not a different provider). And the provider reports facts, not verdicts: which tools ran, how many times, with what arguments, at what cost. Judging those facts is the config's job, which keeps the glue reusable across every agent you operate.

The asserts that catch agent regressions

The config wires the provider and the golden queries from the RAG post together, then adds the asserts a plain-prompt eval could not express. Promptfoo's assertion system passes your provider's metadata straight into javascript asserts via context.metadata — that shortcut is what makes the whole pattern work:

# promptfooconfig.yaml
prompts:
  - '{{question}}'

providers:
  - id: file://./llm4agents-provider.ts
    label: handbook-rag-agent
    config:
      model: anthropic/claude-sonnet-4.6
      system: 'Answer strictly from the handbook corpus. Always call vector_query before answering. Cite the source filename. If nothing relevant returns, say so.'
      allow: [vector_query]
      maxToolRounds: 2

defaultTest:
  assert:
    # 1. the agent actually retrieved — grounding, not memory
    - type: javascript
      value: 'context.metadata.toolNames.includes("vector_query")'
    # 2. it respected its round budget — no runaway loops
    - type: javascript
      value: 'context.metadata.toolCallCount <= 2'
    # 3. it never repeated the exact same call — retry loop detector
    - type: javascript
      value: 'new Set(context.metadata.toolArgs).size === context.metadata.toolArgs.length'
    # 4. per-question cost ceiling, in dollars
    - type: cost
      threshold: 0.05

tests:
  - vars: { question: 'What is the refund window for annual plans?' }
    assert:
      - { type: icontains, value: '30 days' }
  - vars: { question: 'Who approves expenses over the team limit?' }
    assert:
      - { type: icontains, value: 'finance' }
  - vars: { question: 'What color is the CEO's car?' }
    assert:
      # out-of-corpus question: must retrieve, then admit it found nothing
      - type: llm-rubric
        value: 'The answer states the information is not in the corpus and does not invent a color.'

Run npx promptfoo@latest eval and every row executes the real agent. The four default asserts encode the three silent regressions from the top of this post. Assert 1 fails the "stopped retrieving" edit even when the answer text looks right. Asserts 2 and 3 fail the retry-loop regression — note that assert 3 is stricter than 2 and catches the pathology even under a generous round budget. Assert 4 turns the platform's per-call settlement into a regression gate: the number comes from the same reserve-proxy-settle pipeline as your production bills, so the eval prices exactly what production will pay.

The third test row is the one operators skip and should not. An agent that answers out-of-corpus questions confidently is a grounding failure that string asserts reward rather than catch; a one-line llm-rubric — model-graded, so it costs a grader call — is the cheapest honest check. And because the grader also speaks the OpenAI API shape, you can point it at the LLM4Agents endpoint and keep the whole eval, agent and judge, on one balance.

CI: fail the pull request, not the agent

The suite earns its keep when it runs on every change to the system prompt, the allowlist, or the tool descriptions. Promptfoo ships an official GitHub Action that runs the eval and comments the results on the pull request, with a diff view against the base branch:

# .github/workflows/agent-eval.yml
name: agent-eval
on:
  pull_request:
    paths: ['agents/**', 'promptfooconfig.yaml']

jobs:
  eval:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/cache@v4
        with:
          path: ~/.cache/promptfoo
          key: ${{ runner.os }}-promptfoo-v1
      - uses: promptfoo/promptfoo-action@v1
        env:
          LLM4AGENTS_API_KEY: ${{ secrets.LLM4AGENTS_API_KEY }}
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          config: promptfooconfig.yaml
          cache-path: ~/.cache/promptfoo

Two operational notes. Give CI its own registered agent with a small dedicated balance — eval traffic then shows up as its own line in client.wallets.transactions(), separate from production, and a leaked CI key can only spend its own budget. And keep the cache step: Promptfoo caches identical calls between runs, so a PR that only touches one test row does not re-bill the whole suite.

What it means for LLM4Agents

This tutorial closes the weakest box in the migration mapping. "Evals → Promptfoo" was honest but shallow while it only covered plain prompts; with the provider pattern, the eval story now covers what agents actually do on this platform — conversations that spend money calling tools. That matters commercially: the operator deciding whether to move a production workload off a deprecated hosted stack asks "can I test it like I test code," and the answer is now a forty-line file they can copy.

The deeper fit is billing. Most eval stacks treat cost as an estimate derived from token counts and a price sheet. Here the provider reports settled cost from the platform's own billing pipeline, per test row, and the cost assert makes it a gate. No other part of the stack could produce that number honestly — it falls out of reserve-proxy-settle for free. Evals that price themselves are a differentiator the platform gets without shipping anything.

Staying on the frontier

What this walkthrough exposes, in the order it should be addressed:

1. Publish the provider as a package. Forty lines of glue is a blog post; @llm4agents/promptfoo-provider is an integration. Ship it with the metadata contract above so asserts written today keep working.
2. Surface tool rounds in the reply object as a stable contract. This pattern leans on reply.toolCalls and per-call cost. Document both as versioned SDK surface — eval suites are the consumers that break loudest when a field moves.
3. Emit OpenTelemetry spans from the conversation loop. Promptfoo's trajectory asserts and the broader observability ecosystem converge on traces. When the SDK emits tool-round spans natively, operators get trajectory:tool-sequence asserts and production tracing from the same instrumentation.
4. Add a golden-suite starter to agent registration. A new agent should be born with a promptfooconfig.yaml scaffold — the eval habit should be the default path, not a best practice buried in two tutorials.
5. Watch Promptfoo's governance post-acquisition. The commitment to provider diversity is on record. If it weakens, the eval-runner recommendation should move — and because every asset in this post is local YAML and one provider file, moving would cost an afternoon, not a migration.