Measured by Hand: Building an Agent Optimization Pipeline Around Human-Labeled Review

There is a version of agent optimization that looks impressive on a slide and falls apart in a regulated environment. You point an LLM judge at your agent, let it grade a few hundred runs, feed the scores into an optimizer, and watch a pass-rate number climb from 40% to 85%. The demo lands. Everyone claps.

Then a compliance officer reads the transcripts and disagrees with half the passes.

The problem with many agent-optimization pipelines is simple: they optimize toward the opinion of a judge nobody vetted. The judge is confident. The judge is fast. The judge is also wrong in the places that matter most: edge cases, ambiguous extractions, and moments where a regulated workflow has a right answer plus a plausible-looking wrong one. Optimize hard against an unaligned judge and you do not get a better agent. You get an agent that is very good at pleasing a critic your domain experts would never have hired.

We build agents for air-gapped, regulated, and government environments, where "good" is defined by subject-matter experts, statutes, and the specific way an organization does the work. Our optimization pipeline is built around a deliberately old-fashioned commitment: nothing gets optimized until a human has read the actual interaction and said, in their own words, whether it was right.

We think of it as tailoring. You cannot cut a bespoke suit from a size chart. You take measurements off the actual person, you baste a first version, you have them try it on, and you alter it by hand. The fitting is not a step you skip to save time. It is the reason the garment fits. Hand-labeled review is that fitting.

The cast: three agents and an arena

Our optimization pipeline runs inside the Test Arena, a controlled environment where a target agent is exercised, scored, and tuned without touching production. The Arena has a tuning_mode toggle. With it off, the Arena is a regression harness: it runs scenarios and logs results. With it on, the Arena becomes an optimizer that searches for a better configuration.

The Test Agent drives interactions. It replays representative tasks against the target agent: document-extraction jobs, retrieval queries, adversarial inputs, and the awkward cases pulled from real usage.
The Judge Agent scores each interaction against a domain rubric. It starts naive and gets aligned to human labels over time. It is the critic, but one continuously corrected by people.
The Tuning Agent only wakes up when tuning_mode is on. It takes the aligned Judge as its objective function and searches over prompt and configuration variations.

Underneath all three sits Langfuse, our observation and system-of-record layer. Every run, every LLM call, every tool invocation, every Judge score, and every human label lands in Langfuse as a trace with attached scores. Nothing is ephemeral. If it happened in the Arena, it is inspectable, queryable, and labelable after the fact.

Every interaction the Arena has ever produced can be pulled up, labeled by a human, and folded back into the loop.

Step 1: Instrument and version the target agent

Before we can improve an agent, we have to see it. We wrap the target agent in Langfuse tracing so that a single invocation produces a full trace: inputs, retrieved context, tool calls, intermediate reasoning, final output, latency, and token cost.

from langfuse.decorators import observe, langfuse_context

@observe()
def run_target_agent(task: dict, config_version: str) -> dict:
    # config_version pins the exact prompt and settings under test,
    # so every trace is attributable to a specific agent build.
    langfuse_context.update_current_trace(
        tags=["arena", f"config:{config_version}"]
    )
    return target_agent.invoke(task)

We also treat the agent's system prompt and configuration as a versioned artifact, not a string someone edits in place. Every candidate the Tuning Agent later produces gets its own version and its own tag. When a run is good or bad, we can trace it back to the exact build that produced it and roll back without archaeology.

Step 2: Give the Judge Agent a rubric

The Judge Agent needs to know what "good" means for this domain before it can score anything. This is where we encode the first draft of domain expertise: what the agent must always include, what counts as a hard failure, and what is merely a stylistic preference.

For a document-extraction agent working environmental compliance filings, the rubric might require every extracted value to cite its source location, normalize units, flag low-confidence extractions instead of guessing, and avoid fabricating anything outside the source document. The Judge reads the trace and returns a pass/fail plus a rationale on each dimension.

The first rubric is a hypothesis about what experts want. It is a starting point, not the standard. The standard comes from review.

Step 3: Run the Arena and establish a baseline

With tuning_mode off, the Test Agent runs the full scenario set against the target agent. The Judge scores each run. Everything streams into Langfuse.

for scenario in arena.scenario_set("deq-extraction-v3"):
    trace = run_target_agent(scenario.task, config_version="baseline")
    judge_score = judge_agent.evaluate(trace, rubric="deq-extraction")
    langfuse.score(
        trace_id=trace.id,
        name="judge_pass",
        value=judge_score.passed,
        comment=judge_score.rationale,
    )

Now we have a baseline: a pass rate, and more importantly a browsable set of traces tagged with the Judge's verdict. In Langfuse, an extraction run shows up as green or red, and one click opens the full interaction: what the agent saw, what it retrieved, what it produced, and why the Judge scored it the way it did.

This is already useful as a regression harness. It is not yet trustworthy, because the only opinion in the loop so far is the Judge's. So we go get a better one.

Step 4: The fitting, hand-labeled review

This is the step that makes the rest worth running.

We route a sample of the baseline traces into a human review queue. Domain experts, the people who actually know what a correct answer looks like in this domain, open each interaction in Langfuse, read the real thing, and label it. Not a rating on an abstract scale. A verdict on a concrete interaction they can see in full: this extraction is right; this one silently dropped a qualifier; this one looks right but cites the wrong table.

# A human reviewer's verdict, attached to the exact trace they read.
langfuse.score(
    trace_id=trace_id,
    name="expert_pass",
    value=False,
    comment="Value is correct but sourced to the wrong section; "
            "in a filing, provenance is the whole point.",
    data_type="BOOLEAN",
)

Now every reviewed trace carries two verdicts: what the Judge said, and what a human expert said. The gap between them is the most valuable signal in the pipeline, because it is a precise list of places where the automated critic disagrees with the people whose judgment we trust.

We use that gap to align the Judge Agent. Disagreements are analyzed, distilled into concrete guidelines, and paired with real labeled traces as reference examples. "Fail any extraction that omits provenance, even if the value is correct" is much more useful than a generic instruction to be stricter. We measure alignment as agreement rate between the Judge and a held-back set of human labels, and we do not trust the Judge as an optimization target until that agreement is high.

What we refuse to compromise on

The human reads the interaction, not a summary. Langfuse gives reviewers the complete trace: retrieval, tool calls, and output. They grade what actually happened.
The label is written, not just clicked. A boolean pass/fail is the score; the reviewer's rationale is the gold. "Missing sample size context" or "hallucinated a section number" teaches the rule behind the verdict.
Disagreement among reviewers is a finding. When two experts label the same interaction differently, we surface it. Either the rubric is underspecified or the domain genuinely has a judgment call in it.

Everything downstream inherits trust from this step: the Judge's calibration, the Tuning Agent's target, and the regression suite. Skip it and you have automated confident nonsense. Do it well and you have a formal definition of quality that came from your actual experts.

Step 5: Turn tuning mode on

Now, and only now, do we let the optimizer work. With tuning_mode on and an aligned Judge as the objective function, the Tuning Agent proposes variations of the target agent's prompt and configuration, runs each candidate through the Arena, and scores every run with the Judge that now agrees with your experts.

arena.configure(tuning_mode=True, judge="deq-extraction@aligned")

result = tuning_agent.optimize(
    seed_config="baseline",
    scenario_set="deq-extraction-v3",
    objective="expert_aligned_pass_rate",
    max_candidates=30,
)
# result.best_config is a new, versioned, promotable build.

Because the objective function was calibrated against hand labels, the Tuning Agent is climbing a hill your experts actually want it to climb. The optimizer is only as good as its target, and the target is only as good as the review that shaped it.

Step 6: Run held-out scenarios

Finally, we run the baseline build and the tuned build head-to-head on held-out scenarios the Tuning Agent never saw during optimization. Both runs go into the same Langfuse experiment so we can compare them trace by trace.

Held-out evaluation separates real improvement from overfitting. If the tuned agent only wins on scenarios it saw during tuning, we have taught it to memorize. If it wins on interactions it has never encountered, scored by a Judge calibrated to our experts, the gain is real. Then we promote the new config version to production. Any code loading the production alias picks up the improved agent without a redeploy of application logic.

Why the labels compound

If the pipeline stopped at "run it once and promote a winner," it would be a useful exercise. It becomes a platform capability because the labels keep paying rent.

They align the Judge Agent. A disagreement between a human label and the Judge's score becomes a guideline and a reference example.
They inform the Tuning Agent. Labeled failures become concrete targets to design around; labeled successes become patterns to preserve.
They become permanent regression fixtures. The exact interaction re-runs in the Arena forever, so a fix in one release cannot silently regress in the next.

This can run in CI/CD. Every merge to the target agent can trigger the Arena, score with the current aligned Judge, and log everything to Langfuse. The build fails if the aligned pass rate on the held-out set drops below threshold. Meanwhile, interesting new interactions from those runs sit in Langfuse waiting for review whenever an expert has a moment.

Test Agent
  -> target agent runs scenarios
  -> Langfuse records traces and scores
  -> experts read traces and write labels
  -> labels align the Judge Agent
  -> aligned Judge guides the Tuning Agent
  -> tuned candidate returns to the Arena
  -> held-out scenarios decide promotion

The result is a labeled corpus that only grows. The Judge gets more aligned over time. The Tuning Agent gets a richer set of examples to optimize against. The regression suite gets denser. Each hand-labeled interaction is a small, permanent investment in every future run.

Why hand labels matter more in our world

In a general-purpose SaaS product, you might get away with a lightly supervised judge and a lot of traffic. In the environments we work in, you cannot.

"Good" is defined by experts and statutes, not consensus. A compliance extraction that is 95% correct and silently wrong about provenance can be worse than useless. Only a domain expert reading the actual interaction can catch that, and only their written rationale can teach the Judge to catch it next time.

The labeled corpus is a sovereign asset. It is built from your documents, your workflows, and your experts' judgment. It never leaves the air gap. It is not something a competitor can buy off the shelf, and it is not something an off-the-rack model arrives with. Over time, it becomes the most defensible thing in the system: an encoding of how your organization defines quality.

Auditability is mandatory. When someone asks why an agent behaves the way it does, "the optimizer found it improved a number" is not an answer that survives a regulator. "Here is the interaction, here is the expert who reviewed it, here is what they wrote, and here is the guideline it produced" is.

The measurements are the point

It is tempting to focus on the optimizer: the automated tailor that tries thirty variations and picks a winner. But the optimizer is the least distinctive piece. Anyone can run a search over prompts.

What makes an agent fit is the measurements. In our pipeline, those measurements are hand-labeled reviews of real interactions. Experts read what actually happened and write down whether it was right. The Test Arena gives us a place to produce interactions safely. Langfuse makes every one of them permanent and labelable. The Judge and Tuning Agents turn labels into calibration and improvement.

Off-the-rack agents are measured against a size chart. Bespoke agents are measured against a person. We build the second kind, and we do it by hand.

Mark Avallone, on behalf of the team at Canall.ai

Filed under · Technical · Evals · Agent optimization ← All posts

Measured by hand:
building an agent optimization pipeline.