<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://germangerken.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://germangerken.github.io/" rel="alternate" type="text/html" hreflang="en" /><updated>2026-11-18T13:05:45+00:00</updated><id>https://germangerken.github.io/feed.xml</id><title type="html">German Gerken</title><subtitle>Practical engineering for people who run LLM agents in production: measurement, traces, and the Python behind the tools.</subtitle><author><name>German Gerken</name></author><entry><title type="html">Your agent’s success rate barely moved. One tool’s error rate tripled.</title><link href="https://germangerken.github.io/2026/11/18/agent-regression-case-study/" rel="alternate" type="text/html" title="Your agent’s success rate barely moved. One tool’s error rate tripled." /><published>2026-11-18T12:38:00+00:00</published><updated>2026-11-18T12:38:00+00:00</updated><id>https://germangerken.github.io/2026/11/18/agent-regression-case-study</id><content type="html" xml:base="https://germangerken.github.io/2026/11/18/agent-regression-case-study/"><![CDATA[<p><em>A reproducible experiment on why aggregate metrics can miss agent regressions, and what analyzing traces across the whole dataset reveals instead.</em></p>

<p>When releasing a new version of an agent, teams typically review a dashboard showing
metrics such as success rate, mean cost, and p95 latency. If these numbers are similar to
the previous week’s, the release is considered acceptable.</p>

<p>I tested how well this approach detects a regression I intentionally introduced. The
results were unsatisfactory, and the reason is worth examining.</p>

<h2 id="experiment-setup">Experiment setup</h2>

<p>I developed a small, seeded simulator for a customer-support agent. Each run produces a
trace that includes a router, LLM planner calls, retrieval, tool calls, and a final guard.
The simulator supports three intents: order status (45% of traffic), refunds (30%), and
product questions (25%).</p>

<p>I then created a second version with one intentional fault in the refund flow, which shows
up in two ways:</p>

<ul>
  <li>the planner repeatedly checks the refund policy, adding one or two extra rounds;</li>
  <li>the refund API times out more often, and retries often fail.</li>
</ul>

<p>Both versions generate 600 traces. This synthetic data has a known outcome, which is
intentional: I know exactly what failed, so I can evaluate what each method detects.</p>

<h2 id="dashboard-results">Dashboard results</h2>

<table>
  <thead>
    <tr>
      <th>Metric</th>
      <th>Baseline</th>
      <th>Candidate</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Success rate</td>
      <td>94.8%</td>
      <td>92.8%</td>
    </tr>
    <tr>
      <td>Mean cost per trace</td>
      <td>$0.0168</td>
      <td>$0.0181 (+7.5%)</td>
    </tr>
    <tr>
      <td>Error events</td>
      <td>0.6%</td>
      <td>1.8%</td>
    </tr>
  </tbody>
</table>

<p>Success rate dropped by two points. With 600 traces per version, the 95% confidence
interval for this difference ranges from −4.8 to +0.7 points. Since this interval includes
zero, a careful reviewer would consider the change noise. By the numbers, they would be
right to.</p>

<p>Cost increased by 7.5%, which most teams would not consider a release blocker.</p>

<p>The regression is real and large, but it is not visible in these metrics: refunds account
for only 30% of traffic, and most refund runs still succeed after a retry. The average
masks the issue.</p>

<h2 id="path-analysis-results">Path analysis results</h2>

<p>I analyzed both exports using <a href="https://github.com/GermanGerken/journeygraph">JourneyGraph</a>,
an open-source tool I maintain. It does not collect traces: it reads existing exports,
aggregates them into a single graph of steps and transitions, and then compares two such
graphs.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pipx <span class="nb">install </span>journeygraph
journeygraph diff baseline.jsonl candidate.jsonl <span class="nt">--output-dir</span> diff
</code></pre></div></div>

<p>The first three reported changes are as follows (step labels are shortened for clarity):</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>change: tool:refund_api error-event rate moved from 9% to 30% (+20 percentage points).
change: Path variant request → router → planner → order_lookup → planner → policy_check
        → planner → refund_api → planner → answer_guard covers 10% of candidate traces
        versus 28% (-18 percentage points).
change: Path variant … → (llm:planner → tool:policy_check)+ → … covers 12% of candidate
        traces versus 0% (+12 percentage points).
</code></pre></div></div>

<p><img src="/assets/images/diff-top.png" alt="JourneyGraph comparison report: notable changes" /></p>

<p>Both symptoms of the planted fault are identified, and in the order I would want to read
them:</p>

<ol>
  <li><strong>One tool’s error rate tripled</strong>, from 9% to 30% (62 of 210 calls).</li>
  <li><strong>The main refund path decreased</strong> from 28% of all traces to 10%.</li>
  <li><strong>A new looped path appeared.</strong> The <code class="language-plaintext highlighter-rouge">(planner → policy_check)+</code> notation indicates that
this pair repeats. This path now covers 12% of traces and did not exist previously.</li>
</ol>

<p>The single-version report for the candidate provides supporting counts: traces with a
repeated <code class="language-plaintext highlighter-rouge">refund_api</code> call increased from 9 to 41, and the
<code class="language-plaintext highlighter-rouge">policy_check → planner → policy_check</code> loop appears in 91 traces compared to zero
previously.</p>

<p><img src="/assets/images/candidate-graph.png" alt="Aggregate graph of the candidate version: refund_api is highlighted with a 30% error rate" /></p>

<h2 id="why-the-two-perspectives-differ">Why the two perspectives differ</h2>

<p>Success rate is a single value per trace, averaged across all traces. A regression limited
to one intent and partially masked by retries has minimal impact on this metric.</p>

<p>A path represents a structural change. For example, “this sequence of steps did not exist
last week and now covers 12% of runs” is not averaged away. Similarly, a per-step error
rate isn’t diluted, since its denominator is the number of calls to that step rather than
all traffic.</p>

<p>Trace viewers such as Langfuse, Phoenix, or LangSmith are built to inspect individual
traces in detail, and they are good at it. The relevant question here is different: what
patterns emerge across 600 runs, and how have these patterns changed between versions?
Answering it requires aggregation across traces before using a viewer.</p>

<h2 id="integrating-into-ci">Integrating into CI</h2>

<p>The same comparison can be used to gate a pipeline:</p>

<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># rules.toml</span>
<span class="nn">[regressions]</span>
<span class="py">max_success_rate_drop</span> <span class="p">=</span> <span class="mf">0.03</span>
<span class="py">max_error_event_rate_increase</span> <span class="p">=</span> <span class="mf">0.01</span>
<span class="py">max_mean_cost_per_trace_increase</span> <span class="p">=</span> <span class="mf">0.15</span>
<span class="py">forbid_new_categories</span> <span class="p">=</span> <span class="kc">true</span>
</code></pre></div></div>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>journeygraph check candidate.jsonl <span class="nt">--rules</span> rules.toml <span class="nt">--baseline</span> baseline.jsonl
</code></pre></div></div>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PASS max_success_rate_drop: success among known outcomes dropped by 0.02 (limit 0.03)
FAIL max_error_event_rate_increase: error-event rate increased by 0.0119583 (limit 0.01)
PASS max_mean_cost_per_trace_increase: mean cost per trace increased by (relative) 0.0747896 (limit 0.15)
check: 5 of 6 rule(s) passed        # exit code 5
</code></pre></div></div>

<p>The build fails, which is the correct outcome. However, the margin is narrow: it failed one
global rule by 0.002. With a slightly higher threshold, this regression would have passed.
Global thresholds have the same averaging limitations as the dashboard. Rules scoped to a
single step or a single cohort are the next thing I am building, and this experiment is
the reason.</p>

<h2 id="limitations">Limitations</h2>

<ul>
  <li><strong>The data is simulated.</strong> It demonstrates that the method can detect a known, planted
fault. It does not indicate how often real regressions resemble this scenario.</li>
  <li><strong>Associations do not imply causation.</strong> The report states “traces that include X had a
lower success rate”. It does not claim that X caused the failure.</li>
  <li><strong>Clear step names are important.</strong> This simulation uses eight step types. A raw
OpenTelemetry export from a microservice system contains dozens of low-level spans per
trace, which can make the graph noisy. Improved grouping for such cases is in progress.</li>
  <li><strong>Scalability.</strong> On my laptop, processing 240,000 events takes 16 seconds and uses about
830 MB of memory. Growth is linear, so one million events would need an estimated
3.5 GB. I have not measured that yet; I plan to benchmark larger datasets and will
update this post with the results.</li>
</ul>

<h2 id="reproducing-the-experiment">Reproducing the experiment</h2>

<p>The simulator, the rules file, and the expected output are in
<a href="https://github.com/GermanGerken/journeygraph/tree/main/examples/agent-regression">examples/agent-regression</a>.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python simulate_agent.py baseline baseline.jsonl
python simulate_agent.py candidate candidate.jsonl
journeygraph diff baseline.jsonl candidate.jsonl <span class="nt">--output-dir</span> diff
</code></pre></div></div>

<p>JourneyGraph is Apache-2.0, runs locally, has no runtime dependencies, and sends no
telemetry. It reads JSONL, CSV, Parquet, and OTLP exports, with field mappings for
Langfuse, Phoenix, and LangSmith exports.</p>

<p>If you run it on your own traces and the report is inaccurate or unhelpful, I welcome your
feedback: please <a href="https://github.com/GermanGerken/journeygraph/issues">open a GitHub issue</a>.</p>]]></content><author><name>German Gerken</name></author><summary type="html"><![CDATA[A reproducible experiment on why aggregate metrics can miss agent regressions, and what analyzing traces across the whole dataset reveals instead.]]></summary></entry></feed>