Compare

Compare is where you go a few days after splitting traffic in Routing, to read the result. It is also the screen to watch in the first days after a new variant starts taking traffic, in case error rate or latency spikes. Per-variant numbers are read here, side by side.

Success metricsdefine the bar Testingtry candidates Routingroute traffic Comparejudge outcomes

The chart and the variant cards

Area	What you see	How to read it
Basis selector	Choose an event’s conversion rate or a success metric as the basis, and pick the two variants to compare	A Primary metric, if there is one, is selected from the start
Chart	The chosen basis over time, one line overlaid per variant	Read a difference only when the lines separate and hold their gap
Variant cards	Per variant: cost per request, average latency, error rate, and the value of the basis you chose	Four numbers, read side by side. The first three average one request across every node in the feature
Per-run comparison	Per variant: run count, requests per run, and cost per run	A variant that is cheaper per request can still cost more per run when each run sends more requests. Read it next to the cost per request on the variant cards
Node details	Per node and variant: request count, cost per request, average latency, p50, p95, p99, and error rate	When node request counts differ between variants, the variant cards move with them. This table compares node with node

The per-run comparison counts one agent or workflow run as the unit. It counts only calls for which your server SDK sends a run ID, and the table does not appear when no call carries one. A variant with successful requests whose cost is unknown shows no cost per run. To group several requests into one run, see the server SDK docs for JavaScript, Python, or JVM.

The variant cards and node details count one request your app sent as the unit.

• Request count: a request ABTO retried still counts once.
• Average latency: the time a successful request takes to get its response, including time spent on retries. Failed requests are left out.
• Cost per request: based on requests whose cost is known.
• Error rate: the share of requests that ended in a provider error (5xx), a timeout, a connection failure, or an ABTO internal error. Rate limits (429), malformed requests, API key or billing problems, and requests your app cancelled are not the variant’s fault, so they are not counted. You can find those failures on the Requests screen.

Under 100 users per variant, the screen shows a warning

The bar counts people who got that variant, not requests. When that number is small, conversion swings heavily on luck. While the warning is up, do not say which side is better; wait for more traffic to accumulate. The same bar decides whether you are ready to raise a ratio in Routing.

ABTO does not hand down statistical verdicts like p-values. It shows the numbers as they are.

Judge in this order

Order	Metric	The call
1	Error rate	A variant with visibly higher errors is out, regardless of outcomes
2	Success metric or conversion	Is the new candidate level with or ahead of the incumbent
3	Cost per request	At similar outcomes, pick the cheaper one

If conversion is slightly higher at several times the cost, whether that is worth it is a business decision. Once the choice is settled, go back to Routing and move the traffic.