Testing

Testing is the step you take once you have a model or prompt you want to change, before anything reaches real users. Calls do go to the real provider, but the results are not stored anywhere, so nothing mixes into production data and you can run trials as many times as you want.

Success metricsdefine the bar Testingtry candidates Routingroute traffic Comparejudge outcomes

The input and the slots

Area	What you see
Shared input	The user message every candidate receives. “Load from history” pulls in an input a real customer actually sent, which is the most trustworthy kind
Slots	Each slot takes a name, model, system prompt, and parameters. The default setup puts the variant currently in production in A and the new candidate in B, and you can add up to five
Results	Per slot: the response body, tokens, cost, latency, and the prompt-versus-completion token breakdown

With the input and the slots filled in, “Run all” runs everything at once. Judge quality by eye and cost and speed by the numbers, then press “Save as variant” only on the slots you are happy with, promoting them to routing candidates.

Which parameters you can tune depends on the model. Pick one without tunable parameters and the screen says “No tunable parameters for this model.” Inputs and slot setups persist until you close the browser, so you can spread your runs across the day.

Saving changes nothing for your users

A newly saved variant starts at ratio zero (no traffic). Until you raise its ratio in Routing, nothing happens to real users. The save notice’s “Route this variant” takes you straight to the Routing screen.

The final verdict happens in Compare

Testing only takes you as far as screening candidates on a few chosen inputs. A candidate that looked great in Testing can behave differently on real traffic. An experiment is only finished once you confirm the outcome in Compare.