How-to guides
Roll out a new model
Follow the order for moving to a new model, from screening to a full switch.
This is the whole path from a new model to real users, and not a single code deploy is needed. Two things are worth having before you start: a Primary success metric to judge by (see Prove the impact), and a history of real calls to use as test inputs.
1. Screen the candidate in Testing
Section titled “1. Screen the candidate in Testing”In Testing, keep the variant currently in production as your baseline and put the new model next to it, leaving the system prompt the same for now. Fill the input with “Load from history”, using requests real customers actually sent. Results from inputs you made up are hard to trust.
Press “Run all” and compare quality by eye, cost and latency by the numbers. Repeat over a handful of different inputs, and if quality visibly collapses, stop here. That candidate never meets a real user.
2. Save, then route just 10%
Section titled “2. Save, then route just 10%”Once it passes the screening, press “Save as variant” and give it 10% in Routing. From this moment, one user in ten gets the new model’s responses. The rule that keeps a given user on the same variant is covered in Routing.
3. For a few days, just watch for problems
Section titled “3. For a few days, just watch for problems”Put both variants on Compare and check only that the new variant’s error rate is not spiking and its p50 latency is where you expect it. It is not time to judge outcomes yet.
If something looks off, filter Requests to the new model and find the cause, and if it is serious, drop the new variant to zero in Routing. That is the rollback, and it takes seconds.
4. Judge in Compare
Section titled “4. Judge in Compare”Put both variants into Compare and the feature’s Primary success metric is already set as the default basis.
Not rushing matters here. Until 100 users per variant build up, Compare shows a low-sample warning, and numbers under that warning are not verdicts. If you want to judge sooner, raising the ratio speeds up sample collection.
Once the warning clears, read the error rate, the success metric, then cost per call, in that order. If conversion is slightly lower but cost is much cheaper, a “revenue per AI dollar” metric folds both into one number.
5. If the results are good, expand
Section titled “5. If the results are good, expand”The stages for raising the ratio, and what to confirm at each, are in Routing. Drop the variant you are not keeping to zero and it drops out of the routing. There is no need to delete it.
Whether it is a prompt change or parameter tuning, whatever you change, the sequence is the same. Take the cost change before and after the switch from Overview’s 30-day trend and share it with your team.