Verify a prompt change

A prompt change carries less risk than a model swap. It is easy to undo, and the cost structure stays put. That makes splitting traffic in half, to collect samples faster, the better move. The example here is a prompt revised after feedback that “the summaries feel flat”.

Requestspull tricky inputs Testingchange only the prompt Routingsplit traffic 50/50 Comparejudge by behavior

The input the complaint came from

In Requests, find the calls that likely caused the feedback, the tricky ones, and bring them over with “Open in Testing”. The point of this step is that the very input behind the complaint becomes the test case.

Change only the prompt

Run the current variant and the revised prompt side by side in Testing. Keep the model and parameters as they are. Change several things at once and you cannot tell later what made the difference.

If the responses are visibly better, press “Save as variant”. If you cannot tell the difference, it is fine to stop here. One meaningless change that almost shipped got filtered out.

Split in half to collect samples

Give each variant half the traffic in Routing. Samples build twice as fast. If you want to be careful, starting at 10% works too. Either way, on the first day check only the two variants’ error rates on Compare.

The evidence that it got better

Compare the two variants on the success metric in Compare. The evidence that “summaries got better” is not nicer-looking sentences but the share of users who copy the summary and go on to purchase. Behavior close to the feature (copying the summary) is sensitive to change, so the verdict comes fast.

If the two lines keep trading places even after the low-sample warning clears, the change did not make a difference that mattered to users. If one side is clearly ahead, raise it to 100%; if the lines stay tangled, drop the revised prompt to zero and move to the next candidate.