How-to guides
Prove the impact
Wire success metrics to events and answer "is this AI feature paying for itself" with numbers.
The goal of this guide is to be able to say this to leadership a few weeks from now.
The review summary’s success rate is 12.4%, with revenue at 18x the AI cost. Last month’s model switch lifted revenue per AI dollar by 42% with the success rate held.
Call counts and cost alone never reach a sentence like that. What it takes is two lines of events and two metrics.
Two lines of events
Section titled “Two lines of events”Outcomes are defined by user behavior, so that behavior has to be collected first.
If the client SDK is already in your site or app, add one capture line to each behavior that counts as an outcome.
abto.capture('summary_copied'); // copied the summary: a signal it was usefulabto.capture('purchase', { value: 32000 }); // purchased: amount carried as valueSent events appear automatically on the Success metrics screen.
If you are unsure which behaviors to send, one of each of two kinds works well. Behavior close to the feature (copy, save) is sensitive to change, so experiments resolve fast, and behavior close to the money (purchase, payment) is a number executive reports can use directly.
Two metrics
Section titled “Two metrics”On the same screen, create two metrics.
| Metric | Form | Definition | Question it answers |
|---|---|---|---|
| Summary success rate | Ratio | purchase 70% + summary_copied 30% | Is this feature driving the target behavior |
| Revenue per AI dollar | Value | sum of purchase value ÷ sum of call cost | How much revenue does the AI spend bring back |
Mark summary success rate as Primary and it becomes the Overview summary number and Compare’s default basis. Revenue per AI dollar is the direct answer to the executive question.
The math from here is automatic
Section titled “The math from here is automatic”From the moment the metrics are defined, aggregation is automatic. A user’s behavior accrues to the variant that user received, so per-model outcomes split accurately, person by person.
In Overview you overlay the cost line and the success rate line, and in Compare this metric becomes the basis for every experiment verdict.
Putting it in a report
Section titled “Putting it in a report”The numbers a report needs are already on the dashboard. Take the cost and success rate trends from Overview’s 30-day range, and the headline figure from the “revenue per AI dollar” metric. Opening Compare live in the quarterly meeting to show “this is how we experiment and this is how we verify” is persuasive too.
The day you route traffic to a new feature, set up its Primary metric alongside.