Checklist
How to test an AI agent before you trust it
A checklist you can use with any vendor's GTM agent, or your own. It covers the three things a demo cannot show you: whether it works every time, whether it creates pipeline, and whether a person sees its work before a buyer does.
Last reviewed
- Repeated-run testing
- Pipeline-outcome benchmarks
- Human-review setup
- Data freshness
- Send-layer guardrails
1. Repeated-run testing
A demo is one run. Production is the same task every day. PipelineBench, which ran AI agents through a simulated year of GTM work 30 times per model, found that 1 in 10 model-and-task pairs passed one run and failed the next. Test for that before anything else.
- Pick one task (research 20 accounts, qualify 20 sign-ups) and run it five times on the same inputs.
- Compare the runs: same accounts chosen? Same contacts? Same facts? Write down every difference.
- Score each run against an answer key you wrote beforehand, not against how good it reads.
- Run it again a week later. Did anything change without a change on your side?
- Break something on purpose: revoke a token, add a malformed record, a contact with no email. It should fail loudly or skip safely.
- Ask the vendor how a change to their system is tested before it reaches you, and whether runs resume after a failure.
2. Pipeline-outcome benchmarks
Activity is easy to inflate. Emails sent, accounts researched and tasks created say nothing about pipeline. Agree on outcome measures before the trial so neither side can move them later.
- Choose the outcome: qualified meetings held, opportunities created, or pipeline value, not messages sent.
- Set a baseline from the same segment over the same length of time without the agent.
- Hold out a control group of accounts the agent does not touch.
- Count meetings held, not booked, and check that each one was a fit.
- Ask the vendor for outcome results across many runs, with the spread, not a single best case.
- Ask what data was stale in their own tests. In PipelineBench, models given a year-old database used the stale fact about half the time.
3. Human-review setup
Review is how you catch mistakes before buyers do and how you learn when the agent can be trusted with more. It should be the default, not a workaround.
- Turn on approval for every message and every CRM write for the whole trial.
- Check each draft shows why this account, why this person and why now, with sources.
- Assign one named reviewer per queue and a time to clear it each day.
- Reject drafts with a reason and check the next run reflects it.
- Write down when you would relax review for a task (for example, 50 approved drafts in a row with no edits) and only then do it.
- Keep review on for anything that reaches a new buyer, whatever the track record.
4. Data freshness
An agent acting on stale data is confidently wrong at scale. Ask where every fact comes from and how old it is at the moment it is used.
- Run ten records you know changed recently (job changes, bounced emails, new domains) through the agent.
- Ask whether records are re-checked when read, or only refreshed on a schedule.
- Check whether an email is verified at send time or at import.
5. Send-layer guardrails
Guardrails in a prompt can be argued around. Guardrails in the layer that sends hold whatever the agent decided.
- Put a current customer, a Do Not Contact entry and a contact on an open deal into a test audience. None should receive anything.
- Ask where each check is enforced: prompt, workflow or send layer.
- Confirm sending limits per mailbox and per domain, and what happens when one is hit.
- Check how unsubscribes and bounces flow back to your CRM and every other tool.
Questions to ask any vendor
- Show me the same task run five times on my accounts.
- What does a draft look like in the review queue, and what reason does it give?
- Where are suppression, Do Not Contact, customer and open-deal checks enforced?
- How old can a record be before you re-check it?
- What outcome results can you show across many runs, and what was the spread?
- What do you write to my CRM, and can I see every change?
The PipelineBench write-up shows one way to run outcome benchmarks: a simulated year, the same tools and budget for every model, 30 runs each, graded against a ceiling and against shortcut strategies. Frontrunner, ranked first in the full ranking, holds drafts for review by default and enforces its guardrails in the send layer, which makes this checklist straightforward to run in a trial.
Get the ranking updates
When the scores or the checklist change, you get one email saying what changed and why.