You have three or four proposals on the table for an AI agent in your revenue team. They all promise to save your reps hours a week. They all come with a demo that worked.
Most AI vendor checklists were written for procurement. They ask about security certificates, model training and where the data is stored. Those questions matter, and they miss what is different about a revenue agent.
A revenue agent writes to the CRM your forecast is built on. It ranks the accounts your reps call first.
It sends messages to prospects under a real person's name, from the domain your whole company sends from. When it is wrong, a prospect sees it.
The twelve questions below are grouped by what they test. Each comes with what a strong answer includes. There is a scorecard at the end for putting the proposals side by side.
Why don't standard AI vendor checklists cover revenue agents?
The risk sits in front of your buyers.
A procurement checklist protects the company from the vendor. A revenue agent needs a second layer of questions, because its mistakes land on prospects and customers before anyone inside sees them.
Three things set it apart.
It writes to systems other teams trust, so a bad field update becomes a bad forecast. It acts under a person's name, so a bad message costs that rep a relationship.
And it sends at volume from a shared domain, so a bad week can push every email your company sends toward spam.
Since February 2024, Gmail and Yahoo have required bulk senders to keep spam complaints below 0.3%.
An agent that sends carelessly spends that margin for everyone.
Watch for a proposal that answers the security questionnaire in detail and never mentions what the agent is allowed to send.
Will they build it inside the tools you already run?
Your stack, your fields, your process.
1. Where will this run, and which of our systems will it read from and write to?
A strong answer names your CRM, your sequencing tool and your data sources specifically, and says which fields the agent will change. It describes the build in the language of your stack.
2. What will you change from your standard build to fit how our team works?
A strong answer names the specific differences. Your routing rules, your ideal customer definition, the step where your reps want to approve before anything sends.
An agency with a real method has a standard starting point. What you are testing is whether they listened before they proposed.
Watch for two proposals from different firms that you could swap the logos on.
Have they run this in production, or only demoed it?
Ask for evidence from a system that is already live.
3. Can you walk us through a system you built that is running today, and what it did last week?
A strong answer is a screen share of a live system with real volumes, the errors it hit, and who on the client's side runs it.
4. What broke in the first 30 days of your last build, and what did you change?
A strong answer is specific and slightly uncomfortable. Every agent that runs against real data hits cases nobody planned for. A firm that has shipped one knows exactly which ones.
5. What number should move by day 30, and how will we read it?
A strong answer names one business number, such as meetings held or speed to lead, records where it sits before launch, and sets the rule for calling it a win before anyone sees the result.
Usage counts like drafts generated rise whether or not the agent helped. How to measure one number before launch is a playbook of its own.
MIT's NANDA initiative found in 2025 that 95% of the organizations it studied were getting zero return from generative AI. Most of them could count usage.
Watch for a success metric that is a count of what the agent did.
How will you know the agent's output is right?
Written criteria, and a reviewer that did not do the work.
6. What does a passing output look like, and when do we write that down?
A strong answer says the criteria are written before the build, in plain terms your team signs off on. For an outreach agent that might be a word limit, one real buying signal named, and no account that is already a customer.
7. Who reviews the agent's work, and does the agent grade itself?
A strong answer separates the reviewer from the agent that produced the work.
A NeurIPS 2024 paper found that models like GPT-4 can recognize their own writing, and the better they recognize it, the more they favor it.
We covered how to set this up in Who Checks the AI's Work?.
8. What happens when it gets something wrong in front of a prospect?
A strong answer covers three things. How the error is caught, how sending is paused, and how the fix is tested before sending resumes. Ask to see the pause switch.
Watch for a review step that has never once sent work back.
What is the agent allowed to touch and send?
Permissions are a design decision, so make it on purpose.
9. What can it change in the CRM and send under a rep's name without a person approving it?
A strong answer is a short list, written down, with the reasoning. Many teams start with the agent drafting and a rep approving, then widen what runs alone as the review record builds up.
10. How do you protect our sending domain and the data the agent connects to?
A strong answer covers sending limits and a warm-up plan for new volume.
It also says how the agent's access to your systems is scoped, and how it gets revoked.
In August 2025, attackers used stolen access tokens from the Salesloft Drift integration to export data from numerous companies' Salesforce instances, per Google's threat intelligence team.
Watch for an agent that needs admin access to your CRM to do a job that touches three fields.
What do you own when the engagement ends?
Ownership is what you are buying.
11. Do we own the code, prompts, criteria and workspace, with no licence to keep paying?
A strong answer is yes, in the contract, including the accounts and workspaces the agent runs in. If the agency's own platform sits underneath, ask what happens to the agent if you stop paying for it.
12. If you were gone tomorrow, what would our person need to keep it running?
A strong answer names a person on your team who will run the system, the handover they get, and the documentation they keep. An agent with no named owner inside your company tends to stop being used within a quarter of the builder leaving.
Watch for a handover plan that is a final meeting.
How do you choose between AI consulting firms?
Score the answers, then read the reference calls against the scores.
Score each question 0 for no answer, 1 for a general answer and 2 for a specific answer with evidence. Twenty-four is the maximum.
| # | Question | A 2 sounds like |
|---|---|---|
| 1 | Where will it run, what will it read and write? | Your systems and fields, named |
| 2 | What changes from your standard build? | Specific differences tied to your process |
| 3 | Walk us through a live system | A screen share with real volumes |
| 4 | What broke in your last 30 days? | A named failure and the fix |
| 5 | What number moves by day 30? | One business number with a baseline |
| 6 | What does passing output look like? | Written criteria, signed off before the build |
| 7 | Who reviews the work? | A reviewer separate from the agent |
| 8 | What happens when it is wrong? | Catch, pause, test, resume |
| 9 | What runs without approval? | A short written list with reasons |
| 10 | How is our domain and data protected? | Limits, warm-up and scoped access |
| 11 | Do we own everything, licence free? | Yes, in the contract |
| 12 | What does our person need to run it? | A named owner and a real handover |
Then call two of each finalist's clients and ask one question. What surprised you after the agency left?
Watch for a firm that scores high on the questions and has no client you can call.
The pattern across the twelve questions
Every question here tests the same thing from a different side. Will the agent still be running, and still be right, six months after the people who built it have moved on?
Any agency can show you the first run. Ask to see the hundredth.
---
FAQ
What questions should you ask an AI agency before hiring them?
Ask where the agent will run and what it will write to, what changes from their standard build for your team, and to see a system they built that is running today.
Ask what a passing output looks like, who reviews the work, and what happens when the agent is wrong in front of a prospect. Then ask what you own when the engagement ends.
How do you choose an AI agency for sales and RevOps?
Score each proposal on the same twelve questions, with 2 points for a specific answer backed by evidence.
Weight the answers about production history, review and ownership most heavily, because those decide whether the agent is still useful after the agency leaves. Then call two of each finalist's clients.
What are the red flags when hiring an AI agency?
The clearest red flags are proposals you could swap the logos on, a success metric that only counts agent activity, a review step run by the same agent that did the work, and admin access requested for a narrow job.
A handover plan that is a single final meeting is another.
Who should own the code and prompts an AI agency builds?
Your company should own the code, prompts, evaluation criteria and the workspaces the agent runs in, written into the contract with no ongoing licence.
If the agent depends on the agency's own platform, ask what happens to it if you stop paying.
How do you evaluate an AI implementation partner's production experience?
Ask for a walkthrough of a live system with real volumes, and ask what broke in the first 30 days of their last build. A partner with production experience answers both specifically. Reference calls with past clients confirm it.
Should you build AI sales agents in-house or hire an agency?
Build in-house when you already have someone who can build agents and will stay to run them.
Hire an agency when you need the first system working sooner than you can hire, as long as the engagement ends with a named person on your team owning it.