How to choose an AI agent vendor: criteria, a fair test and lock-in

Most vendor selections for AI agents start in the wrong place: a polished demo. The demo shows the best case, on the vendor's data, run by the vendor's engineer. A week later the team likes one product and the selection is effectively over, although nobody has checked whether it fits the process, the risk or the budget.

This article is for the manager who has to recommend a vendor or platform for an AI agent, and who will be asked later why that one was chosen. It assumes you have already picked the process, as described in choosing the first process by risk, value and reversibility. If you are still deciding whether to build instead of buy, start with the buy-versus-build checklist and return here once the answer is "buy".

Step 1: Write the requirement before you meet vendors

Put one page together, owned by you and not by the vendor:

• The process and its boundary. What the agent does, what it must never do, and where a human decides.
• The risk level. What the worst realistic mistake costs, and who is affected. A drafting assistant and an agent that changes customer records need very different evidence.
• The success measures. The baseline you have today and the improvement that would justify the cost, as in the guide on measuring value with baseline, rework and review cost.
• The constraints. Systems it must connect to, where data may be stored, who must approve it internally, and the budget range including your own team's time.

Without this page every vendor will happily redefine the problem to match their product.

Step 2: Turn the requirement into weighted criteria

Five to seven criteria are enough. More of them hide the ones that matter. A typical set for an operations or customer-facing agent:

1. Fit for the process. Does it do the actual task, with your systems, in your language and volumes?
2. Quality on your cases. Measured in step 4, not claimed.
3. Control and oversight. Approval steps, audit trail, the ability to stop it. See human approval and audit trail.
4. Security and data handling. Access model, retention, where data goes, whether your data is used for training. See identity and least privilege and data handling and retention.
5. Operability. Logs, monitoring, versioning, support hours and how changes to the underlying model are announced.
6. Commercial terms and exit. Pricing at your expected volume, service levels, liability and the way out, covered in vendor contracts, SLA, liability and exit.
7. Vendor viability. How long they have operated, who else uses the product in a similar setting, and what happens to you if they are acquired.

Give each criterion a weight that adds up to 100, and agree the weights before scoring. Then mark two or three criteria as pass or fail. Typical examples are "no use of our data for training without consent", "an audit trail we can export" and "a documented way to switch the agent off". A product that fails one of them is out, whatever its total score.

Step 3: Build a shortlist of three, not ten

Use references, analyst-neutral sources such as your own network, and the vendors' public documentation to reduce the field to three. For each, send the same short questionnaire based on your pass or fail criteria and ask for written answers. Vague answers, or answers that point to marketing pages, are information in themselves.

The security questions are worth taking from a neutral source. The OWASP Top 10 for LLM applications lists risks such as prompt injection, sensitive information disclosure and excessive agency, and each of them translates into a question for the vendor: how do you limit what the agent can do, and what happens when it is tricked? The UK National Cyber Security Centre's supply chain security guidance is a useful frame for the broader question of how far you trust a supplier and how you check it.

Step 4: Run a fair test on your own cases

A proof of concept is the heart of the selection and the part most often skipped. Keep it short, two to three weeks, and keep it fair:

• One test set for everyone. Assemble 30 to 50 real cases from your process, with the correct outcome known in advance. Include the awkward ones: incomplete requests, unusual customers, conflicting data, and a few attempts to make the agent do something it should not. The method is the same as in evaluating reliability before production.
• Same conditions. The same inputs, the same time frame and the same definition of a correct result for every finalist.
• Independent scoring. Someone from your team, not the vendor's engineer, marks the results. Where marking is a judgement call, let two people score and compare.
• Record effort, not only accuracy. How long did setup take, how much did your people have to fix, how many cases needed a human anyway?
• No production data you cannot share. If real customer data is needed, agree the handling in writing first or use anonymised copies.

The NIST AI Risk Management Framework treats testing and measurement as a continuing activity and not a one-off gate, which is a helpful reminder: keep the test set. You will use it again when the vendor changes the model, and in the operational review after launch.

Step 5: Look at lock-in while you still have leverage

After signature, switching costs are the vendor's leverage. Before it, they are your questions. Ask each finalist, in writing:

• Can we export our data, configuration, prompts, knowledge base and logs, and in what format?
• Which parts of the agent are specific to your platform, such as a proprietary workflow language, and which are portable?
• Who owns the evaluation sets and the improvements made during the project?
• What happens to our data and our access on the day the contract ends, and how fast?
• How are price changes limited during the term?

Then estimate the cost of leaving as a rough number of person-weeks. It need not be exact. If one vendor's number is three times the others, that is a price you pay for the convenience, and it should be visible in the decision. Open interfaces help: a platform that connects to tools through a published standard such as the Model Context Protocol usually makes your integrations easier to move than one that requires its own connectors, although you should confirm this for the product in question and not assume it.

Step 6: Decide, write it down and plan the first review

Score the finalists against the agreed weights, apply the pass or fail rules and then look at what the numbers do not capture: the quality of support during the test, how openly the vendor discussed limits, and how well the team liked working with the product. Prefer the vendor whose weaknesses are known and manageable. A vendor who claims there are none has not been tested hard enough.

Keep a one-page decision record: the requirement, the criteria and weights, the test results, the open risks and who accepted them. It protects you when someone asks in a year why this product was chosen, and it makes the later handover to another team or a replacement much easier. Finally, start small. A staged rollout with a rollback condition lets you confirm in production what the test suggested.

A short checklist

• A one-page requirement written before the first vendor call.
• Five to seven weighted criteria, with two or three pass or fail rules.
• A shortlist of three and written answers to the same questionnaire.
• One test set of real cases, scored independently, with effort recorded.
• Export, ownership and exit questions answered in writing, and a rough cost of leaving.
• A one-page decision record and a staged first rollout.

A good selection is not the one with the most impressive demo. It is the one you can explain to a colleague in five minutes, with evidence from your own cases.

Sources

• NIST: AI Risk Management Framework
• OWASP: Top 10 for LLM Applications
• UK NCSC: Supply chain security guidance
• Model Context Protocol: Introduction