Choosing an AI agent vendor is mostly a matter of comparing demos. The demo shows what the agent can do on its best day. The contract decides what happens on the other days: when it makes a mistake, when the price changes, or when you want to leave. Nine questions, asked before you sign, cover most of that.
This article assumes you have already shortlisted a vendor. How to compare vendors and run a trial is covered in choosing an AI agent vendor, and the build-or-buy decision in the buy versus build checklist. Here we look only at what to ask and what a good answer sounds like.
How to use the questions
Send them by email and ask for written answers. Give the vendor a week. A serious supplier answers most of them without difficulty. If the answers are vague, or the reply is "it depends, we will discuss it later", treat that as information. Attach the answers to the contract, or at least keep them with it.
You do not need a lawyer to ask these questions, though one should read the final contract. The questions only make sure that the contract is written about the right things.
1. Which of our data will the agent see, and where does it go?
List what you expect the agent to read: customer emails, orders, price lists, contracts, employee data. Ask the vendor to confirm the list, name the country or region where the data is processed and stored, and name any other companies involved, such as the provider of the underlying AI model or the hosting service.
A good answer is specific. A weak one is "your data is secure". If personal data is involved, the vendor is normally a processor on your behalf, and the law expects a written agreement describing that role (GDPR, Article 28).
2. Is our data used to train or improve anything?
Ask directly whether your content is used to train the vendor's models or anyone else's, and whether this can be switched off. Get the answer for both the vendor and the model provider behind it. Also ask how long the data is kept after each task is finished.
3. What access does the agent need, and can it be reduced?
An agent that can read everything and change anything is a risk that grows with every mistake. Ask for the list of permissions it needs, and ask whether each can be limited to read-only, to one mailbox, one folder or one system. The reasoning is explained in the least access an AI agent needs. Ask also how its credentials are stored and how quickly you can revoke them yourself, without opening a support ticket.
4. What does the agent record, and can we read it?
You will need to know what the agent did and why, especially after something goes wrong. Ask what is logged, how long the logs are kept and whether you can export them in a readable form. If only the vendor can see the logs, you depend on the vendor to investigate its own product. What is worth recording is described in what to record about an agent's actions.
5. How are mistakes handled, and who is responsible?
Every agent makes errors. Ask what the vendor does to limit them, how you can report one, how fast they respond and what happens if the error costs you money or a customer. Read the liability part of the contract with this in mind. Many contracts limit liability to the fees paid in a recent period, which may be small compared with the damage. You may not be able to change that, but you should know it.
Ask also how the agent hands difficult cases to a person, and whether you control the rules for that.
6. How are changes to the agent announced?
The vendor can update the model, the prompts or the features without asking you. Ask how much notice you get, whether you can test a new version first and whether you can stay on the old one for a while. An update that changes behaviour without warning is a common source of surprises. Ask who tests after a change and how you can go back.
7. What will it really cost?
The pilot price tells you little. Ask for the price at three times and ten times the expected volume, and ask what is charged by the user, by the case, by the message or by the amount of text processed. Ask which items are extra: setup, integrations, extra environments, support, training. Ask how much the price can rise when the contract is renewed, and whether there is a cap.
Then add the hours of your own people who will look after the agent. A cheap licence with heavy internal work is not cheap. How to keep checking this after launch is shown in a monthly review of benefits and costs.
8. How do we leave?
This is the question people skip, because nobody wants to think about leaving before they have started. Ask:
- Can we export our data, settings and conversation history, in which format and at what cost?
- How long after the contract ends is our data deleted, and will you confirm it in writing?
- What is the notice period, and is there a minimum term or a penalty for ending early?
- Is the agent built on tools only this vendor offers, so that moving means starting again?
The practical side of an exit is covered in retiring or replacing an AI agent safely. It is much easier when the right to the data was agreed at the start.
9. Who do we call, and how fast?
Ask for the support hours, the response times for a serious problem and the name of a real contact. Ask what happens if the vendor is sold, changes its product or closes. A small vendor can be an excellent partner, but you should know whether your data and your process would survive its disappearance.
Reading the answers
You are not looking for perfect answers. Small vendors cannot always offer what large ones do. You are looking for three things:
- Clarity. The vendor knows the answer and states it plainly.
- Consistency. The answers match the contract and the website.
- Willingness. The vendor is ready to put the answers in writing.
If two vendors are close, prefer the one who answered the questions about data, access and leaving most clearly. Those matter more than one extra feature. For a wider structure of checks, the NIST AI Risk Management Framework offers a useful vocabulary for thinking about risks, though a small company needs only a fraction of it.
What to do before signing
- Run a small trial with limited access and non-sensitive data first.
- Put one person in charge of the agent on your side, with time set aside.
- Agree in writing on the data, access, logs, price limits and exit.
- Decide in advance what result after three months would make you stop.
A short checklist
- The vendor named where data is stored and who else handles it.
- It is clear whether your data is used for training.
- Access is limited and you can revoke it yourself.
- Logs are readable and exportable.
- The price at larger volume and the renewal limit are known.
- Notice of changes is agreed.
- Exit, data return and deletion are described in writing.
- A real support contact exists.
If a vendor cannot answer these questions, it does not mean the product is bad. It means you would be buying without knowing the terms, and that is the one thing an owner can avoid before signing.
