Most companies plan carefully for the day an AI agent goes live. Far fewer plan for the day something underneath it changes. The vendor announces that a model will be retired. A new version replaces the old one under the same name and answers slightly differently. A price list changes, or a usage limit is lowered. None of this is unusual; it is how the market works.
What matters for an operations or technology lead is whether such a change stops a process or only causes a planned afternoon of work. The difference is a short plan B, written while nobody is in a hurry. This article describes one that fits on a page or two.
What can change
It helps to separate the kinds of change, because each needs a different reaction.
A model is retired. The vendor sets an end date after which the model no longer responds. You have a deadline and must move to something else.
A model changes without a new name. Behaviour shifts: tone, length, how strictly it follows instructions, how it handles unusual input. Nothing is switched off, so nothing alerts you. You find out from complaints, or from a test.
Prices or limits change. The same work now costs more, or the number of requests per minute or per day is lower. The process still runs, but the budget or the speed no longer fits.
Terms change. Where data is processed, how long it is kept, or what the vendor may use it for. This is covered by your contract and data rules, not by a technical test, but it can force a switch just as firmly.
The service is unavailable for a time. A different case with a related answer: for a long outage, see incident response and kill switches for AI agents.
Step 1: Know what you depend on
You cannot plan a switch if you do not know what is being switched. For each process that uses an AI service, keep a one-page entry with:
- the process and its owner;
- the vendor, the exact model name and version, and where the setting is stored;
- the instructions and settings that shape the behaviour, such as the prompt, the temperature and any tool definitions;
- the limits you rely on and what the process costs to run;
- the notice the vendor has promised for retirement or price changes, and where that is written;
- the person who receives the vendor's announcements.
That last item is more important than it looks. Many changes are announced by email to whoever created the account. If that person has left, nobody reads the notice. A shared mailbox or a named deputy solves it. For the wider question of what your contract should promise, read AI agent vendor contracts: SLA, liability and exit.
Step 2: Decide the fallback before you need it
For every process, choose in advance what happens if the current setup is no longer available or no longer affordable. There are usually three options, and a process may have more than one in order.
Another model from the same vendor. The easiest move, but the least independent. If the vendor changes terms or pricing broadly, this may not help.
A model from another vendor. More independent, and more work: prompts often need adjusting, tools may be defined differently, and data handling terms must be checked again.
A manual or reduced route. The process continues without the agent, perhaps more slowly or for fewer cases. It is the one fallback that does not depend on any vendor, and it is the reason to keep your team's skill in the process alive. See human handoff: when the agent should stop and pass the case.
Write down for each process: the first fallback, the second, who may decide to trigger it, and what the limit is - for example, the maximum extra cost you accept before you switch. A fallback nobody is allowed to trigger is not a fallback.
Step 3: Build a test set that belongs to you
Vendor benchmarks tell you how a model does on general tasks. They do not tell you how it does on your invoices, your customer questions or your internal rules. To judge a switch you need your own test.
Collect a fixed set of real or realistic cases from the process:
- ordinary cases that make up most of the work;
- difficult cases that people sometimes get wrong;
- cases where the correct action is to refuse or hand over to a person;
- cases that touch sensitive data or rules, if the process has them.
For each, record what a good result looks like, written by the process owner. Keep personal data out of the set or handle it under the same rules as production. The set can start small and grow whenever a real mistake is found: a mistake that happened once becomes a case that is checked every time.
The same set serves two purposes. It lets you compare a candidate replacement with the current setup, and it lets you notice when the current setup quietly changes under the same name. A routine for the second use is described in scoring AI agent conversations: a weekly quality sampling routine.
Step 4: Rehearse the switch
A plan that has never been tried tends to fail on the first real use. Once for each important process, and again after any larger change, rehearse the switch in a test environment.
- Point the process at the fallback model or vendor.
- Run the test set and compare results with the current setup, case by case.
- Note what had to change: the prompt, the format of the output, the tool definitions, the timeouts.
- Check cost and speed on a realistic volume, not on a handful of requests.
- Check that logging, access rights and data handling work the same way with the new service.
- Write down how long all of it took and who did it.
The last note is the most useful. It turns "we could switch if we had to" into "we can switch in about this much time, with these people". If the honest answer is longer than the vendor's notice period, you have found the gap while it is still cheap to close.
The day a vendor announces a change
When the announcement arrives, work through a short sequence rather than reacting to the headline.
- Read it exactly. What changes, from what date, and which of your processes use it.
- Tell the owners. Each affected process owner confirms whether it is in use and how important it is.
- Run the test set. If a new version is available early, test it now. If the change is a price, calculate the effect on the process.
- Decide. Accept the change, move to the first fallback, or reduce the use of the service. Record who decided and why.
- Switch in steps. Start with a small share of cases or a shadow period in which the new setup runs beside the old one and its results are only compared. The approach is the same as in staged rollout: shadow mode, pilot and rollback criteria.
- Keep the way back open. Until the new setup has run well for an agreed period, do not remove the old configuration if the vendor still allows it.
- Watch the first days. Sample real results more often than usual and check complaints, escalations and costs.
If the change affects what customers see, such as the tone of replies or a limit on how many requests you can serve, decide whether the customer or the team needs to hear about it. See telling customers about an AI agent.
Common weak points
One person knows the setup. The prompt and settings live in a personal account or a head. Move them into a shared, versioned place.
The prompt is tuned to one model. Long, fragile instructions that work only with one model make every switch expensive. Simpler instructions with clear examples travel better.
No budget limit. A price rise or an unexpected loop can double the cost before anyone looks. Set a spending alert and a hard limit, and decide who is called when it triggers.
Nobody owns the vendor relationship. Announcements, renewals and questions should have a named owner and a deputy.
The fallback was never tested. Written plans age. Put a date for the next rehearsal in the register.
Plan B checklist
- Every AI-dependent process has a register entry with model, version, settings, limits, cost and notice terms.
- Vendor announcements go to a shared mailbox or a named deputy.
- Each process has a first and second fallback, including a manual route where possible.
- It is written down who may trigger a fallback and at what cost or risk limit.
- A fixed test set built from your own cases exists and is kept up to date.
- The switch was rehearsed, and the time and people it needs were recorded.
- A spending alert and a hard limit are set.
- After any change, the switch is made in steps and the old setup is kept until the new one has proved itself.
The aim is not to avoid every dependency, which is not realistic. It is to know your dependencies, to decide in calm what you will do when one changes, and to have tried it once. Then a vendor announcement becomes a task on the list instead of an emergency.
