Business Strategy
Hitting Baseline: Response Continuity Is the Gap in Agentic AI
Business continuity used to mean the servers stay up. With AI agents doing real work, it also means the answers stay the same. Hosted models dri…

Over the past 6 or so months, it's become easier than ever to plug an LLM/agent into your life and have it participate in every aspect. From desktop apps to agent platforms like Hermes, forward thinkers have been leaping headlong into the AI-integrated world. Frontier models hosted in the cloud give these LLMs all sorts of superpowers, and models hosted locally open up whole other frontiers of intelligence.
For me, it started with wanting to know how to honestly encode business data within an LLM. I've been squarely focused on the "local and open source" side of building solutions. And in doing so, the shakedown of that process revealed how challenging it can be to build and ensure the data that went into the platform. Having the source inputs stay as the same data that is shared on response requires a lot of testing.
Which got me to a broader industry-wide issue. These LLMs that are out there are versioned, but they aren't exactly transparent about when the tuning, defaults or serving path underneath them change, and regressions have shipped and later been rolled back. And while projects like MarginLab are independently tracking, every day, whether AI coding agents like Claude Code and Codex are getting worse, where does that leave the rest of us?
That's the question collectively we need to start to consider. The responses from our AI agents can't be 'usage beware' based on the internet weather of the day. And even better, science fiction has already started us on the pathway to figuring it out.
The baseline test
In Blade Runner 2049, K sits a baseline test after every job. Thematically, the point wasn't/isn't whether he did the job well. It's whether he is still the same one that was deployed. It's to ensure the values and responses match the expected ones he needs to have to fulfill his role and line of work.
No surprise, AI agents need the same test, and (industry-wide) many of them don't have one.
Why answers drift
Agentic software is still very much in its alchemy age, where anyone with a bit of magic and some GPUs can build and post an LLM. The frameworks of how and what is consistent have been carefully hidden behind the LLM builders as corporate IP, and not exactly presented as an accountability for what the AI agent responds with.
- The hosted model changes. A provider updates weights, safety tuning or the serving stack behind the same model name. Even a pinned model ID can behave differently when the path around it changes. It has happened in public: OpenAI rolled back a GPT-4o update in 2025 after it made the model overly agreeable, and Anthropic has published postmortems on Claude quality drops caused by infrastructure bugs and by product-layer changes.
- The prompt and tools change. Someone edits the system prompt, adds a tool, or an agent saves itself a new "skill". Each is a quiet behaviour change.
- Sampling is random by default. Even with the randomness turned down to zero, hosted APIs don't guarantee identical answers, because results can shift with server load. Same question, same day, different answer.
None of this is a scandal. It's how these systems work. The problem of today is not knowing it happened.
Two kinds of continuity tests
Which brings us to a quiet favourite of mine, business continuity. Working in UX across large organizations, new interfaces, new access points and new platforms all require measuring some assurance that it STILL meets the needs of the business. Add in staff and team members who have a playbook/binder of the types of work they can and can't do. Business continuity is supported structurally by the platforms and by the staff. Modify one side, and the other side shows where the gaps are and where things are broken.
Looking across any business governance, continuity has a governance.
Business continuity is the familiar one: if a system goes down, the business keeps running. Backups, failover, a plan.
Response continuity is the new one: if you ask the same question next week, you get the same decision. Same refund approved, same clause quoted, same customer routed to the same place.
Many businesses using AI agents have neither for their agents. They have an API key and a prompt.
Claude note: in LangChain's survey of agent builders (1,340 respondents, late 2025), about half ran offline evaluations on a test set, fewer than 4 in 10 evaluated live traffic, and 29.5% did no evaluation at all. These are engaged teams, so the wider picture is likely worse.
Drift compounds in chains
One agent drifting is a support ticket. A chain of agents drifting is harder to see.
Rob McDonald, who works in cloud FinOps and service governance, put it this way in a LinkedIn thread on this topic:
"In a chain of microservices, a small drift at one hop gets consumed as truth by the next, and the errors compound quietly until the final output is well off baseline and nobody can say where it started."
His fix is to check at the hand-offs, not only at the end:
- Keep the data plane boring. Services pass data the way they were designed to, with a contract at each hand-off.
- Let the agent layer watch those hand-offs: compare what passed against what was expected, make small bounded corrections, or stop when it falls outside the contract.
- In his words: "The AI earns its keep as the control loop, not as the plumbing."
Older systems are the hard part, as Rob points out. Some have clean seams you can watch. Some don't, and those have to be rebuilt with the checks inside rather than bolted on.
What a baseline looks like
It's smaller than it sounds.
- A set of real tasks with known right answers. Twenty to fifty is enough to start. Pull them from real work: the refund that should be approved, the one that shouldn't, the question that should get "I need more information".
- Run them on a schedule. Daily, and after any change to the model, the prompt, the tools or the provider.
- Compare the decision, not the wording. Did it approve or decline? Did it quote the right clause? Rewording is fine. A different decision is drift.
- Ask more than once. Run each task a few times. If the answers disagree with each other, that task is unstable before anything changed.
- Keep the results. A dated log of passes and failures is your evidence when a client, an auditor or your own team asks "has it changed?"
Claude note: in one of Vergel's LLM+LoRA builds (an exploratory model for condo and city rules, written up in An LLM Without a Harness), a small tuned model inside a harness made the same decision 99% of the time when asked the same question three times. Consistency like that is measurable. It just has to be measured.
Plan for the day the API changes
Even without building a baseline test, plan for change and limit its chaos. If you're using an LLM in your work or personal life, consider documenting and treating every change as an intentional change rather than an ad-hoc change. The alternative is not knowing when, or on what date, the AI changed its answers.
Response continuity also needs a fallback:
- A pinned model you control. An open model, version-locked, running on your own hardware or a tenant you own. It may be less capable than the hosted one, but it is the same tomorrow.
- A human escalation path. When the baseline fails, work routes to a person instead of shipping a different answer.
- A switch you have tested. If you've never run your baseline against the fallback, you don't have one.
Who is set up to do this now
People running their own agents. The way to think of it is: you can hire an agent that does all the jobs, but you really want that agent to be the best at doing the one job you have it in command of. Anyone running their own agents should have enough application surface area to build tests, and explore ways to keep their agents focused on their best work in a consistent way.
Anyone running Hermes Agent from Nous Research or OpenClaw on their own machines already has most of the pieces:
- Schedules. These agents run recurring jobs. A nightly baseline is one more.
- Skills. A baseline test is a skill: a fixed set of tasks and expected decisions. OpenClaw can share skills across a team on a shared gateway, and Hermes Enterprise adds review-gated team sharing it calls Collective Wisdom, so everyone checks against the same standard.
- Logs and local control. They can see what changed, pin what runs, and keep the evidence.
- The habit of owning the stack. They already think about what runs where, and what happens when it changes.
Agents are moving into the enterprise fast. Nous Research has announced Hermes Enterprise inside Microsoft Teams and Azure. The tooling to run agents is arriving. The habit of checking them is not.
That's the opening for the self-hosted crowd. They can run baselines on their own agents today, and they're the people businesses will need when someone asks: "Is our agent still the one we deployed?"
Start small
- Pick one agent that does real work.
- Write down twenty tasks and the decision each should get.
- Run them tonight, then every night.
- When something changes, you'll know before your customer does.
- And if you're having some success, share it my way, either on X at @Vveerrgg or on LinkedIn at Vergel Evans.



