Agentic Development
An LLM Without a Harness: Smart Thinking vs Reliable Business Tech
Most AI agents are a smart model with nothing around it. A harness, the layer of business rules around the model, is what turns smart answers in…

The job
Sometimes residents ask simple questions with complicated answers. Can I keep a chicken coop? Who pays the insurance deductible after a leak? Can I rent my parking stall to a neighbour?
The answer to these questions depends on things the question doesn't say:
- which building, because each strata or condo has its own bylaws;
- which city, because city bylaws sit on top;
- which province, because BC, Ontario and Alberta each have their own condo law;
- who is asking, because owners and tenants often have different rules.
Businesses and consultants who are AI vanguards would suggest a general chatbot could answer those questions. Even set up with a dialogue flow, the response might sound right, but it might just as well be about some other building. For a resident looking for advice, wrong information is annoying. For condo boards, property managers or a city office, it's a liability.
How did we get here?
Think of the model as a sharp new hire. Well read, quick, good with words. Ask them anything and they'll give you a confident answer. What they don't have is the binder on your desk: which building this resident lives in, whether they own or rent, which bylaws apply, and who handles a dispute. Without it, they fill the gaps with a good guess.
The harness is that binder, plus a supervisor who checks the answer before it goes out.
- The binder holds what the business already knows. The model never has to guess the building, the city or who's asking, because the harness hands it those facts with every question.
- The supervisor checks the work. Is the quoted clause really in the bylaw, word for word? Is this a dispute that should go to someone else? If not, the answer gets corrected or flagged before anyone sees it.
That split matters because a new hire who guesses tends to guess wrong in the same direction every time. In this build, the bare model kept assuming every resident was the owner. No amount of rewording the instructions fixed it. Putting "owner or tenant" in the binder did.
Right now most of the industry is trying to make models remember more. The other option is to stop asking the model to remember what the business already knows. Stable facts belong in the harness. The model can then do what it's good at: reading the question, finding the right clause and saying it plainly.
There's one more reason to own the setup. A hosted model can be swapped or updated by its provider without notice, and the same question starts getting different answers. Running your own model doesn't make it consistent by default, but it gives you the controls. Hosted APIs are moving the other way: some no longer accept the settings that used to make outputs more repeatable.
The experiment
The source was municipal and condo bylaws, with 600 questions crafted to test against. The LLM + LoRA + custom harness enabled a level of specificity a general model alone doesn't reach. Similar to training up staff for a specific "job" in a business workflow, these purpose-built agents are able to focus and provide coherent, consistently accurate responses.
A LoRA is a small add-on to the model, trained for one job: specialist training on top of a general education. In the table below both columns use the same harness; the only difference is the custom LoRA.
| Bare model + harness | Full system (custom LoRA + harness) | |
|---|---|---|
| Right kind of reply (answer, ask, refer, decline) | 91% | 96% |
| Answer quotes the right clause, word for word | 82% | 87% |
| Vague questions: asks what you mean | 0% | 65% |
| Rated correct by at least 3 of 4 AI judges | 75% | 88% |
| States a rule that isn't in the documents | 0% | 0% |
Claude note: across the 600 questions, the judges preferred the full system on 97 and the bare model on 20 (p ≈ 3e-13; each judge alone agrees). Judging is via four different model families (DeepSeek, GLM, gpt-oss, Seed-OSS), none from the family the LoRA was trained on. They agree with each other well (Fleiss' κ 0.69).
Switching individual harness checks off shows what each one is worth. Checks on various other business-specific issues helped on vague questions, but cost a little on plain ones. The trade-off, found on the sealed test, gets written down and fixed in the next round, not quietly tuned away. As an iterative process, none of this happens quickly, but methodically, through a variety of purpose-built tests.
Give away the model, keep the binder
The two parts have different value.
- The model is the new hire's general training. It can be open and shared, like the Cooking Assistant model on Hugging Face. Releasing it openly helps a whole industry move forward, and costs the business very little.
- The harness is the binder: your rules, your documents, your way of handling customers. That's where the business value lives, so it stays private.
Because the model can be used without the harness, it has to get the basics right on its own. So problems found in testing get fixed in the model's training, not only caught by the supervisor.
How consistent is "consistent"?
The brand promise for a tool like this is simple: same question, same answer, every time. It's also the thing almost nobody measures.
The full test was run three times on the same setup.
| Matched across 3 runs | |
|---|---|
| Same decision (answer, ask, refer or decline) | 99.2% |
| Same quotes | 92% |
| Exactly the same words | 69.7% |
Claude note: this split matches the published research. Tiny numerical differences, caused by which other requests the server happens to process at the same time, rarely change a decision but often change a word. The fix is a "batch-invariant" mode in the inference engine, at some cost in speed. Running locally makes it possible; it still has to be switched on and measured.
An outside read
After the results went up, I asked Grok for an honest take. Its summary:
"Practical, applied AI engineering with real value for reliability in production settings, though not frontier research. … The approach itself (LoRA + harness/guardrails for consistency and grounding) is well-established rather than novel … The value lies in the measured, open, local-runnable specialization for concrete use cases rather than a breakthrough technique."
That's fair, and it's the point. Nothing here is a new algorithm. It's the discipline around a small model that makes it trustworthy enough to put in front of people.
What's next
Most businesses putting AI agents in front of customers aren't running the same test twice. They don't know how often the agent changes its mind, or whether last month's model behaves like this month's.
The pattern here works for any narrow job with rules that must hold: a small model for the language, a harness for the facts and the rules, sealed tests run once, more than one judge, and consistency measured rather than assumed.
If your business has rules an AI must follow every time, I can build the model and the harness around your documents, on hardware you already own. See what we build, or get in touch.
Project DNA
| Client | Humanjava Enterprises (exploratory build; private demo) |
|---|---|
| Role | Product owner and lead engineer: corpus collection, the harness and its business rules, retrieval, LoRA training, the evaluation and four-judge panel, and the orchestration of AI makers and checkers. |
| Timeline | November 2026 (in progress) |
| Stack | Qwen3-8B + custom LoRA; Qwen3 embedding and reranker search; vLLM; open-weight judge panel (DeepSeek-V4-Flash, GLM-4.7-Flash, gpt-oss-120b, Seed-OSS-36B); all on one local workstation |



