The Tests Were Green. The Code Was Broken.

What happened when we pointed a swarm of adversarial AI agents at our own code — and found 83 bugs before a single one shipped.

A couple of months ago I stepped into the world of Open Source Development. Having explored this Nostr NIPs repo; I wanted to create something login related and make that the cornerstone of everything I intend on building.

And with that I started to build. Every test suite passed. Every package built. Every version number was ready to publish. But agentic coding in these early days REALLY relies on reading the code, or trusting the process.

And in places, the code underneath was completely, quietly broken. Lots of errors, but not a lot of clarity on where they were. Or at least not for a vibe-coder like me. The libraries (and plugins I built based on the libraries) were signing data the wrong way, encrypting messages to the wrong recipient, deriving a different identity from the very same secret depending on which library you used. Not because tests were missing. Because the trusting the process without proper verification does not deliver well written code.

But stick with me here, the realization; the process, while not perfect, can get us to high value and useful code. And in some ways it's why the method, matters more than the bugs.

A library CAN looks perfectly right & be wrong

Starting with this uncomfortable insight, everything follows from it.

Asking an LLM to write test plans, create smoke tests, and run QA against the code it has written might seem obvious. But the details are where the issues arise.

Imagine a library that encodes a value and then decodes it. If the encoder and decoder share the *same* mistake, the round trip works flawlessly. Encode, decode, you get your value back, the test goes green. The output is still wrong for anyone on the outside trying to read it. BUT THE TEST STILL RETURN GREEN ✅ .

Now multiply that across a family of libraries that were each QA script is tested against themselves. The bug is invisible precisely because the code is self-consistent. It agrees with itself. It just doesn't agree with the rest of the world.

And the result; a codebase that is uniformly confident and, in a handful of load-bearing places, AND uniformly wrong. Green checkmarks everywhere. Real interoperability nowhere.

That has been the past couple of months of all the Nostr developer libraries hosted here on LoginWithNostr.com.

An origin story with the best intentions

Passion projects have a funny way of becoming real. For me, it's been that idea that Email/Password is dying. That we need a new way to manage account access. The more I learned about Nostr, the more I got excited. And this project of mine began as something modest: put together a family of open-source libraries that would enable anyone to migrate a web application to a Nostr key-pair based authentication profile.

At the time, the collection of libraries were somewhat limited, and didn't have the coverage I was use to. So why not build focused tools (for these vibe-based ideas) and publish for anyone to use. Figuring that someone had to spend the tokens, why not start here. And better off, align on a convention and make them feel like one coherent set instead of a drawer of loose parts.

Partway through, as what happens with vibe-coding, the drift of LLMs started to land. Drift as not a specific failure, but as a texture on how the code was coming to life. Where ideas start to blur between the does this work, and is this the right way to make it work. The kind of thing you can't point to but can't unsee.

Across projects, small assumptions didn't quite line up. And this is where it's better to work with an LLM, rather than direct an LLM. With Claude Fable & Claude Opus as co-writers, we figured something things out.

The type/genre of work I would give this type of effort would be something like LLMUX. Where the conversation is less about the 'framework of SKILL files' and more about the alignment of project strategy. And in hindsight, was the most important decision in the whole effort.

The code became our relatable way to create lower entropy and increase or cohesion on what a well-formed thought / project should be.

Cohesion is a swarm of agents

Instead of the prompt / response process of the previous age of vibe coding. Where we'd review and reading one file at a time; we ran an adversarial audit. Kicking off dozens of AI agents, each assigned a corner of the codebase, each with a single mandate; find what's actually broken, and prove it.

Two design choices made it work.

• Fan-out - Every library got its own investigator, running in parallel. What would take many "people weeks" of careful reading happened in an afternoon, in breadth no individual could hold in their head at once.
• Adversarial verification - A new'ish way of validating code, is the part that mattered most this time around. Every suspected bug was handed to independent sub-agents whose job was to *refute* it, and prove it wasn't real.

When the dust settled, 83 bugs survived verification. Not CSS style issues; real correctness failures, ranked by severity, each with a concrete way to reproduce it. Only the findings that survived that gauntlet were kept. It's the difference between a brainstorm and a courtroom. Suspicion is cheap; we only wanted verdicts.

What did 83 bugs actually looks like

We're describing these by 'class' rather than naming names. As with every multi-package project, there are the failure modes hiding in a lot of real-world code, ours included:

• Signing done the wrong way. Cryptographic signatures produced with the wrong algorithm entirely; output that every counterpart in the ecosystem would reject, sitting behind tests that only ever checked the code against itself.
• Encrypting to the wrong key. A subtle argument-order mistake "your key and their key swapped" copy-pasted across several libraries. Every message it produced was unreadable by anyone else. The round-trip test passed anyway.
• Two secrets, two identities. The same recovery phrase produced *different* identities in two of our own libraries, because one followed the standard derivation and the other didn't. Back up in one, restore in the other, and you'd be locked out of your own account, and with no error to warn you.
• A trust boundary that didn't hold. An authentication path that could be satisfied without actually proving what it was supposed to prove.
• Speaking a protocol incorrectly. A networking layer that formatted its messages in a shape real servers silently reject, so it looked like it worked in tests and did nothing on the open network.

None of these were exotic. They were the ordinary consequence of code that had never been checked against an outside source of truth. The real deliverable is in making sure these *class* of bug could never hide again.

The root cause was the same everywhere: nothing tested the code against a known, external, correct answer. One thing that's great about agentic coding. Frameworks are easy to derive if you know how to ask.

The result: a shared set of known-answer vectors, creating fixed inputs with their independently-verified correct outputs, and drawn from the published standards. Committed once, shared across every library and every language; now we're building better cohesion. Each agent or sub-agent in the library doesn't get to grade its own homework. It has to match the answer key the rest of the world uses.

That's the durable part. The bugs were symptoms. The missing answer key was the disease.

Across repositories, code bugs can't hide

The twist that justified the whole approach; after all the fixes, every individual library was green again. Every test passed, in every package. Using a unified set of references that sits above any one repository of code. By any per-project measure, we were ready to ship.

The magic task that levelled up the collection; "Run a coherence review". Checking not just each library in isolation, but whether they still fit together.

That review found release-wrecker(s) that no per-library test could ever see: the packages were still configured to depend on the *old, broken* versions of each other.

If the collection was published as-is; everything would have installed itself right back into the bugs we'd just fixed. Green everywhere. Broken on delivery. The lesson in miniature, correct parts don't guarantee a correct whole. Someone has to check the seams.

A vote for LLMUX and the 'Who did what'

It's tempting to tell this as "AI fixed our bugs." That's not what happened, and the real division is the whole point.

• *The LLM (Claude Fable & Claude Opus) read twenty-odd codebases, found and proved 83 defects, wrote the fixes and the tests that lock them in, and re-checked the entire set for coherence. All performed at a scale and speed no single engineer could match.
• The human (me / Vergel) decided the things that couldn't be delegated: to stop before shipping. To hold the bar at "correct," not "passing." To spend the effort building an answer key instead of patching symptoms. To refuse to ship a set of tools we wouldn't build on ourselves.

Agents and sub-agents to supplied the leverage. The judgment of what "done" means, what's worth doing, and when to hold the line; stayed with the me. The keeper of the spark of ideas always has to pilot the project to its conclusion.

Shipped !!! - 'Login With Nostr • Developer'

One coherent, genuinely interoperable set of libraries. Every fix backed by a real regression test. A shared answer key so the whole family is measured against the same external truth. And a release worth celebrating.

If there's a single takeaway for anyone building with AI right now, it's this: these LLM agentic tools give you extraordinary leverage over 'the work'.

These tools don't give you leverage over the judgment. And in all honesty, they're not supposed to. With the right prompt and ultracode plan, an LLM can find the bugs in the code. Where the difference is made. It's in deciding that code coherence matters, that you'd rather find the bugs across the collection of projects and make that the definition of job done.