Why we test everything live before we recommend it

We maintain a structured registry of agent-harness components — memory systems, skill/tool kits, subagent patterns — as part of building Workain. The rule behind it is simple to state and easy to skip if you're in a hurry: a claim from a search-result summary, a README, or a vendor's own marketing page never gets cataloged as fact until it's re-confirmed against the primary source.

That sounds obvious. It's also the single most common corner every "landscape roundup" of AI tooling cuts, because primary-source verification is slower than reading a summary and moving on. Here's what actually doing it looks like, and why it changes what we publish.

Three checks, not one

Every component we catalog goes through the same three-lens pre-screen before anything gets written down:

• Claims match — does what the project says it does match what the code/docs actually show, fetched directly?
• Completeness — are there real limitations the project's own marketing surface doesn't mention, but its issue tracker or source does?
• Red flags — anything that looks off (a suspiciously quiet abandonment, an unresolved security report, a benchmark that doesn't reproduce)?

For anything we can run without a paid API key, that pre-screen is followed by a live run — not a read of someone else's benchmark post, an actual execution against real tasks, compared to a bare-model floor and a simple flat-file baseline.

Where this actually catches something

Most of the time, primary-source verification just confirms what the summary already said — unglamorous, and exactly the point. Occasionally it catches something real: a license field that a search snippet reported as one thing turned out, on fetching the repository's own LICENSE file directly, to be a different one entirely. Small correction, but a concrete example of the rule doing its job rather than being a policy nobody actually follows.

The live-run half of the process is where the bigger findings show up — a component's documented API missing a whole class of results, a benchmark score that looks fine in isolation until you compare it against a baseline the vendor's own materials never mention. See our memory-systems roundup for what that looks like across a full category, not just one component.