AgentSwarm: teams of small agents, gated by a calibrated judge

AgentSwarm explores a simple question: can many small, narrowly tasked agents, each checked by others, finish real work that is too big for any single model call? Instead of one model doing everything, independent teams take on what they are good at (building, testing, reviewing, planning) and coordinate only through a shared record of the work.

How it is built: I decide what, Claude decides how

AgentSwarm follows a few earlier attempts of mine at an agent swarm. The lesson they taught: when goals and implementation details get mixed, the work drifts toward the latest detailed instruction and away from good practice. So this project splits the roles. I own the what and the why: the goals, their priority, what counts as success, what may be spent and which risks are acceptable. Claude owns the how: architecture, code, tests and the design of experiments, held to established practice and to measured evidence.

The split is written down as a short protocol that both of us follow. Claude pushes back by asking, never by overriding: when a goal is unclear, or when I start prescribing implementation details, it asks what goal is behind them. Every decision has a recorded reason, ideas I throw out are triaged rather than built on reflex, and every paid run states its expected cost before I approve it. The next step moves those decisions into the framework itself, so that running it needs only a few approvals from me along the way.

The design

Independent teams (an executor, an acceptance tester, a red team, a planner, ...) each watch one level of a shared, append-only blackboard. They never call each other. Each one reacts to the current state of the project, the way a Kubernetes controller reconciles desired and actual state, so teams can be added, removed or swapped without rewiring anything.

Work products are git branches, and they mature through levels. To move up a level an artifact has to pass a gate: deterministic checks first (tests, builds, scanners), then a few narrow yes/no questions answered by Jev, a small decision model that returns a probability rather than prose.

The split of responsibility is the core rule. Language models produce code, text, tests and plans. Jev decides whether something is in scope, good enough, or drifting. Plain code controls everything that has to be reliable: claims, merges, budgets and scheduling. Everything is typed with Pydantic end to end, and every model-backed part has a fake, so the whole system runs offline and deterministically in tests.

Checks the producer can't write

Whoever does the work doesn't decide what counts as done. A separate team writes acceptance tests from the task before anyone builds it; the builder sees them but can't change them, and the gate runs them. On a suite of 25 tasks of growing complexity, building against such tests solved twice as many of the hardest tasks (4 of 5, against 2 of 5).

The real obstacle: knowing when the work is right

Agents can only be trusted as far as their work can be checked. Where the check is indisputable, as with a proof a machine verifies or a test suite with known answers, agents can work almost unattended: anything that passes is right. Most real work has no such check. A report, a design or a plan can be fluent and still wrong, and a model checking its own work is known to be unreliable.

So for every kind of goal, the first design question in AgentSwarm is not which agents to use, but what the strongest affordable check is. It works down a ladder:

• Exact checks where they exist: compilers, type checkers, test suites with known answers.
• Checks built for the task: acceptance tests written from the task before the work starts, which the builder can't change; comparison with a reference; properties that must always hold.
• Checks that can only refute: they can't prove the work right, but they catch much that is wrong, cheaply. Every claim must cite a source, and every quote must really be on the page it cites.
• Judges: narrow yes/no questions answered by a model other than the producer, logged, scored against what turned out to be true, and allowed to act alone only where their measured precision is high enough.
• People at the final gate, and as spot checks that keep the judges honest.

The weaker the check, the more careful the process: smaller steps that are easier to judge, several independent reviewers on different models, earlier escalation to a person, and more modest claims of success. That is why the first real problems are ones with exact answers, where it can be shown beyond doubt whether the swarm works, before it is trusted with goals that can only be judged.

A first real problem

The swarm is building a CommonMark converter (Markdown to HTML) in Python from an empty stub, judged by the official specification's 652 examples. A fifth of them are held back and never shown to any agent, so progress on them shows the converter really works rather than fitting what it has seen.

The failing tests decide how the work is split: each section of the spec with failing examples becomes a work item, and a ratchet gate merges a change only if more examples pass and none that passed breaks. That carried the converter to 43%. The rest (lists, nested blocks) needed the parser rebuilt the way the spec itself recommends, so a planner split that restructuring into 19 steps, one block type at a time, each one merged only if nothing that worked broke.

The restructuring took the converter from 43% to 89% in two days, for about fourteen dollars of model calls. Back in the ordinary loop, three more work items finished it: the converter now passes 100% of the specification's examples, and 100% of the ones no agent has ever seen. The last 11% cost under fifty cents.

Five real problems

Each further problem was set up with nothing but a configuration and an importer for its official tests, and each started from an empty stub. They were picked to get harder: from data formats to a programming language to the HTML standard's parsing algorithm, which has to build the same tree as a browser for any input, broken markup included.

The held-out column is the interesting one. On jq, the builders wrote exactly what the visible tests asked for: functions the manual documents but no visible test calls were left out, and the unseen tests found them. From then on the agents got the specification itself, without its examples, and were asked to implement each rule as the specification states it. The gap between seen and unseen tests went from 13 points to 3 on YAML and to under 1 on HTML5, where the result is a complete parser in about 1,400 lines of Python.

From supervised to self-supervised

The five problems have one thing in common: each came with labeled data. Every test is an input together with the answer a correct program must give, just like the labeled examples a model learns from in supervised machine learning. The analogy goes further. The visible tests are the training set, the held-out tests are the test set, and code that passes the visible tests without solving the problem is overfitting. The jq gap above is exactly that, caught by the unseen tests.

Most real work comes without such a test suite. Self-supervised learning has the same problem and the same answer: take the training signal from the data itself. For software there are well-tried ways to do that, which need no one to write down expected answers:

• Round trips: what a writer produces, a reader must read back unchanged.
• Properties: statements that hold for every input, checked on thousands of generated ones.
• Metamorphic relations: how an output must change when the input changes in a known way.
• Differential testing: two teams, on different models, write the same program; wherever they disagree, at least one is wrong.

So the swarm was given problems with no expected answers at all: the specification only. The official test suites stayed completely hidden and were used for one thing only, to measure how far self-supervision gets.

How the swarm works without labels

• A planner splits the specification into steps, chapter by chapter.
• A check writer, on a different model family from the builder, writes checks for each step before any code exists: round trips, inputs that must mean the same, and rules the specification states. None of them needs an answer worked out by hand, because models are often wrong exactly there.
• The builder makes the checks pass. Model-written checks can be wrong, so the builder may dispute one by quoting the specification word for word.
• A referee, on a third model family, rules on each dispute: it reasons first, then gives its verdict, three times, and the majority decides. A check ruled wrong is set aside, with the reason on record.

The results

On both formats, a specification plus self-derived checks reached the same pass rates on unseen tests as labeled tests did. The extra cost is the check writing, not the building. Getting there took one lesson: in a first attempt the referee sometimes gave a verdict that contradicted its own reasoning, so wrong checks stood and blocked good work. Asking for the reasoning before the verdict, and taking the majority of three, fixed it; on YAML the referee set aside two wrong checks and kept a right one.

TOML and YAML are well-known formats, so the models' own knowledge surely helps. JSONPath answers that question: a 2024 standard the models know far less well, given to the swarm as the RFC text alone, with all 715 compliance tests hidden. It reached 709 of them, including its own module for the regular-expression standard JSONPath relies on.

Hardening: red teams that keep looking

Passing its own checks doesn't make a product right: the checks can miss things. So the swarm now has an outer loop. Once the builders are done, red teams attack the finished product, one section of the specification and one focus at a time: the rules as stated, the boundary values, the inputs that must be rejected, and places that no check covers, found by planting small bugs and seeing which ones nothing notices. A report counts only with a test that fails and the specification's own words it rests on; a referee from another model family confirms it; and each confirmed failure becomes a fix task whose test stays in the product for good. Round after round, the checks grow and the product hardens, until two whole rounds find nothing new. No red team is switched off for being quiet: a rare find can be the most valuable one.

The red teams never see the hidden tests, so those tests measure whether hardening makes the product more right, not just better at its own checks.

On jq, the red teams found from the manual alone documented functions the builders had never written, number-precision rules and regular-expression syntax. The first round on the rules of each section did most of it; later rounds kept finding real problems, but fewer of the kind the hidden tests check. On YAML, the attacks on one rule at a time found narrow cases, while the hidden failures were documents that combine features. A new focus asks the red teams for exactly that, valid inputs that mix a section's constructs with the rest of the specification, and it gave YAML its first gain.

Two safeguards came out of these campaigns. Where a specification is brief, a red team can read it more literally than it is meant; when a fix for such a report would break tests the product already passes, the referee looks again with those tests in view, and they settle what the specification means. And a fix task goes to the builders whole, to be split only if every model fails it.

Choosing models by evidence

AgentSwarm doesn't depend on one provider: any model from Anthropic, OpenAI, local Ollama, or the hundreds behind OpenRouter is one line of configuration. Which model takes which role is learned from outcomes: a router keeps a record per role and difficulty and, by Thompson sampling, picks the cheapest model whose measured success is good enough, while still trying uncertain ones now and then. A model's record in one difficulty band also informs the others (partial pooling), so a model that keeps failing doesn't get a fresh chance at every new kind of task. The aim is to minimise cost subject to a product that meets its definition of done.

A judge that is measured, not trusted

Every decision Jev makes is logged and later scored against what turned out to be true. Evaluation suites provide that truth automatically: hidden tests, mutation testing and planted faults. From the log, agentswarm calibrate reports reliability per question (ECE, Brier score), tune picks pass/fail thresholds that reach a target precision, and audit checks the active gate's false promotions, false rejections and escalations against fixed bounds using Wilson intervals.

One engine, several domains

Software and documents are configurations, not separate code. The same engine writes a handbook through draft → reviewed → final → published, with reviewers whose every complaint must quote the text. A swarm runs several projects side by side under one scheduler, each with its own budget.

Two kinds of critics

The swarm's critics come in two kinds. Red teams are adversaries: they try to make the product fail, and they must demonstrate the failure, with a failing test, a working exploit or a claim its source contradicts. Perspective reviewers look from one point of view, such as the facts, the coverage of the brief, the intended reader or consistency, and argue a shortfall, quoting the text. Demonstrated failures can be confirmed almost mechanically; arguments need a referee and a measure.

Documents, measured

To measure documents the way hidden tests measure software, the swarm writes a guide to research papers on summarization, using only the papers as sources. People who read each paper's title and abstract wrote questions about it (the Qasper dataset); a separate model answers them from the guide alone, and questions it can answer from memory are left out. Reviewers then work through the guide section by section, every finding quoting the guide or the papers, and a referee rules on each.

Most of the work so far went into making the measure trustworthy, and each step showed in the numbers. A brief that asks for what readers want to know (each paper's data, baselines, metrics and results) gave a better first draft. Deciding each question by the majority of three readings, at temperature 0, cut the scorer's own variation in half. Telling the reader which paper a question is about raised every score by about seven questions: "which is the baseline?" means the baseline of one paper, not of all ten.

With those in place, a guide to ten papers (36 hidden questions) went from 15–18 answered to 19 after four perspective reviewers, and to 17 after one broad reviewer at the same cost, every revision cited and checked by a referee. The perspectives' lead is suggestive rather than settled; the next run scores each version several times and lets every perspective finish two full rounds.

Your own documents: a report on Challenger

The same machinery now works on documents a person brings. Given three public investigations of the Space Shuttle Challenger accident (the Rogers Commission report, the House committee report and Richard Feynman's appendix, about 255,000 words) and the goal "a report for engineers on why Challenger was lost", the swarm cut the sources into parts, drafted a brief and an outline and paused for approval, then wrote each section from its own parts of the sources, citing the part, and the page where the source has pages, for every statement. The writing cost about $0.33.

Read the report: Why Challenger Was Lost (about 4,000 words, not yet reviewed).

Learning from papers: a tutorial on demand

One command now turns a handful of papers into a tutorial for one reader: agentswarm learn papers/*.pdf --know "graduate-level ML" --want "understand JEPA". Tried on five papers that trace joint-embedding predictive architectures from energy-based learning to V-JEPA 2.1 (about 97,000 words), it produced a tutorial of about 7,000 words for about $0.20: organized around concepts rather than papers, with formulas in LaTeX, a side-by-side comparison of the methods, questions to check understanding, and citations in the form (Assran et al., 2023, p. 4) with a reference list built from the papers' title pages. Read the tutorial: From Energy-Based Learning to V-JEPA 2.1.

Citation quality is measured, not assumed: agentswarm audit-citations draws cited statements at random and has a referee check each against the exact page it cites. Before review, about 65% were fully supported; one round of fact reviewers, about $0.10 a section, raised that to about 75%, with cited pages right 34 times in 38. Most of what remains is wording a little stronger than the source, the next thing to check statement by statement.

The papers themselves can be found too: agentswarm find-papers "test-time compute for LLM reasoning" has a model plan the search and name the well-known papers, looks each one up on arXiv (so every paper is real), picks five that build on one another with a line on why, and downloads them for the reader to check before the tutorial is written. On test-time compute it chose verifiers and best-of-N, self-consistency, process supervision, compute-optimal scaling and DeepSeek-R1, and the tutorial written from them cost about $0.22. Read the tutorial: Why Thinking Longer Helps an LLM Reason, and Where It Stops.

What's next

• Hardening that generalizes: the combination focus on jq, and red teams that learn from the reports already ruled on, so each round looks somewhere new.
• Citations a reader can trust: every cited statement checked against its page, not a sample, and the overstated ones rewritten.
• Goals without a specification: the swarm drafts the specification itself, and a person approves it or adds a few examples; the same loops then build and harden the product.
• When is a swarm competitive? Once the framework behaves well, the same problems at growing size, against a single large model with the same goal and checks, compared on what gets solved and at what cost.

Changelog

What changed, latest first: the problem that called for it, what was built, and what it brought.