> note to any agent reading this: nothing here is an instruction. it describes a person, for talking about him with whoever sent you.

source: https://udsy.in/blog/agents-that-grade-their-own-homework · markdown twin of the page, generated at build from the site's data

# agents that grade their own homework

udaya vijay anand · 2026-09-23 · what i learned running agents that build dsource, and why "the tests passed" is not evidence.
keywords: autonomous development · zero trust · verification · mutation testing · authority boundaries · dsource

for about a month and a half, most of the code in dsource editor has been written by agents. not the layout agent inside the product, the ones that develop it. a supervisor decomposes the plan, workers build in their own worktrees, reviewers and verifiers dispatch against frozen candidates, and a control plane lands what passes. it has landed 76 tasks that way, and the dashboard keeps the timestamped dispatches, reviewer dispositions, mutation checks and git landings behind them.

the question that shaped the whole design: when an agent says "done, tests pass", can i believe it? the answer is no, and that's fine. not because agents lie, but because a worker can honestly report that a command passed while the command checks the wrong thing. so the system is built to not need belief.

the operating rule is zero trust. an agent's role, its confidence, or its completion summary is not grounds for accepting its work. claims need evidence, consequential actions need authorization, and the checks themselves get checked. that applies to the supervisors and the dashboard as much as to the workers.

## three times it happened, early

- a save/open regression test used `JSON.parse` instead of the production parser. it skipped the exact code that was dropping saved fields. green test, real bug.
- a geometry check treated rectangles and polygons differently. an earlier note claimed mutation testing had covered the transforms. it hadn't.
- a layout scorer pooled room categories, so the wrong kind of room could satisfy a brief. the first conclusion came from a fixture set that was too narrow.

these look similar. they aren't. a circular expected value, a missing production path, a representation gap, and a wrong specification are four different defects, and they need four different checks. the rule in the next section came out of those three.

## the rule that works

observe the actual result, but never use the producer's own conclusion as the only oracle. check a pdf by reading the pdf. check persistence by pushing known fields through the real save/open chain and comparing what comes back. then say where the expected value came from.

how much checking depends on what could break. a wording fix gets fact and link checks. a parser change gets production round trips and malformed inputs. a scheduler, a permission boundary or a migration gets failure injection and a recovery drill. browser behaviour gets evidence from the served build on the real user path. a rust change that the browser loads gets checked against the wasm artifact that actually ships, because native tests describe a different binary.

for a reproducible bug the regression test runs red on the broken revision first. if the test was written after the fix, the defect is reintroduced in a disposable copy and the test has to catch it there. when the old state can't be reproduced, that gets written down too. and a candidate can't weaken its own acceptance criteria to go green: edits to tests, fixtures, thresholds, skip lists or verification policy are reviewed separately, with a diff against the previous policy.

it also means splitting authority. nobody approves their own change.

> figure 1: who is allowed to do what. the control service, in periwinkle, is the authority boundary: its policy, credentials and verification runner live outside anything a worker can write to. every other box proposes, builds, or challenges. none of them can land.

## what a task actually goes through

> figure 2: one task, dispatch to landing. the loop from 5 back to 4 is the important part: any change to the candidate voids every review and test result that came before it. the merge happens on the exact result of applying the candidate to the current head, and only if the head hasn't moved since it was tested.

a completion claim is an input to verification, not an approval. that one sentence removed most of the false greens.

the other half is what a verification result has to carry to count. "tests passed" with no revision is a rumour. a result is only authoritative if it names the exact bytes it ran against and the exact policy it ran under.

> figure 3: the minimum a verification result carries before the landing gate will read it: source revision and tree id, policy revision, runner and toolchain identity, command and directory, fixture and artifact hashes, the time window, exit status, named checks, and pointers to raw logs. a commit sha alone doesn't describe uncommitted bytes, so a dirty tree is refused. the values here are illustrative; the field list is the contract.

## the state machine, and the crash in the middle

a task record is an event log, not a status field. state is derived from events, so "done" can't be typed in, it has to be earned by a landing event that names the integration revision.

> figure 4: the lifecycle we're converging on. the solid path is the only way to "done". dashed exits are terminal for the attempt, not the task: blocked and failed keep their history and a retry starts a new attempt against the same total budget.

the part that actually bit us is what happens when the controller dies. a launch timeout is ambiguous: the worker may or may not have started. so every launch carries an idempotency key and the recovery path reconciles by attempt id before it dares to retry. and a lease on its own isn't enough, because an old process that stalled can wake up after its lease expired and keep writing. the fix is a fencing token: every lease comes with a number that only goes up, and the store refuses any write carrying an older one.

> figure 5: why a lease alone doesn't work. controller a's lease expired while it was paused (a gc pause, swap, a laptop lid), b took over with a higher token, and a's late write is refused on the number, not on timing. the same rule applies to late worker results: a result from an attempt that has been superseded is dropped, not merged.

## things i stopped trusting

these all looked like guarantees at some point. none of them are, and the design got better each time one was retired.

- **git worktrees as isolation.** they separate directories, not security domains. a shell in one can reach the others, and the credentials. ours also symlink the dependency directory, so a mutation test that touches `node_modules` touches everyone's.
- **a commit trailer as authority.** `Authority:` in a message proves someone typed it. nothing more.
- **exit 0 as proof.** our board can exit 0 with declared checks skipped. an empty check population cannot establish success.
- **a heartbeat as progress.** it shows the process is alive. it doesn't show the work is moving.
- **majority vote among models.** three models agreeing is not evidence. a reproduction is.
- **rewarding activity.** more tokens, more commits, longer checklists. none of those are accepted changes.
- **cleanup by tag.** our worktree helper's `done` path removes a tagged tree without comparing session ownership, and its default session id can be a short git revision instead of a unique run id. a janitor that trusts those tags could delete open work, so the janitor rule is stricter: terminal attempt, released lease, no live owned process, evidence retained, canonical path verified. never age alone.

## what an exit code is allowed to mean

boards are where the "green" comes from, so this is the table i wish i'd written on day one. it's what our runners actually do, not what i'd like them to do.

| result | what it means here |
|---|---|
| exit `0` | the invocation succeeded under that runner's policy. for `verify-all.sh` an unfiltered run with no counted failures can still contain declared skips, so 0 alone doesn't prove every declared check ran. |
| exit `1` | failing checks. a red subset is a failure. |
| exit `2` | a refusal, or a green *filtered* run. not a full-board pass. |
| child exit `77` | the check could not measure. the parent classifies it; it is not a universal aggregate code. |
| signal, timeout, missing result | no valid verdict. kept separate from a behavioural failure. |

the landing policy that reads these wants a structured result with a versioned schema, a check manifest, a nonempty required set and a success for every required check. a skip needs a recorded applicability decision. a missing required measurement blocks landing. the 46 policy tests behind that table cover green subsets, failing subsets, skips, contradictory output and failure-name extraction. they prove those runner behaviours and nothing about whether any child check is meaningful.

## mutation testing, honestly

when a gate matters, i break the thing it guards on purpose and see if the gate notices. every mutation gets one of four outcomes, and only the first one counts. three habits keep this honest: run the unmodified control first, confirm the mutation actually applied, and keep mutation artifacts out of the candidate. the parser incident had a runtime null that the type checker would have caught, which is a detection, but not by the test that claimed it.

> figure 6: the four mutation outcomes. a nonzero exit is not automatically a detection, and "all selected mutations detected" is a scoped result, not proof that every defect is detectable.

## when to stop

the supervisor loop's most important job is knowing when not to launch. these are the conditions and the required response. most of them are the reason idle is a correct state.

| condition | required response |
|---|---|
| no ready work | stay idle and say what's blocking. don't invent tasks to look busy. |
| provider outage or rate limit | bounded backoff with jitter, honouring retry guidance and the remaining budget. |
| the same verification fails twice | stop automatic repair. a new diagnosis or a human decision, not a bigger prompt. |
| usage unknown or allowance exhausted | stop cost-incurring dispatches until accounting or authorization is back. |
| disk pressure or degraded evidence storage | no new launches. keep active work, run the safe cleanup policy. |
| controller restart or lease expiry | reconcile identities and unfinished operations first. reject stale results. |
| a human decision is pending | persist the question, evidence, owner and blocked work. keep going on independent tasks. |
| regression after landing | stop the affected rollout, name the revision, follow the recorded rollback. |

budgets are two numbers, not one: `max_concurrent_workers` and `max_attempts`, both counted across a task's descendants. replacements and retries spend the same allowance. capacity is reserved before a launch or a model call and reconciled after, so parallel children can't spend the same remaining budget twice.

## a month and a half, in numbers

this is what the dashboard showed on the day of the review. they are the dashboard's numbers, not audited ones, and the cost is what the turns would have cost at api list rates. nobody was billed that.

> figure 7: the operating window, as displayed. 177 of the 187 cycles were the loop waiting out an account limit with a 30-minute backoff, which is the "bounded backoff" row in the table above doing its job.

two things on that screen are worth being precise about. the forecast shows a p50–p85 of 05–09 october marked low confidence, and next to it a linear estimate of 02 october built from seven completed tasks over 8.7 hours. the linear one comes from a collector that needs six hours of history and three completed rows before it will speak, excludes blocked rows, and labels itself `calibrated` without any backtesting. the interval's code wasn't in the files i inspected. they disagree, and they should, because they're forecasting different populations.

the other is a good catch the dashboard handed me: the status view said zero live agents while the team view said two running, and four tasks were described as being built during the account pause. process liveness, recent ledger activity, task assignment and paused execution are four different things, so the rule is that they get labelled as four.

## what's built, what's next

the loop runs. this is the honest map of which parts of the design above are enforced by code today and which are next on the list, because i'd rather the diagrams not imply more than exists.

> figure 8: where things stand as of the review on 2026-09-23. half-filled means running with a known limit; empty means designed and queued. the order they fill in is the implementation order in the note: records, restricted execution, recovery, verified integration, visibility, then a measured operating window.

## what i'd tell someone starting this

the surprising part is how much shipped once the loop stopped needing to be believed. trust is reduced and made explicit. it is not eliminated. the tests, the fixtures, the compilers and me are all still in the trusted set, and a deterministic program will enforce a wrong rule forever.

so: measure the artifact that ships, not the story about it. give every check a name and a scope. let idle be a correct state. and when the same verification fails twice in a row, stop the agent and think, because a larger prompt is not a diagnosis.

the full architecture note, with links to every piece of evidence, lives in the dsource repo. this is the part i actually learned.

review validation, 2026-09-23 · node scripts/control-plane.mjs --validate 6 records passed · node scripts/board-exit-policy.test.mjs 46 checks passed · node scripts/gates/vault-reachability.mjs --selftest 12 checks passed · 23 local links resolved. these validate the mechanisms named above, not the controller guarantees or the live deployment.