10/10
K=5 Dev Gates
0/9
Frozen Held-Out
8
Recovery Strategies
10
Failure Categories
Every moving part of Portage: the compose-stack architecture, the LangGraph node lifecycle, checkpoint + lease durability, network-off sandbox verification with anti-gaming predicates, artifact-producing plans, the recovery strategy table, the Flask → FastAPI recipe system, K-run eval methodology with explicit non-claims, and the failure taxonomy with evidence, including the frozen held-out set that scored 0/9 and is published beside the development gates rather than behind them. Every number comes from the runs/metrics tables or documented DoD scripts.
Portage is one core engine exposed through two interfaces. The autonomous agent + eval harness is the credibility engine; the MCP tools are the product wedge: build the moat first, the wedge second. The frontend never owns schema, and the CLI and dashboard are both thin REST clients: neither touches the queue or DB directly.
One core engine, two interfaces. The CLI and dashboard are thin REST clients; neither touches the queue or database directly. Jobs land in Postgres; a LangGraph worker claims them atomically with FOR UPDATE SKIP LOCKED plus a heartbeat lease, and checkpoints state after every node so a killed worker resumes instead of restarting. Verification runs in an ephemeral network-off Docker sandbox; LLM calls go through a LiteLLM driver/escalation ladder. The MCP stdio server exposes the same sandbox and graph primitives to co-pilot agents without needing the compose stack.
Autonomous migrations · developer / CI
Verified primitives for Claude Code, Cursor
Observability + eval proof for humans
How a queued job becomes a running graph, and what separates a real resume from an accidental restart at Ingest.
A job submitted via POST /jobs lands as queued; a worker claims it atomically and runs this graph with state keyed by thread_id = job_id. The runner checks checkpoint state first: no checkpoint → fresh start; pending nodes → ainvoke(None) without re-passing input; the difference between “resume” and “accidentally restart from Ingest.”
A submitted job runs this LangGraph graph, checkpointing to Postgres after every node: kill the worker mid-run and a restarted worker resumes from the last completed node. When Verify fails, Recover classifies the failure and routes back: targeted rollback + regenerate to Execute, replan to Plan for planner misses, or give up to Integrate once budgets are exhausted, reporting an honest red rather than a gamed green.
| Node | What it does |
|---|---|
| Ingest | Clone (optionally SHA-pinned, optional --subdir), snapshot as a git worktree, build the structural code graph. Runs exactly once on resume. |
| Plan | Everything is frozen here: tasks ordered by a cycle-safe SCC condensation of the real import graph (dependencies first); an interface manifest freezes every cross-file symbol's target shape; a bounded architect call proposes new artifacts and a deterministic contract compiler completes what the engine already knows; executable cuts define which files must be mutually coherent before tests can honestly run; the oracle census freezes what tests may change. Replan may append, never mutate. |
| Execute | LLM generation in dependency order, in bounded coordinated units. Every draft passes mechanical AST gates before the sandbox: contract shape, defined-vs-invented capability ownership, import direction, decorator/middleware shape, new-cycle rejection. Violations get one accounted repair call. Content-hash idempotent on resume; driver → escalation model ladder. |
| Verify | Per-cut tests in an ephemeral --network none sandbox, JUnit-parsed, stale report deleted before every run. All-skipped suites are failures (passed > 0 required); Recover sees stdout + stderr. |
| Recover | Uniquely attributable failures repair one artifact on a separate bounded ledger; otherwise replan / targeted rollback / widen-on-repeat / skip-and-continue. Failure fingerprints stop no-progress loops. |
| Integrate | Full suite as the final gate; always recomputes the migration diff from the worktree, never trusting a stale cached diff. An Integrate-only regression can route back through Recover once. |
| Report | Reloads task truth from Postgres; emits the artifact plan, oracle census, per-call cost ledger, recovery actions, diffs, and verdict. |
Honest green requires all five
A run that recovery rolls back to original sources will pass the original suite, and is scored red. That false-green class was caught live (“GREEN 24/24” with an empty diff) and fixed structurally.
State is checkpointed after every node, so a worker that dies mid-run is replaced by one that resumes instead of starting over.
The eval harness cannot SIGKILL the worker it depends on, so crash-resume is covered by demo_kill_resume.sh and the stricter dod_check.sh, separately from the K-run grid.
Durability is the core product edge: not “the LLM is smart,” but “the run survives process death and still tells the truth.” LangGraph's AsyncPostgresSaver persists state after every node; a worker that dies mid-graph is replaced by another that resumes from the last checkpoint, not from zero.
3.1: Queue + lease
The claim is a single atomic SQL pattern: UPDATE … WHERE id = (SELECT … FOR UPDATE SKIP LOCKED LIMIT 1). A job is claimable if it is queued, or running with a heartbeat older than JOB_LEASE_SECONDS (worker crashed). The heartbeat runs on its own asyncio task with its own DB connection, so a stuck graph node cannot starve the lease.
3.2: Content-hash idempotency
Each Execute step is keyed by job + task + sha256 of the written file. Resume after a mid-Execute crash skips tasks already applied instead of re-calling the model. Ingest is likewise written so resume does not re-clone or re-build the graph unnecessarily.
3.3: Durable evaluation rows
Eval identity travels in the job config, so it outlives the harness process that submitted it. Worker and harness share one idempotent runs upsert keyed by job id, and worker startup reconciles terminal eval jobs whose harness died before writing a row. Each run persists a tree_state of migrated, restored_coherent, or hybrid, and only migrated trees aggregate as green. The held-out suite closed the loop: nine jobs, nine reports, zero missing rows.
Every verification runs in a throwaway Docker container with no network, so untrusted migrated code cannot reach the host.
Verification must be isolated (untrusted migrated code must not touch the host network or sibling jobs), reproducible (same image, same pins, same offline constraint as corpus admission), and structured (JUnit parsed into totals plus failing test names). Every verify run gets an ephemeral Docker container with --network none, so no pip install at test time; hosted deployments can switch the runtime to gVisor with SANDBOX_RUNTIME=runsc.
The blast_radius primitive in action: when db.py changes, Portage walks the structural code graph outward, direct callers first (hop 1) and then their dependents (hop 2), and selects only the tests that cover the impacted set. Verify uses this to iterate fast; the final honesty bar still runs the full suite. The same query is exposed to co-pilot agents as the blast_radius MCP tool.
During iteration, Verify scopes to the tests implicated by changed files, the blast_radius query below. Scoped runs are a speed lever, never a scoring lever: the honesty bar for green still requires the full suite.
Anti-gaming predicates (learned the hard way)
| Failure mode | What happened | Fix |
|---|---|---|
| False green after skip-and-continue | Recovery rolled the worktree to originals; suite passed; report showed stale task counts and an empty diff | Report reloads tasks from Postgres; Integrate always recomputes the diff; green requires all tasks done and none skipped |
| Skip-out false pass | Model decorated every test with @pytest.mark.skip; pytest reported total>0, failed=0; Verify treated it as PASS | Require passed > 0; an all-skipped suite is a failure that must enter Recover |
| Stderr-only crashes | Conftest-chain import errors appeared only on stderr; Recover saw empty errors | Verify feeds Recover stdout + stderr |
MCP reuse
verify_patch_in_sandbox is the same sandbox contract, exposed for co-pilot use: copy → apply diff → run → return structured result, never modifying the caller's tree. That reuse is intentional: the eval proves the loop; MCP sells the loop.
Who owns what when a run goes wrong: rollback, regeneration, and replanning, plus the budgets that stop runaway loops.
Recover classifies and rolls back; Execute owns regeneration; Plan owns replanning. Inputs: the last verify output (stdout + stderr), the planned file set vs the worktree, per-task attempt counts and prior blame targets, and the budgets: max_task_attempts=3, max_recover_visits=4, escalate_after_attempts=2.
| Strategy | Trigger | Action |
|---|---|---|
| Targeted contract repair | Failure maps to exactly one frozen contract owner: missing module/export, an import-cycle edge, or a unique application traceback leaf | Roll back and regenerate only that artifact against the failure + its rejected draft + its frozen contract; siblings are content-hash skipped; the whole enclosing cut is re-verified. Runs on its own bounded ledger. Measured: a stale .decode() repaired for $0.011 without regenerating its ten-file cut. |
| Replan | An unplanned source file still imports Flask (planner miss) | Route to Plan; append the missing task(s). Fault scenario: drop_task. |
| Targeted rollback + regenerate | Crash traceback implicates specific planned files | git checkout only those files; reset tasks to pending; Execute regenerates with the failing output as context. Fault: bad_patch. |
| Widen-on-repeat | The same lone file is implicated twice running | Single-file blame isn't converging (crash site ≠ offender); widen to reset all active tasks. Rescued flaskr mid-run. |
| Behavioral retry-all | Assertions fail with no crash | Roll back and regenerate every non-skipped file task, attaching the failing output. |
| Model escalation | Task attempts exceed escalate_after_attempts | Execute switches to the escalation-tier model; every attempt records tier + model in attempts_log. Fault: bad_patch_until_escalation. |
| Skip-and-continue | A task hits max_task_attempts | Roll the file back to original source; mark it skipped; keep the run alive. |
| Give up → Integrate | max_recover_visits exhausted or nothing left to retry | Integrate + Report with an honest red. |
Self-review retries
Rolled-back attempts keep their failing diff, so retries see it (“debug your own code”) instead of regenerating blind. Measured on flaskr: the app factory went from exhausted-and-skipped after 3 blind attempts to completing all 6 tasks.
Integrity rule
Skip-and-continue can make the suite green by restoring originals. That must never score as a successful migration: green = suite green ∧ every planned task done ∧ none skipped ∧ migration_outcome = success ∧ oracle integrity 1.0 ∧ tree_state = migrated.
Coherent-cut preservation: the highest-leverage fix in the project
Before this, a single bad file inside a multi-file verification cut triggered a full rollback of every file in that cut, so one local mistake could sink an otherwise-correct ten-file migration. Recover now checkpoints the last coherent state before attempting a targeted repair and restores that, not the original sources, when the repair fails. Alongside it, one shared generation gate (caller bindings, capability ownership, import direction, cycle rejection, contract shape) runs identically across first-draft, contract-repair, and targeted-repair paths, replacing four separately-maintained checks that could silently drift apart. This is what turned watchlist and flaskr from occasional greens into repeatable ones.
Attribution beats budget (measured)
Two autopsies reconstructed failing runs byte-exact from LangGraph checkpoints and peeled them by hand. Both found the same thing: whole-file regeneration against an unattributed bug is a paid no-op. One run spent 19 recover visits and never found a two-line middleware-ordering fix, because every failure surfaced as one identical ExceptionGroup rooted in framework internals. The engineering answer was better attribution (contract ownership, import-cycle edges, unique traceback leaves), not more retries.
What a recipe declares, and why a repo it does not recognize degrades to an honest red rather than a false green.
Flask → FastAPI spans exactly the things deterministic tools cannot do reliably: routing decorators and HTTP methods, path/query/body parsing, blueprints → APIRouters, error handlers, app factory + config semantics, and the test-client seam. A recipe declares four things: detection (which files are in scope), task types and subtasks, a per-task verify_spec, and prompt guidance encoded after observed failures. A recipe that doesn't recognize the repo yields an empty plan; the run degrades safely to ingest → verify → report with an honest red.
| Subtask | Intent |
|---|---|
| app_factory | Flask()/create_app → FastAPI(); app.config as a plain dict on app.state.config; keep the factory name and shape |
| blueprint_to_router | Blueprint → APIRouter; preserve the export name importers expect |
| route_to_endpoint | @bp.route → @router.<method>; path converters → typed params; preserve status codes |
| request_parsing | request.args / get_json → typed query and body params |
| error_handler | @errorhandler → @exception_handler + JSONResponse with the same status and body |
| test_harness | Rewrite plumbing only; never delete or weaken assertions |
| templates_render / sessions_flash / auth_login | Jinja2Templates wiring · SessionMiddleware + flash equivalents · session-auth guidance (the v1 frontier) |
What recipes do not solve alone: encoded rules are cheap and effective for known idioms, but they cannot supply a target architecture, which is what artifact-producing plans (next section) exist for, and they don't cover full fidelity for every Flask extension without per-extension surface contracts.
The capability that moved the hard repos. Some migrations need entirely new modules, not rewrites of the files already there.
The capability that moved the hard repos. Three independent lines of evidence converged on the same conclusion: some migrations are unreachable by rewriting existing files. The canonical Flask tutorial app (flaskr) was migrated by hand under the identical sandbox oracle: 24/24, but the winning solution required four new modules: a contextvars request-context layer replacing g/session, a Flask-shaped test surface, a Jinja rendering layer, and a werkzeug-format password checker. On another repo, the model imported a compatibility module that did not exist; it had identified the right architecture and had no mechanism to own one, so it hallucinated the import. And correct executable cuts alone left external green at 0/4: scheduling was necessary and demonstrably not the binding constraint.
| Piece | What it does |
|---|---|
| Bounded architect call | One Plan-time call proposes 0–4 create artifacts: path, purpose, capabilities, exports, class members, consumers, dependencies. Strict JSON; deterministic validation; at most two repairs, each of which must strictly reduce the violation count. |
| Closed-choice placement | Paths are selected from a collision-free set derived from the repo's real application roots; path naming was a repeated convergence failure, so it became a selection, not free-form text. |
| Deterministic contract compiler | The recipe completes what the engine already derives: required consumers, typed module exports, uniquely-attributable class members (called ⇒ method, read ⇒ attribute). Wrong kinds are rejected, never overwritten; ambiguous ownership stays a model decision. |
| Frozen contracts | Created exports enter the same interface manifest, dependency ordering, cuts, prompts, diffs, checkpoints, rollback and reporting as rewrites. Retries, escalations, replans, and crash-resumes all converge on the same interfaces. |
| Ownership-based capability checks | A framework-shaped capability is valid only when a frozen contract owns it, the owner mechanically implements it, and the consumer is a declared one, checked receiver-aware, so a hallucination can't be laundered through a matching attribute name. |
| Provider-first topology | Consumer/dependency contradictions and proposal-level cycles are rejected at Plan; providers may not import their declared consumers; module-level providers are ordered before consumer imports. |
| Action-aware rollback | rewrite restores the worktree HEAD version; create removes the file. New files appear in git diff from the first write; the diff stays authoritative and rollback stays transactional. |
Result
The frozen plan for a flaskr migration now contains an application-owned context module exporting real g/session proxies with the test file as a declared consumer: the same architecture the manual migration used. First fully autonomous green: 12/12 tasks, 24/24 tests, zero recovery visits, $0.154, five model calls, reproduced in two further independent samples.
Measurement discipline it forced: full runs multiply two independent random variables: architect acceptance and generation quality given an accepted plan. Measuring their product makes every fix unattributable, so the harness gained a frozen-plan replay mode (~$0.2–0.5 a probe). Replays are diagnostic-only and never aggregated into headline green rates.
The oracle behind every score: a behavioral suite that passes before migration has to pass after it too.
The oracle: every corpus repo ships a behavioral pytest suite that is green on the unmodified repo, verified during admission in the same sandbox the eval uses. After migration, the same assertions must pass against the migrated app. Every (repo × scenario) cell runs K times through the real queue/worker path; per-run rows land in runs, mean±variance aggregates in metrics. Variance is reported, not smoothed: minimal-flask-api's 2/3 baseline was a finding that motivated the export-contract work, not noise.
Oracle integrity, mechanically
Test files are protected artifacts with a per-file strategy frozen at Plan. A census records test names, normalized assertion expressions, raises/parametrize, skip/xfail sites, fixture names and lifecycles. Only an explicit normalization list may differ (e.g. get_json()→json(), or an audited import swap to a plan-owned context proxy, recorded line-by-line in the report). Adversarial unit tests prove deletions, renames, added skips, and weakened assertions are all caught at Execute time, before the sandbox.
What these numbers do NOT show
| Non-claim | Why |
|---|---|
| Generality across migrations | One recipe and one language. R5 v1 measured the remaining gap directly: 0/9 on three unseen Flask repositories. The architecture is recipe-agnostic; the evidence is recipe-specific. |
| Stronger-model lift | If driver and escalation resolve to the same deployment, escalation-rescue measures the retry-ladder machinery, not a stronger model. Swapping LLM_ESCALATION_MODEL measures real lift, env-only. |
| Big-repo behaviour | Corpus repos are small (≲ ~40 files). Thousand-file horizons are unproven. |
| Immunity to prompt-tuning bias | Several recipe rules were learned from the development corpus. R5 v1 exposed the consequence rather than disproving it: the known-corpus gates stayed strong while unseen performance was 0/9. |
| Replay results as autonomous results | Frozen-plan replays isolate generation quality from architect variance. They are diagnostic, tagged as such, and never aggregated into headline green rates. |
| "Recipe-neutral" as proven | The engine contains no corpus identity and the contract machinery is framework-agnostic by construction, but neutrality is only proven by a second recipe, which is deliberately deferred. |
Where the engine converged, where it did not, and the reliability-gate history shown in full so a passing gate cannot read as rerun-until-green.
The engine that scored 13/21 (61.9%) on the July grid is not the engine running today, so that grid is kept as the baseline every later number is measured against rather than as a current claim. What closed it was one mechanism, not five separate patches: coherent-cut preservation. Every red in that grid was root-caused off its own checkpoint, and all five classes (test-harness semantic drift, two deterministic-renderer defects that died without a report, extension-surface gaps, and one import-cycle collection failure that accounted for three identical microblog runs and 77% of the grid's cost) are closed on the current engine, every fix derived from AST facts rather than from a repository, path, or test name.
Reliability-gate history, disclosed in full
Every generation the gate went through, not only the passing one. Shown specifically so a 5/5 cannot read as rerun-until-green.
| Gate generation | Flaskr (K=5) | Watchlist (K=5) |
|---|---|---|
| v1 | 2/5 | 5/5 |
| v2 | 3/5 | 3/5 |
| v3 | 3/5 | 5/5 |
| v4 (current code) | 5/5 | 5/5 |
Items, RESTX, Structural, and Minimal each independently hold their own 3/3 K=3 gate on the same code. A fresh full-corpus sweep, one autonomous sample per repo in a single sitting, came back 6/7 green: flaskr 24/24 tests at 12/12 tasks, watchlist 15/15 at 13/13, and the four smaller repos green for $0.02 to $0.08 each. The sole red is a planning-stage variance rather than a capability gap. Microblog's bounded architecture call occasionally proposes a malformed relationship graph; strict validation rejects it, the run falls back to a rewrite-only plan, four files then fail the exact contract gates that closed the import-cycle class, and the worktree is restored coherently. Replaying microblog's own accepted architecture reaches 26/26 tasks and 4/4 tests with zero recovery, which separates the two variables cleanly.
R5 held-out validation: the generalization check failed
R5 v1 froze commit 3b25ee9, corpus/heldout.toml, one network-off sandbox image, GPT-4o on both tiers, scenario baseline, K=3. The suite ran once; no red was renamed, replaced, or rerun.
| Repo | Baseline | K=3 | Terminal shape | Dominant failure |
|---|---|---|---|---|
| ws-example | 42/42 | 0/3 | 2 migrated, 1 restored | generated test-client facade shadowed FastAPI route decorators; the two migrated samples stalled at 13/42 |
| silicon | 34/34 | 0/3 | 3 restored | invalid Python signatures; raw FastAPI constructed instead of the frozen facade |
| flask-email-login | 18/18 | 0/3 | 2 migrated, 1 restored | architect missed the required context owner; the fallback left CSRF and mail providers as None |
The scoring machinery held even though the recipe failed
Strict green 0/9, architect acceptance 6/9, trees 4 migrated / 5 restored-coherent / 0 hybrid, 119 LLM calls, 19 recovery visits, $3.8643, and 9/9 durable reports with zero missing run rows. Rejected cuts restored the original suite, and those restored passes contributed zero migration score. R5 rejects the claim that the recipe is generally reliable today; it does not erase the development gates, it bounds them.
The August 4 audit, and what it did not change
R5 originally reported ws-example oracle integrity at 0.75, which reads as a generation that deleted protected tests. A forensic audit disproved it: every protected test file was byte-identical, and the score was a false positive because Execute and Report each read only the first 8 KiB of a longer test file. Both readers now inspect full content, and a >8 KiB regression covers the bug. The result does not move. All three samples were independently red: two stalled at 13/42 behind the shadowed decorators, and one restored the original tree with tasks still incomplete. Correcting a measurement is not the same as revising an outcome, and the outcome is still 0/9.
Because these failures now drive production changes, all three repositories are development inputs. The first remediation gate is green: ws-example passed strict autonomous K=1 at 42/42 tests, 5/5 tasks, tree_state=migrated, and oracle integrity 1.0. That is development evidence, not a revision of R5 v1. Silicon and flask-email-login remain the next gates, and any later held-out claim must keep this 0/9 visible, hold the untouched ClipBin reserve, and add at least two newly scouted repositories.
Ten categories, ordered easy → hard, each with a status and evidence. Statuses below are read against both corpora: several that were closed on development repos were reopened, or bounded, by what the unseen set exposed. Standing fault scenarios (bad_patch, bad_patch_until_escalation, drop_task) recovered across the development entries after the injectors themselves were fixed, and flaskr's frozen-plan drop-task diagnostic passed 3/3; recovery quality is always reported as a delta against the same repo's baseline, never as one flattering cross-repository average.
| # | Category | Status |
|---|---|---|
| 1 | Routing / parsing / responses / error handlers: pitfalls like JSONResponse status override, HTTPException body shape, 302 vs 307 redirects | SOLVED by recipe rules |
| 2 | Cross-file name contracts: dropped router export caused ~50% flake on a 3-file app | SOLVED structurally (export-contract AST pass) |
| 3 | Deprecated / hallucinated APIs: @app.on_event and invented fastapi_flash / fastapi_login are closed, but the held-out set produced a subtler collision, a generated test-client .get helper shadowing FastAPI's route decorator | PARTIAL: the class is open-ended even when each observed instance is cheap to encode |
| 4 | Environment gaps: e.g. python-multipart for Form() took watchlist from collection-crash to all 15 tests executing | SOLVED case-by-case in the sandbox image |
| 5 | App factory & config: known app.config, instance-path, lifespan and canonical test-config shapes are covered, but held-out Silicon repeatedly failed to construct the frozen TestApp facade | PARTIAL: ownership contracts need stronger realization across unseen factory shapes |
| 6 | Templates / sessions / flash / auth: the app gets an owned request-context artifact, an owned rendering layer, and correctly-ordered session middleware; flaskr migrates autonomously at 24/24 with zero recovery | SOLVED for the canonical case, architecturally rather than by prompts |
| 7 | Cross-file call-shape drift: get_db() drifting between plain function / needs-request / context manager was the dominant residual (19/24 failures in one probe) | SOLVED structurally: SCC ordering + frozen interface manifest + pre-sandbox enforcement of both DEFINES and CALLS |
| 8 | Flask-coupled extensions: flask_restx holds 3/3 and watchlist's SQLAlchemy facade holds 5/5 with real pagination, but held-out flask-email-login retained CSRF and mail owners as None and then crashed at init_app | SOLVED for development cases, OPEN generally: extension identity is not enough, every source-exercised provider must be materially realized |
| 9 | Framework-inspecting tests: flaskr passes because the plan owns real session / g / app.testing surfaces with an audited import swap; the held-out ws-example harness broke instead on a generated client facade that shadowed route decorators, and the 0.75 integrity score first blamed on it was later shown to be a truncated-reader false positive | SOLVED for flaskr; the oracle reader is fixed and regression-covered, but generation still does not preserve an unseen harness |
| 10 | Provider initialization / import cycles in multi-package apps: cycles are rejected at generation and at Verify, and microblog's accepted plan replays to 26/26 tasks and 4/4 tests | PARTIAL: gaps remain on both sides, microblog's autonomous proposal varies and held-out fallbacks never realized their providers |
What a repo must prove before it enters the corpus, and the dependency-pin finding that cost four candidates.
Admission requires: a real Flask app, a real pytest suite green on the unmodified repo in the offline sandbox, sandbox-runnable with no network, small (~≤25 Python files / ≤2k LOC for v1: reliability, not context-window heroics), permissively licensed, and SHA-pinned for remotes. The original ≥10-repo target was traded for a documented finding: a single shared sandbox image cannot serve mutually incompatible dependency pins; four candidates dropped for that shared cause; the unlock is per-repo sandbox images.
Development corpus · 7 repos, 4 tiers
| Repo | Tier | Role |
|---|---|---|
| flask-items-fixture | baseline | Bundled offline-clean Phase-2 fixture |
| flask-structural-fixture | baseline | Bundled structural fixture: factory + g/current_app SQLite + Click + blueprint; makes structural regressions catchable for ~$0.04 instead of $0.30 |
| minimal-flask-api | baseline | First real OSS repo migrated green |
| flask-restx-api | framework | Extension / marshalling wall, now 3/3 |
| flaskr (Pallets tutorial) | structural | Templates + factory + auth + CLI; the acceptance benchmark; a hand migration under the same oracle defines the target |
| watchlist | structural | flask_sqlalchemy + sessions |
| microblog | heavy | Multi-extension / long recovery |
Ten repositories now, not seven. The three R5 repositories joined after the fact, because they shaped the fixes that followed; they are listed with their frozen baselines below rather than given a difficulty tier here, since they were admitted as held-out inputs and never classified against this taxonomy.
Frozen held-out corpus
Eight new candidates were inspected statically and baseline-vetted without running Portage. Three were admitted to corpus/heldout.toml, ClipBin was frozen as reserve, and every temporary clone was deleted after admission. The one-shot result is final evidence, not a tuning loop: any held-out repository used to change the recipe moves permanently into the development corpus, and a later held-out set must be newly frozen. That rule has now been paid rather than merely stated. All three R5 repositories shaped the fixes that followed, so all three are development inputs and cannot produce held-out evidence again. ClipBin stays untouched in reserve, and the next held-out set needs it plus at least two newly scouted repositories, with this 0/9 still published beside whatever it returns.
| Repo | Pinned baseline | R5 v1 |
|---|---|---|
| ws-example | 42/42 | 0/3 green |
| silicon | 34/34 | 0/3 green |
| flask-email-login | 18/18 | 0/3 green |
| ClipBin (reserve, still unseen) | 232/232 | not run |
Sandbox accommodations stand in for each repo's own documented dev setup, never for test logic: repo root on PYTHONPATH (≙ pip install -e .), test_args scoping (≙ the repo's CI selection), documented test_env vars, and a schema-provision hook.
The full command surface and its exit codes, held to the same honest-green bar as the eval harness.
| Command | Purpose |
|---|---|
| portage migrate <repo> [--ref SHA] [--subdir D] [--recipe R] [--watch] | Submit + optional live attach |
| portage status <id> | Task tree, attempts, verdict |
| portage jobs [--limit N] | Recent jobs |
| portage report <id> [--diff] | Report JSON; optional full migration diff |
PORTAGE_API or --api selects the control plane. Exit codes: 0 honest green · 1 red · 2 usage/infra, the same bar as the eval harness.
| MCP tool | Input (conceptual) | Output |
|---|---|---|
| verify_patch_in_sandbox | repo_path, optional diff, test_args, timeout_seconds | {ok, applied, passed, tests, failing?, output_tail?, error?} |
| repo_graph | repo_path (git root) | {ok, build: full|incremental, files_parsed, total_nodes, total_edges} |
| blast_radius | repo_path, changed_files[] | Impacted files / callers / tests |
Errors return readable dicts, never protocol crashes. An empty diff means “is the suite green as-is?”; a malformed diff returns {ok: false, error: "diff does not apply"}. The MCP server is standalone: Docker and the sandbox image are required on the host, but the compose stack does not need to be up.
How local scripts stay untouched while a hosted demo avoids unbounded model spend.
Designed so local DoD scripts stay unchanged while a hosted demo doesn't get burned by unbounded LLM spend. AUTH_MODE=disabled locally (synthetic admin), github hosted; GitHub OAuth is the sole provider. Browsers get a 15-minute access JWT plus a rotating refresh cookie with family reuse-detection; machines get revocable pk_ API keys (sha256 at rest). Authorization is ownership-or-admin on every /jobs* route: 404, never 403, so nothing leaks existence. Eval endpoints stay public and aggregate-only.
| Limit | Default | Effect |
|---|---|---|
| Per-user concurrency | 1 | 429 when exceeded |
| Per-user daily jobs | 5 | 429 when exceeded |
| Per-job LLM cost ceiling | $2.00 | Remaining tasks skipped → honest red |
| Global daily spend cap | 0 (off) / set in prod | 503 “at capacity” |
Secret redaction runs at every seam where repo content leaves the sandbox (prompt context, retry errors, report diffs) via a path deny-list plus pattern scrub (agent/nodes/redaction.py). The spend ledger is the same attempts_log the eval numbers use.
Every technology choice and the tables behind them, from the queue claim down to where each run's evidence is stored.
| Concern | Choice |
|---|---|
| Language / package | Python 3.12 · import package portage_agent |
| API | FastAPI, async throughout |
| Agent | LangGraph + langgraph-checkpoint-postgres |
| ORM / migrations | SQLAlchemy 2.0 async + asyncpg · Alembic (domain tables only) |
| Database | Postgres 16 + pgvector; checkpoints via psycopg3 in the same DB |
| LLM | LiteLLM ladder (driver / escalation / cheap); provider is env config |
| Sandbox | Ephemeral Docker, --network none; gVisor (runsc) option for hosting |
| Retrieval | code-review-graph behind a Protocol (graph + blast radius) |
| Frontend | Next.js App Router, TypeScript, pnpm; REST only |
| Interfaces | CLI (portage console script) + FastMCP stdio server |
Domain tables (Alembic)
| Table | Purpose |
|---|---|
| jobs | Queue row: recipe, status, config, lease (worker_id, heartbeat_at), report paths, summaries, user_id |
| tasks | Plan DAG: file tasks + subtasks (parent_id), verify_spec, content_hash, diff, attempts_log |
| runs / metrics | Eval harness output; the leaderboard contract |
| users + auth tables | GitHub identity, role; refresh families, API keys (sha256) |
LangGraph checkpoint tables live in the same database but are created by the worker via AsyncPostgresSaver.setup(), never put in Alembic. One POSTGRES_* env derives both DSNs: postgresql+asyncpg:// for domain, postgresql:// for checkpoints. The attempts_log ledger (attempt, tier, model, action, tokens, cost_usd, failing_diff) feeds recovery timelines in the UI, escalation-rescue queries, per-job cost ceilings, global spend caps, and eval cost metrics.
Current phase: recipe excellence, not deployment
After Phase 6, external review made the call that shapes everything since: the strongest claim is depth, not breadth. Deployment was parked by decision (the repo is deploy-ready) until the recipe meets a readiness bar set before the work: JSON-API tier ≥90% green, template/session tier 70–80%, extension tier supported or honestly rejected, no fault-scenario degradation, no false greens or weakened tests, and results reproduced on held-out repositories never touched during development. The development side is largely met. R5 v1 failed the required final clause at 0/9, so the bar is not met yet, and that result is now the governing constraint rather than something met by redefinition. Next is R5.1: generalize from the held-out failure classes while preserving every development and fault gate, then freeze a fresh unseen set built from ClipBin plus at least two newly scouted untouched repositories. R5.1 is underway and its first gate is green: ws-example now passes strict autonomous K=1 at 42/42 tests, 5/5 tasks, a migrated tree and oracle integrity 1.0, closing the returned-client receiver gap on a repository that is a development input from here. Silicon and flask-email-login are the remaining gates, and none of this revises R5 v1.
The honesty bar, the budgets, and the queue claim in one block, for when you already know what you are looking for.
Source
Everything described here is in the repo. docker compose up brings up the API, worker, Postgres, and dashboard together; scripts/dod_check.sh proves the kill-and-resume claim on your machine.