10/10
K=5 Gates · Flaskr + Watchlist
6/7
Fresh Full-Corpus Sweep
0/9
Frozen Held-Out · Published As Is
0
Humans in the Loop
Give it a Flask repo and a recipe; it plans the target architecture, rewrites existing files and creates the new modules the migration requires, verifies against the repo's own tests in a network-off Docker sandbox, recovers from failures under bounded budgets, and reports honestly, including when it fails.
One core engine, two interfaces: the CLI drives fully autonomous migrations, and an MCP server hands the same verified primitives to co-pilot agents like Claude Code and Cursor. The dashboard is the proof surface: live task trees, diffs, recovery timelines, and the eval leaderboard. Below is the core durability claim, live: the worker is killed mid-migration, and a restarted worker resumes from the Postgres checkpoint instead of starting over.
reproduce: bash scripts/demo_kill_resume.sh · stricter: scripts/dod_check.sh
Most "AI migration" demos are single-shot prompts with no verification story. Portage is built around the opposite claim: a migration is only real if the repo's own tests still pass, every planned file was actually migrated, and recovery cannot game the score by giving up. The governing principle is narrow + measured beats broad + unproven. One hard migration (Flask → FastAPI) instead of a catalogue of half-working recipes. An eval harness that runs the real queue/worker path, not a mocked agent loop. A failure taxonomy with SOLVED / PARTIAL / OPEN statuses and evidence, not an all-green sheet. The autonomous + eval core is the credibility engine for the MCP product: if the verify/recover loop is measured, another agent can trust verify_patch_in_sandbox over a raw sandbox.
A submitted job runs this LangGraph graph, checkpointing to Postgres after every node: kill the worker mid-run and a restarted worker resumes from the last completed node. When Verify fails, Recover classifies the failure and routes back: targeted rollback + regenerate to Execute, replan to Plan for planner misses, or give up to Integrate once budgets are exhausted, reporting an honest red rather than a gamed green.
v1 ships one recipe: Flask → FastAPI. That target is deliberate. Routing decorators, request/response handling, blueprints → routers, error handlers, app factories, and ambient request context (g, session) need understanding, not mechanical rewriting; deterministic codemods cannot do this reliably. The architecture is recipe-pluggable; the evidence is recipe-specific by design. The capability that unlocked the hard repos: some migrations are unreachable by rewriting existing files. Flask's g/session have no FastAPI equivalent. A correct port needs a new request-context module, a test-compatibility surface, a rendering layer, and every consumer wired to them coherently. Portage plans those artifacts with a bounded architect call, freezes their contracts before generation, compiles the deterministic parts itself, and enforces that a framework-shaped capability is only valid when the plan owns and implements it, so a model can't reference a helper it wishes existed. A second wave, coherent-cut preservation, closed the gap the first one left open. One bad file inside an otherwise-correct migration used to trigger a full rollback of every file in its verification cut, so a single local mistake could sink a ten-file run. Recover now checkpoints the last coherent state before a targeted repair and restores that on failure instead of the whole migration, and one shared gate (caller, capability, import-direction, cycle, and contract checks) runs identically across every generation path: first draft, contract repair, and targeted repair alike. That is what took watchlist, a Flask-SQLAlchemy app that had never gone green, to autonomous 15/15, and pushed flaskr to a 5-for-5 reliability gate. Then the first frozen held-out evaluation supplied the correction. On three repositories that had never been migrated during development, Portage scored 0/9 strict green. The engine failed honestly: five trees restored coherently, four stayed migrated-but-red, zero were hybrid. But the recipe did not generalize. The project now has both halves of a credible result, strong development convergence and a measured unseen-repository gap, and publishes them together. A later forensic audit then corrected one of its own measurements. R5 had reported ws-example's oracle integrity at 0.75, which reads as deleted tests; byte-level reconstruction showed the protected files were identical, and the 0.75 was an artifact of Execute and Report each reading only the first 8 KiB of a long test file. Both readers now inspect full content and a >8 KiB regression covers the bug. The score stays 0/9: every sample was independently red for migration reasons. Those three repositories are development inputs now, and the first remediation gate is green, with ws-example passing strict autonomous K=1 at 42/42 tests, 5/5 tasks, a migrated tree and oracle integrity 1.0. That is development evidence, not a revision of R5 v1. One core engine, two interfaces. Autonomous mode: `portage migrate <repo> --watch` drives the full graph. Co-pilot mode: Claude Code / Cursor call verify_patch_in_sandbox, repo_graph, and blast_radius over MCP, the same verified primitives the eval numbers were measured on. The dashboard is the observability and proof surface, not the front door.
The portage console script is a thin httpx client over the REST API; it never touches the DB or queue directly, the same boundary the dashboard respects. migrate --watch streams live task transitions; status, jobs, and report --diff cover inspection. Exit codes are the eval bar: 0 means honestly green, 1 means finished but not complete-and-green, 2 means usage or infra.
The MCP server exposes the verified core so another AI agent can test its own work before writing to the caller's tree: verify_patch_in_sandbox copies the repo, applies a unified diff, runs the tests network-off, and returns structured pass/fail with failing test names, never mutating the caller's files. repo_graph and blast_radius give it structural awareness. Here is what a blast-radius query actually computes:
The blast_radius primitive in action: when db.py changes, Portage walks the structural code graph outward, direct callers first (hop 1) and then their dependents (hop 2), and selects only the tests that cover the impacted set. Verify uses this to iterate fast; the final honesty bar still runs the full suite. The same query is exposed to co-pilot agents as the blast_radius MCP tool.
Next.js App Router, REST only; the frontend never owns schema. Jobs list with launch form, job detail with live pipeline route, per-file diffs, and attempt tier/model timelines, and a public /eval leaderboard rendering per repo×scenario green rates, mean±variance, cost, and recovery straight from the runs/metrics tables.
A bounded architect call proposes new target-architecture modules; a deterministic contract compiler fills in what the engine already derives; contracts freeze before generation and bind every retry, escalation, replan, and resume. Created files get the same ordering, diffs, rollback, and cost accounting as rewrites.
A Flask-shaped capability (test_client, app_context, g, session) is accepted only when a frozen plan artifact owns and implements it, checked receiver-aware. "The model referenced a module it wished existed" becomes a pre-sandbox rejection, not a silent runtime failure.
LangGraph Postgres checkpointer after every node; worker lease with heartbeat. Kill the worker mid-run and a restarted one resumes: Ingest runs once, Execute skips already-applied files via content hashes.
Uniquely attributable failures repair the single owning artifact (measured: a stray .decode() fixed for $0.011 without touching its ten-file cut). Otherwise: targeted rollback + regenerate, widen-on-repeat, replan, skip-and-continue as last resort, all budget-bounded.
A failed targeted repair restores the last known-coherent checkpoint, not the original sources, so one bad file can no longer roll back the nine correct ones beside it. The single highest-leverage fix in the project: it converted watchlist and flaskr from occasional greens into repeatable ones.
Test files are protected artifacts: names, assertions, raises/parametrize/skip structure and fixture lifecycles are frozen at Plan; only sanctioned plumbing may differ. The guard also had to survive an audit of itself. A held-out 0.75 integrity score looked like deleted tests, but the files were byte-identical and both readers had truncated them at 8 KiB. Fixed, regression-covered, and the run stayed red on its own merits.
Green cannot be gamed by skip-and-continue, empty diffs, or all-skipped suites. Report reloads task truth from Postgres, Integrate recomputes the diff, Verify requires passed > 0, and engine errors count against the score.
First N attempts use the driver model tier; later attempts escalate. Every attempt lands in attempts_log with tier, model, tokens, and USD, so "how often does escalation rescue?" is a SQL query.
Every LLM call's tokens and USD recorded per attempt, summed per job, averaged per eval cell, with retries, escalations, and architect calls included. Cost scales with recovery, and that relationship is part of the result.
Three repositories were frozen, baseline-vetted, and unseen at the time R5 v1 ran. It ran once from a pinned commit and scored 0/9. No failed sample was renamed, replaced, or rerun, and the result is published beside the development gates rather than behind them. All three are development inputs now, so the next held-out claim needs freshly scouted repositories.
Development gates are strong; unseen generalization is not. Both halves are published together, because only one of them is a claim about the future. Green requires the full suite passing, every planned task done, zero skips, oracle integrity 1.0, and a tree_state of migrated: a run that recovery rolls back to original sources passes the original suite and still scores red.
| Evidence set | Green | What it means |
|---|---|---|
| Flaskr + Watchlist · K=5 gates | 10/10 | the hard known structural and extension apps converge repeatably |
| Items / RESTX / Structural / Minimal · K=3 | 12/12 | the smaller development tiers hold on the same code |
| Fresh seven-repo sweep · one sample each | 6/7 | Microblog red on architect variance; its accepted-plan replay is 26/26 tasks, 4/4 tests |
| Frozen R5 v1 · three then-unseen repos × K=3 | 0/9 | the recipe does not yet generalize to repositories it has never seen |
| Post-R5 ws-example · K=1 | 1/1 | first remediation gate is green, on a repo that is now a development input: not held-out evidence |
The held-out set, in full
R5 v1 ran exactly once, from frozen commit 3b25ee9 against corpus/heldout.toml, one offline sandbox image, GPT-4o on both tiers. No failed sample was renamed, replaced, or rerun.
| Unseen repo | Baseline | K=3 | Dominant failure |
|---|---|---|---|
| ws-example | 42/42 | 0/3 | generated test-client facade shadowed FastAPI route decorators; two samples stalled at 13/42 |
| silicon | 34/34 | 0/3 | invalid generated signatures; a raw FastAPI object constructed instead of the frozen facade |
| flask-email-login | 18/18 | 0/3 | architect missed the required context owner; the fallback left CSRF and mail providers as None |
The scoring machinery held even though the recipe failed
This is the part worth reading. Rejected cuts restored the original suite, and those restored passes contributed exactly zero migration score. Trees came back 4 migrated / 5 restored-coherent / 0 hybrid. All nine jobs produced durable reports with no missing run rows: 119 LLM calls, 19 recovery visits, $3.8643, architect acceptance 6/9.
The audit that followed cuts the other way, and belongs here too. R5 reported ws-example oracle integrity at 0.75, which reads as deleted tests. It was a false positive: the protected files were byte-identical, and Execute and Report had each read only the first 8 KiB of a longer test file. Both now read full content, with a >8 KiB regression. The 0/9 is unchanged, because all three samples were independently red: two stalled at 13/42 behind the shadowed route decorators, one restored the original tree with tasks still incomplete.
What convergence looks like when it works
flaskr, the canonical Flask tutorial app (templates + factory + auth + SQLite + Click CLI), went from never green in any grid to 24/24 tests, 12/12 tasks, zero recovery, for $0.15 to $0.23 a run, and now holds 5/5 at K=5. watchlist, a Flask-SQLAlchemy app that had never gone green, holds 5/5 at 15/15 tests. Both needed new modules to exist; the engine designed and wired them. What made them repeatable rather than occasional was coherent-cut preservation.
Where it stands now
All three R5 repositories are development inputs from here, so they can no longer produce held-out evidence. The first remediation gate is green: ws-example passed strict autonomous K=1 at 42/42 tests, 5/5 tasks, a migrated tree and oracle integrity 1.0. Silicon and flask-email-login are the next gates. Any future held-out claim has to keep R5 v1 visible, hold the untouched ClipBin reserve, and add at least two newly scouted repositories. The current tree passes 331/331 backend tests, Ruff on src tests, and git diff --check. Evidence current through 2026-08-04.
Every implementation choice, numbered, from the queue claim to the sandbox runtime.
Friction
Takeaways
portage migrate --watch · exit 0 = honest green
verify_patch_in_sandbox · repo_graph · blast_radius
jobs · recovery timelines · public /eval
Every moving part explained: the animated architecture, graph-node lifecycle, checkpoint and lease mechanics, sandbox anti-gaming predicates, artifact-producing plans, the eight recovery strategies, K-run eval methodology with non-claims, and the ten-category failure taxonomy with evidence.