Portage: Technical Deep Dive
12
Sections
7
Recovery Strategies
K=3 × 6
Eval Grid
9
Failure Categories
Every moving part of Portage: the compose-stack architecture, the LangGraph node lifecycle, checkpoint + lease durability, network-off sandbox verification with anti-gaming predicates, the recovery strategy table, the Flask → FastAPI recipe system, K-run eval methodology with explicit non-claims, and the failure taxonomy with evidence. Every number comes from the runs/metrics tables or documented DoD scripts.
Architecture Overview
Portage is one core engine exposed through two interfaces. The autonomous agent + eval harness is the credibility engine; the MCP tools are the product wedge — build the moat first, the wedge second. The frontend never owns schema, and the CLI and dashboard are both thin REST clients: neither touches the queue or DB directly.
One core engine, two interfaces. The CLI and dashboard are thin REST clients — neither touches the queue or database directly. Jobs land in Postgres; a LangGraph worker claims them atomically with FOR UPDATE SKIP LOCKED plus a heartbeat lease, and checkpoints state after every node so a killed worker resumes instead of restarting. Verification runs in an ephemeral network-off Docker sandbox; LLM calls go through a LiteLLM driver/escalation ladder. The MCP stdio server exposes the same sandbox and graph primitives to co-pilot agents without needing the compose stack.
Autonomous migrations · developer / CI
Verified primitives for Claude Code, Cursor
Observability + eval proof for humans
Job Lifecycle & Graph Nodes
A job submitted via POST /jobs lands as queued; a worker claims it atomically and runs this graph with state keyed by thread_id = job_id. The runner checks checkpoint state first: no checkpoint → fresh start; pending nodes → ainvoke(None) without re-passing input — the difference between “resume” and “accidentally restart from Ingest.”
A submitted job runs this LangGraph graph, checkpointing to Postgres after every node — kill the worker mid-run and a restarted worker resumes from the last completed node. When Verify fails, Recover classifies the failure and routes back: targeted rollback + regenerate to Execute, replan to Plan for planner misses, or give up to Integrate once budgets are exhausted — reporting an honest red rather than a gamed green.
| Node | What it does |
|---|---|
| Ingest | Clone (optionally SHA-pinned, optional --subdir), snapshot as a git worktree, build the structural code graph. Runs exactly once on resume. |
| Plan | Recipe detects framework usage, classifies each file, builds a task DAG with per-task verify_spec. An export-contract AST pass states what sibling files import. |
| Execute | Per-file LLM rewrite on the worktree. Content-hash idempotency: resume skips already-applied files. Driver → escalation model ladder after N failures. |
| Verify | Blast-radius-scoped tests in an ephemeral --network none sandbox, JUnit-parsed. All-skipped suites are failures (passed > 0 required). |
| Recover | Classify failure → targeted rollback + regenerate / model escalation / replan / skip-and-continue. Budgets bound everything. |
| Integrate | Always recomputes the migration diff from the worktree — never trusts a stale cached diff. |
| Report | Reloads task truth from Postgres; emits report with recovery actions, LLM cost, and verdict. |
Honest green requires all three
(1) the full test suite passes — not just the blast-radius subset used during iteration; (2) every planned task completed; (3) zero tasks rolled back or skipped by recovery. A run that recovery rolls back to original sources will pass the original suite — and is scored red. That false-green class was caught live (“GREEN 24/24” with an empty diff) and fixed structurally.
Durability Model
Durability is the core product edge — not “the LLM is smart,” but “the run survives process death and still tells the truth.” LangGraph's AsyncPostgresSaver persists state after every node; a worker that dies mid-graph is replaced by another that resumes from the last checkpoint, not from zero.
3.1: Queue + lease
The claim is a single atomic SQL pattern: UPDATE … WHERE id = (SELECT … FOR UPDATE SKIP LOCKED LIMIT 1). A job is claimable if it is queued, or running with a heartbeat older than JOB_LEASE_SECONDS (worker crashed). The heartbeat runs on its own asyncio task with its own DB connection, so a stuck graph node cannot starve the lease.
3.2: Content-hash idempotency
Each Execute step is keyed by job + task + sha256 of the written file. Resume after a mid-Execute crash skips tasks already applied instead of re-calling the model. Ingest is likewise written so resume does not re-clone or re-build the graph unnecessarily.
The eval harness cannot SIGKILL the worker it depends on, so crash-resume is covered by demo_kill_resume.sh and the stricter dod_check.sh, separately from the K-run grid.
Sandbox & Verification
Verification must be isolated (untrusted migrated code must not touch the host network or sibling jobs), reproducible (same image, same pins, same offline constraint as corpus admission), and structured (JUnit parsed into totals plus failing test names). Every verify run gets an ephemeral Docker container with --network none — no pip install at test time; hosted deployments can switch the runtime to gVisor with SANDBOX_RUNTIME=runsc.
During iteration, Verify scopes to the tests implicated by changed files — the blast_radius query below. Scoped runs are a speed lever, never a scoring lever: the honesty bar for green still requires the full suite.
The blast_radius primitive in action: when db.py changes, Portage walks the structural code graph outward — direct callers first (hop 1), then their dependents (hop 2) — and selects only the tests that cover the impacted set. Verify uses this to iterate fast; the final honesty bar still runs the full suite. The same query is exposed to co-pilot agents as the blast_radius MCP tool.
Anti-gaming predicates (learned the hard way)
| Failure mode | What happened | Fix |
|---|---|---|
| False green after skip-and-continue | Recovery rolled the worktree to originals; suite passed; report showed stale task counts and an empty diff | Report reloads tasks from Postgres; Integrate always recomputes the diff; green requires all tasks done and none skipped |
| Skip-out false pass | Model decorated every test with @pytest.mark.skip; pytest reported total>0, failed=0; Verify treated it as PASS | Require passed > 0; an all-skipped suite is a failure that must enter Recover |
| Stderr-only crashes | Conftest-chain import errors appeared only on stderr; Recover saw empty errors | Verify feeds Recover stdout + stderr |
MCP reuse
verify_patch_in_sandbox is the same sandbox contract, exposed for co-pilot use: copy → apply diff → run → return structured result, never modifying the caller's tree. That reuse is intentional — the eval proves the loop; MCP sells the loop.
Recovery Strategies
Recover classifies and rolls back; Execute owns regeneration; Plan owns replanning. Inputs: the last verify output (stdout + stderr), the planned file set vs the worktree, per-task attempt counts and prior blame targets, and the budgets — max_task_attempts=3, max_recover_visits=4, escalate_after_attempts=2.
| Strategy | Trigger | Action |
|---|---|---|
| Replan | An unplanned source file still imports Flask (planner miss) | Route to Plan; append the missing task(s). Fault scenario: drop_task. |
| Targeted rollback + regenerate | Crash traceback implicates specific planned files | git checkout only those files; reset tasks to pending; Execute regenerates with the failing output as context. Fault: bad_patch. |
| Widen-on-repeat | The same lone file is implicated twice running | Single-file blame isn't converging (crash site ≠ offender); widen to reset all active tasks. Rescued flaskr mid-run. |
| Behavioral retry-all | Assertions fail with no crash | Roll back and regenerate every non-skipped file task, attaching the failing output. |
| Model escalation | Task attempts exceed escalate_after_attempts | Execute switches to the escalation-tier model; every attempt records tier + model in attempts_log. Fault: bad_patch_until_escalation. |
| Skip-and-continue | A task hits max_task_attempts | Roll the file back to original source; mark it skipped; keep the run alive. |
| Give up → Integrate | max_recover_visits exhausted or nothing left to retry | Integrate + Report with an honest red. |
Self-review retries
Rolled-back attempts keep their failing diff, so retries see it (“debug your own code”) instead of regenerating blind. Measured on flaskr: the app factory went from exhausted-and-skipped after 3 blind attempts to completing all 6 tasks.
Integrity rule
Skip-and-continue can make the suite green by restoring originals. That must never score as a successful migration: green = suite green ∧ every planned task done ∧ none skipped.
Recipe System (Flask → FastAPI)
Flask → FastAPI spans exactly the things deterministic tools cannot do reliably: routing decorators and HTTP methods, path/query/body parsing, blueprints → APIRouters, error handlers, app factory + config semantics, and the test-client seam. A recipe declares four things: detection (which files are in scope), task types and subtasks, a per-task verify_spec, and prompt guidance encoded after observed failures. A recipe that doesn't recognize the repo yields an empty plan — the run degrades safely to ingest → verify → report with an honest red.
| Subtask | Intent |
|---|---|
| app_factory | Flask()/create_app → FastAPI(); app.config as a plain dict on app.state.config; keep the factory name and shape |
| blueprint_to_router | Blueprint → APIRouter; preserve the export name importers expect |
| route_to_endpoint | @bp.route → @router.<method>; path converters → typed params; preserve status codes |
| request_parsing | request.args / get_json → typed query and body params |
| error_handler | @errorhandler → @exception_handler + JSONResponse with the same status and body |
| test_harness | Rewrite plumbing only; never delete or weaken assertions |
| templates_render / sessions_flash / auth_login | Jinja2Templates wiring · SessionMiddleware + flash equivalents · session-auth guidance (the v1 frontier) |
What recipes do not solve alone: cross-file call-shape drift (contracts pin names, not signatures), deep framework-inspecting tests (flask.session, g, app.testing), and full fidelity for every Flask extension without per-extension sub-strategies.
Eval Methodology
The oracle: every corpus repo ships a behavioral pytest suite that is green on the unmodified repo, verified during admission in the same sandbox the eval uses. After migration, the same assertions must pass against the migrated app. Every (repo × scenario) cell runs K times through the real queue/worker path; per-run rows land in runs, mean±variance aggregates in metrics. Variance is reported, not smoothed — minimal-flask-api's 2/3 baseline was a finding that motivated the export-contract work, not noise.
What these numbers do NOT show
| Non-claim | Why |
|---|---|
| Generality across migrations | One recipe, one language, six repos. The architecture is recipe-agnostic; the evidence is recipe-specific. |
| Stronger-model lift | If driver and escalation resolve to the same deployment, escalation-rescue measures the retry-ladder machinery, not a stronger model. Swapping LLM_ESCALATION_MODEL measures real lift — env-only. |
| Big-repo behaviour | Corpus repos are small (≲ ~40 files). Thousand-file horizons are unproven. |
| Immunity to prompt-tuning bias | Several recipe rules were added after corpus failures; the grid partly measures a recipe tuned to this corpus. Disclosed, not pretended away. |
Failure Taxonomy
Nine categories, ordered easy → hard, each with a status and a fix direction. The fault-recovery grid on the stable tier: 100% green on the fixture (3/3 for both bad_patch and bad_patch_until_escalation), degrading on the real repo where baseline already flakes 1-in-3 and injected faults consume the retry budget the organic flake then needs.
| # | Category | Status |
|---|---|---|
| 1 | Routing / parsing / responses / error handlers — pitfalls like JSONResponse status override, HTTPException body shape, 302 vs 307 redirects | SOLVED by recipe rules |
| 2 | Cross-file name contracts — dropped router export caused ~50% flake on a 3-file app | SOLVED structurally (export-contract AST pass) |
| 3 | Deprecated / hallucinated APIs — @app.on_event; invented fastapi_flash / fastapi_login | SOLVED by rules 11/12; each new instance is cheap to encode |
| 4 | Environment gaps — e.g. python-multipart for Form() took watchlist from collection-crash to all 15 tests executing | SOLVED case-by-case in the sandbox image |
| 5 | App factory & config — config-as-plain-dict, instance path, lifespan | MOSTLY SOLVED; residual bugs in rarely-exercised branches |
| 6 | Templates / sessions / flash / auth — template apps complete DAGs, most tests pass (0.67 avg), all-or-nothing green not met | PARTIAL — the v1 frontier |
| 7 | Cross-file call-shape drift — flaskr's get_db() drifts between plain function / needs-request / context manager (19/24 failures in a final probe) | OPEN, dominant residual; fix direction: signature-level contracts or a shared interfaces step |
| 8 | Flask-coupled extensions — flask_restx reached green 1/3 (was: never collected); flask_sqlalchemy completes but fails behaviorally | OPEN / v1 boundary; needs per-extension sub-strategies |
| 9 | Framework-inspecting tests — assertions on flask.session / g / app.testing internals cannot pass unchanged against FastAPI | OPEN by design; harness rewrites plumbing, never assertion meaning |
Corpus & Admission
Admission requires: a real Flask app, a real pytest suite green on the unmodified repo in the offline sandbox, sandbox-runnable with no network, small (~≤25 Python files / ≤2k LOC for v1 — reliability, not context-window heroics), permissively licensed, and SHA-pinned for remotes. The original ≥10-repo target was traded for a documented finding: a single shared sandbox image cannot serve mutually incompatible dependency pins — four candidates dropped for that shared cause; the unlock is per-repo sandbox images.
| Repo | Tier | Role |
|---|---|---|
| flask-items-fixture | baseline | Bundled offline-clean Phase-2 fixture |
| minimal-flask-api | baseline | First real OSS repo migrated green |
| flask-restx-api | framework | Extension / marshalling wall |
| flaskr (Pallets tutorial) | structural | Templates + factory + auth-shaped flows |
| watchlist | structural | flask_sqlalchemy + sessions |
| microblog | heavy | Multi-extension / long recovery |
Sandbox accommodations stand in for each repo's own documented dev setup, never for test logic: repo root on PYTHONPATH (≙ pip install -e .), test_args scoping (≙ the repo's CI selection), documented test_env vars, and a schema-provision hook.
CLI & MCP Contracts
| Command | Purpose |
|---|---|
| portage migrate <repo> [--ref SHA] [--subdir D] [--recipe R] [--watch] | Submit + optional live attach |
| portage status <id> | Task tree, attempts, verdict |
| portage jobs [--limit N] | Recent jobs |
| portage report <id> [--diff] | Report JSON; optional full migration diff |
PORTAGE_API or --api selects the control plane. Exit codes: 0 honest green · 1 red · 2 usage/infra — the same bar as the eval harness.
| MCP tool | Input (conceptual) | Output |
|---|---|---|
| verify_patch_in_sandbox | repo_path, optional diff, test_args, timeout_seconds | {ok, applied, passed, tests, failing?, output_tail?, error?} |
| repo_graph | repo_path (git root) | {ok, build: full|incremental, files_parsed, total_nodes, total_edges} |
| blast_radius | repo_path, changed_files[] | Impacted files / callers / tests |
Errors return readable dicts, never protocol crashes. An empty diff means “is the suite green as-is?”; a malformed diff returns {ok: false, error: "diff does not apply"}. The MCP server is standalone — Docker and the sandbox image are required on the host, but the compose stack does not need to be up.
Auth & Demo Protection
Designed so local DoD scripts stay unchanged while a hosted demo doesn't get burned by unbounded LLM spend. AUTH_MODE=disabled locally (synthetic admin), github hosted — GitHub OAuth is the sole provider. Browsers get a 15-minute access JWT plus a rotating refresh cookie with family reuse-detection; machines get revocable pk_ API keys (sha256 at rest). Authorization is ownership-or-admin on every /jobs* route — 404, never 403, so nothing leaks existence. Eval endpoints stay public and aggregate-only.
| Limit | Default | Effect |
|---|---|---|
| Per-user concurrency | 1 | 429 when exceeded |
| Per-user daily jobs | 5 | 429 when exceeded |
| Per-job LLM cost ceiling | $2.00 | Remaining tasks skipped → honest red |
| Global daily spend cap | 0 (off) / set in prod | 503 “at capacity” |
Secret redaction runs at every seam where repo content leaves the sandbox — prompt context, retry errors, report diffs — via a path deny-list plus pattern scrub (agent/nodes/redaction.py). The spend ledger is the same attempts_log the eval numbers use.
Stack & Data Model
| Concern | Choice |
|---|---|
| Language / package | Python 3.12 · import package portage_agent |
| API | FastAPI, async throughout |
| Agent | LangGraph + langgraph-checkpoint-postgres |
| ORM / migrations | SQLAlchemy 2.0 async + asyncpg · Alembic (domain tables only) |
| Database | Postgres 16 + pgvector — checkpoints via psycopg3 in the same DB |
| LLM | LiteLLM ladder (driver / escalation / cheap) — provider is env config |
| Sandbox | Ephemeral Docker, --network none; gVisor (runsc) option for hosting |
| Retrieval | code-review-graph behind a Protocol (graph + blast radius) |
| Frontend | Next.js App Router, TypeScript, pnpm — REST only |
| Interfaces | CLI (portage console script) + FastMCP stdio server |
Domain tables (Alembic)
| Table | Purpose |
|---|---|
| jobs | Queue row: recipe, status, config, lease (worker_id, heartbeat_at), report paths, summaries, user_id |
| tasks | Plan DAG: file tasks + subtasks (parent_id), verify_spec, content_hash, diff, attempts_log |
| runs / metrics | Eval harness output; the leaderboard contract |
| users + auth tables | GitHub identity, role; refresh families, API keys (sha256) |
LangGraph checkpoint tables live in the same database but are created by the worker via AsyncPostgresSaver.setup() — never put in Alembic. One POSTGRES_* env derives both DSNs: postgresql+asyncpg:// for domain, postgresql:// for checkpoints. The attempts_log ledger (attempt, tier, model, action, tokens, cost_usd, failing_diff) feeds recovery timelines in the UI, escalation-rescue queries, per-job cost ceilings, global spend caps, and eval cost metrics.
Quick Reference
Source
Fully open-source · Docker Compose stack
Everything described here is in the repo. docker compose up brings up the API, worker, Postgres, and dashboard together; scripts/dod_check.sh proves the kill-and-resume claim on your machine.