DevSkill Arena

Problem

Automated code feedback usually does the least useful thing — it hands over a corrected answer, and the learner takes the fix without the reasoning. I wanted grading that is rubric-grounded rather than impressionistic, and feedback that asks the questions a good reviewer would ask.

What I built

• A code playground with live preview for markup challenges, rendered in a sandboxed iframe with scripts allowed but same-origin denied.
• A three-agent evaluation pipeline in a single route handler: spec compliance, logic tested one assertion per numbered constraint, and a mentor that only ever asks questions.
• Retrieval over a rubric corpus: 768-dimension embeddings, L2-normalised, cosine similarity, a strict threshold and a small top-k.
• A print-quality report built from client state — no PDF library, no server round trip, no extra model calls.
• A project that runs on one install: no database, no backend service, no infrastructure.

Technical decisions & trade-offs

• Corpus vectors are precomputed and committed, then read from disk rather than imported, so the app runs for anyone who clones it and the vectors can never be pulled into a client bundle.
• L2 normalisation is mandatory at truncated embedding widths: only the full-width vector arrives pre-normalised, and cosine search assumes unit vectors.
• Agent calls declare a response schema rather than relying on a JSON mime type — constrained decoding fixed intermittent unparseable output from the longest-prompt agent.
• Graceful degradation over hard failure: a failed step marks the result degraded and appends a warning; only both scoring agents failing returns an error. Partial feedback beats an error page.
• Rubrics are the corpus and challenge text is never embedded, so retrieval ranks grading criteria instead of restating the question.

Challenges solved

• Printing correctly: the report is portalled out to the document body because fixed and transformed ancestors paginate unreliably, and page breaks are applied per finding rather than per section so browsers stop ejecting blank pages.
• Proving the API key stays server-side rather than hoping it does — server-only module guards, verified against the production bundle.
• Token budgeting: reasoning tokens draw on the same ceiling as output, so a tight limit silently starved the agent carrying most of the score.

Outcome

Open source under MIT, and a working answer to whether retrieval-grounded, multi-agent assessment can be honest about its own uncertainty. It proved the pipeline well enough to rebuild it as a full system with an authoring console behind it.