SkillForge

Problem

Rubric-grounded grading is only as good as the rubrics, and the prototype had no way to write them — the corpus was committed to the repository, so changing a criterion meant a code change and a redeploy. Grading criteria are content, not code.

What I built

• An admin console for authoring challenges and rubrics, where saving embeds and editing re-embeds, so a vector can never drift from the text it represents.
• A Go assessment API with full CRUD, dashboard aggregates and a health probe, its OpenAPI spec generated from the handler annotations so the docs cannot fall behind the code.
• The evaluation pipeline as a service: embed the submission, retrieve the closest rubrics by cosine, run spec compliance, then logic, then a mentor that only asks questions — weighted 40/60 towards functional correctness.
• A learner playground with an editor, a sandboxed preview, an animated score dial and a feedback drawer that stages each agent’s findings.
• A full audit trail: every submission stores the retrieved rubrics, all three agent reports, warnings, latency and the model names it ran against.

Technical decisions & trade-offs

• Vectors moved from a committed file into the database once rubrics became editable — retrieval is a SQL function over an indexed column, so authoring and search share one source of truth.
• Embeddings are truncated to 768 dimensions and re-normalised, because only the full-width vector arrives normalised and cosine assumes unit vectors. A width mismatch fails loudly rather than writing a corrupt row.
• Submitted code is fenced and both code-facing agents treat it as untrusted data; the mentor sees only the structured reports, so nothing runs from a submission to the agent that speaks to the learner.
• Every step degrades rather than fails: lost retrieval means the agents run ungrounded and say so, a dead scoring agent means the survivor carries the score, and a failed write still returns the feedback.
• Model names, thresholds and top-k are configuration, not constants — providers retire models on a schedule, and that should be an environment change rather than a release.

Challenges solved

• A vector index on a small table quietly returned nothing while exact matches existed — approximate search partitions the space before there is enough data to partition. The fix was dropping the index until the table earns it.
• The dashboard pass rate and the API’s own verdict came from two places — a SQL aggregate and a configurable threshold — and could disagree. Documented as a constraint rather than papered over.
• Grading without executing code is an honest limit, not a feature: the logic agent reasons about behaviour, and containerised test runs are written down as the next step rather than implied.

Outcome

Open source, and the version I would hand to someone else: the prototype proved the pipeline, this one makes the rubrics — the part that decides grading quality — something a subject expert can own without touching the codebase.