SkillForge

Problem

Rubric-grounded grading is only as good as the rubrics, and the prototype had no way to write them — the corpus was committed to the repository, so changing a criterion meant a code change and a redeploy. Grading criteria are content, not code, and whoever owns the subject matter should be able to author them.

What I built

• An admin console for authoring challenges and skill rubrics, where saving a rubric embeds it and editing one re-embeds it, so a vector can never drift from the text it represents.
• A Go assessment API with full CRUD over challenges and rubrics, dashboard aggregates and a health probe, its OpenAPI spec generated from the handler annotations so the documentation cannot fall behind the code.
• The evaluation pipeline as a service: embed the submission, retrieve the closest rubrics by cosine similarity, run spec compliance, then logic, then a mentor that only asks questions, and weight the score 40/60 towards functional correctness.
• A learner playground with an editor, a sandboxed preview, an animated score dial and a feedback drawer that stages each agent’s findings.
• A full audit trail: every submission stores the retrieved rubrics, all three agent reports, any warnings, latency and the model names it ran against.
• A static demo page with recorded pipeline responses, so the system can be read end to end without a database, an API key or a single network call.

Technical decisions & trade-offs

• Vectors moved from a committed file into the database once rubrics became editable — retrieval is a SQL function over an indexed column, so authoring and search share one source of truth.
• Embeddings are truncated to 768 dimensions and re-normalised, because only the full-width vector arrives normalised and cosine search assumes unit vectors. A width mismatch fails loudly rather than writing a corrupt row.
• Submitted code is fenced and both code-facing agents are told to treat it as untrusted data; the mentor never sees the raw submission at all, only the structured reports, so there is no path from a submission to the agent that speaks to the learner.
• Every step degrades rather than fails: lost retrieval means the agents run ungrounded and say so, one dead scoring agent means the survivor carries the score, and a failed write still returns the learner their feedback.
• Model names, thresholds and top-k are configuration, not constants — providers retire models on a schedule, and that should be an environment change rather than a release.
• The rubric label is embedded along with the rubric body, so retrieval can match on the name of a skill and not only on its description.

Challenges solved

• A vector index on a small table quietly returned nothing while exact matches existed — approximate search partitions the space before there is enough data to partition, so the fix was to drop the index until the table earns it.
• Keeping two frontends, a Go API and a SQL schema agreed on one contract: the generated spec and typed clients mirror the domain models, so a field rename surfaces at compile time instead of in a broken response.
• The dashboard pass rate and the API’s own verdict came from two different places — a SQL aggregate and a configurable threshold — and could disagree. Documented as a constraint rather than papered over.
• Grading without executing code is an honest limit, not a feature: the logic agent reasons about behaviour, and containerised test runs are written down as the next step rather than implied.

Outcome

Open source, and the version I would hand to someone else: the prototype proved the pipeline, this one makes the rubrics — the part that decides grading quality — something a subject expert can own without touching the codebase.