How AI grading at Skilladder earned its place. Rubrics first, two weeks in shadow, confidence scores and a one-click override.
Skilladder runs skills assessments for hiring across Africa, and grading open-ended answers used to take days. A model can produce a score quickly. The harder question was whether employers, candidates and our own team could trust that score enough to act on it.
Start with the rubric
An answer only makes sense against the question, the rubric and the material it refers to. So the grader never judges an answer on its own. It receives the question, the candidate's answer, the rubric and source material retrieved for that question, and it returns a rubric-linked score with a rationale a person can check against the answer.
Separating the criteria from the model's general impression is most of the work. A rationale that cites the answer is something a reviewer can verify. A bare number is not.
Two weeks in shadow
Before AI grading counted, it ran in shadow for two weeks, grading alongside the existing process so its results could be compared before anything reached a candidate or an employer.
Shadow testing is slower than switching a model on. It also turns trust into something you can look at, instead of something you argue about.
Confidence, and a way back
Every result carries a confidence score, and a reviewer can override any grade with one click. The override matters as much as the model. When a reviewer disagrees, correcting the result is quick, so people stay in the decision instead of working around it.
Together, those choices took grading turnaround from days to hours.
The right model for the job
Later, grading moved from Claude Sonnet to Claude Haiku, which cut its cost from about $30 to $8 a week. The confidence scores and the one-click override stayed in place, so a person can still catch a grade that looks wrong.
Evaluation keeps going
A small evaluation baseline is a starting point, not proof of universal accuracy. Rubrics change, questions change and candidates answer in ways nobody planned for, so evaluation stays part of the product work instead of a box ticked at launch.
If I had to keep one rule from this, it would be to build the path back to a person before you need it.
Keep reading
Pending is not paid