The AI Grading Pipeline¶
Grading 114+ open-ended software engineering projects is fundamentally different from grading multiple-choice exams. Every team builds something different, in a different way, with a different architecture. Traditional autograders can't handle that kind of variance.
We built a pipeline that can.
The Four-Phase Pipeline¶
Phase 1: Ingestion¶
The pipeline ingests everything a team produces: - GitHub repository contents (code, commit history, branch structure) - GitHub Wiki pages (architecture docs, design decisions) - GitHub Project Boards (task tracking, sprint planning) - CI/CD logs and test results - Changelogs with semantic versioning
All artifacts are bundled into a per-team grading snapshot and stored in MongoDB for auditability.
Phase 2: Staged AI Evaluation (Extractor → Validator → Scorer)¶
The evaluation is decomposed into three LLM stages, each with a focused role:
Extractor — Parses student submissions and extracts claims, architecture decisions, and design patterns. It captures what students say they built from their reports and wikis.
Validator — Cross-references extracted claims against the repository reality. This is the "truth engine" — it compares claimed features against actual commit history, code structure, and CI/CD results. The validator also runs Code Analysis (blind to student claims) to build a semantic map of what the repository actually contains, then performs Cross-Examination to reconcile the two.
Scorer — Applies the 10-point technical rubric (Report Quality 0-2, Steel Thread 1-3, Engineering Practices 1-5) to the validated evidence and produces structured scores with reasoning traces.
Model Routing¶
Different stages use different models based on task complexity:
| Stage | Model | Why |
|---|---|---|
| Code Analysis | Local LLM (vLLM/Ollama) | High-throughput commit summarization |
| Report Analysis | Cloud Gemini | Nuanced extraction of design claims |
| Cross-Examination | Cloud Gemini | Complex reasoning about claims vs. reality |
| Rubric Scoring | Cloud Gemini | Synthesis of evidence into rubric-aligned scores |
This dual-LLM routing prevents cluster GPU throughput from competing with cloud reasoning quality.
Phase 3: Post-Hoc Calibration¶
After all teams are scored, the pipeline performs a class-wide calibration pass:
- Item Difficulty Index (IDI) analysis identifies which rubric items were too hard or too easy
- Candidate Penalty Bank surfaces every unique penalty type across all teams with its fail rate and standard deduction
- Semantic clustering groups similar penalties into canonical categories
- Statistical thresholding detects when the AI is being inconsistently harsh or lenient across teams
- Calibration produces keep/reduce/waive guidance for the human reviewer — it never rewrites scores unilaterally
Phase 4: Human-in-the-Loop (HITL) Review¶
A Streamlit dashboard presents the instructor with: - AI-proposed scores and penalties with full reasoning traces - A forensic dossier for each team — what they claimed, what the repo shows, and how the grader reconciled the two - Academic metrics computed deterministically from Git data (no LLM involvement): - Total Effort Score — volume of meaningful contributions - Code Churn Rate — how often code is rewritten - Procrastination Index — how much work happens in the final 48 hours - Gini Coefficient — how evenly work is distributed across team members - PR-to-Push Ratio — how much work flows through review vs. direct pushes - The ability to accept, modify, or reject AI-proposed penalties - Provenance tracking — every score traces back to specific submission evidence
The final output is a Canvas-ready CSV with columns Student_Email, Score, Feedback. AI does not directly post grades to Canvas.
What We Found¶
The pipeline has processed hundreds of student submissions across multiple semesters. Key findings:
- The 26-point floor: Lab 4.0's lowest score was 26/50 — the pipeline's design prevented catastrophic failure through a safety net of partial credit for setup infrastructure
- The 40-point bottleneck: A high concentration of scores at 40/50 revealed a specific late-stage integration hurdle that became a teaching target
- Gini coefficient of 0.82: In one team, a single developer did nearly all the work — the pipeline caught what manual grading would have smoothed over
- Hardcoded credentials: The pattern
michael:123@localhostappeared in real submissions, confirming the need for automated anti-pattern detection - Left-skewed distributions: Intentionally designed — teams building working systems score higher, but individual differentiation prevents freeloading
Architecture¶
The pipeline is structured as a Nested Directed Acyclic Graph (DAG) of five independent stages, each opening and closing its own MongoDB session. It is Dockerized, deployed on Kubernetes with PersistentVolumeClaims, and exposed via a Cloudflare Zero-Trust Tunnel. A CI/CD workflow auto-publishes the Docker image on main-branch merges.
Related Research¶
See the AI-Assisted Autograding research page for deep-dive architecture documentation.
Related Projects¶
- GradeFlow — Visual pipeline builder that abstracts this grading architecture.
- Capstone Survey — Student preference data ingested by the grading pipeline.
- Team Evaluation — Peer assessment scores factored into final grades.
- T4 + TA Copilot — TA workflow tools connected to the grading pipeline.