Skip to content

AI-Assisted Autograding for Software Engineering Capstones

While Test Forge focuses on generating practice questions, the CSED Lab also investigates how AI can evaluate open-ended student work — specifically the complex, multi-faceted lab assignments in CSE120's software engineering capstone.

The Challenge

Traditional autograders work well for multiple-choice or simple coding exercises, but capstone-level software engineering projects involve architecture diagrams, requirements analysis, system design, and team-based deliverables. These open-ended artifacts resist simple rubric checking.

Our Approach: Stage-Decomposed LLM Evaluation

We developed a pipeline that decomposes evaluation into distinct LLM stages, each with a focused role:

  1. Extractor — Parses student submissions and extracts claims, architecture decisions, and design patterns
  2. Validator — Cross-references extracted claims against the assignment requirements and consolidates duplicate penalties sharing identical source quotes
  3. Scorer — Applies the rubric to validated evidence and produces structured scores

The pipeline's scoring logic follows a two-pillar architecture: Deterministic Rules (binary GitHub checks for infrastructure, worth 5 pts) operate alongside Qualitative Evaluations (a staged Extractor-Validator-Scorer LLM pipeline for the Action Plan, worth 15 pts, and Blueprints, worth 30 pts), ensuring objective pass/fail checks coexist with nuanced rubric-based assessment.

Key Features

Blueprint Routing: The system automatically detects submission type (Mermaid diagrams, ERDs, mockups, architecture documents) and routes to specialized evaluation paths.

Model Profile Routing: Different LLM profiles are used based on task complexity — smaller models for pattern extraction, frontier models for nuanced architectural evaluation. The pipeline uses a mixed-model Gemini routing strategy: flash-lite for text extraction, flash for multimodal diagram evaluation, and pro for adjudication and cross-examination.

Post-Hoc Calibration: Raw AI scores are adjusted using item analysis and penalty calibration to ensure fairness across different graders and assignment versions. Calibration employs the Item Difficulty Index (\(p\)) from psychometrics — class-wide failure rates exceeding 75% trigger automatic penalty waivers, shifting mean scores from 13.24 to 28.52 in observed deployments. Calibration operates over a canonical penalty bank derived by semantically clustering candidate observations (e.g., 402 raw candidates consolidated into 33 distinct penalty types), each with a fail rate, standard deduction, and calibrated deduction.

Human-in-the-Loop: A Streamlit dashboard allows instructors to review AI-proposed penalties, override decisions, and trace evaluation provenance back to specific submission evidence.

Research Status

This pipeline has been deployed and iterated across multiple semesters of CSE120 at UC Merced, processing hundreds of student submissions with a five-phase architecture that separates data ingestion, LLM evaluation, calibration, HITL review, and final export.

  • GradeFlow — Visual grading pipeline builder informed by this autograding architecture.
  • Test Forge — AI-based test generation engine for complementary assessment workflows.
  • Capstone Survey — Collects student project preferences feeding into the grading pipeline.
  • Team Evaluation — Peer evaluation platform used alongside the grader.

Candidate-Only AI Penalty Mode

Rather than auto-applying deductions, the pipeline emits AI-detected penalties as unchecked candidate suggestions. The reviewer sees many potential penalty items in the HITL dashboard and decides which ones to enforce. Calibration then operates over this candidate bank, producing keep/reduce/waive guidance for the reviewer — it never rewrites final scores unilaterally.

Feature Graph Traceability (Cross-Lab Contract)

The grader traces feature promises from earlier lab assignments (requirements, user stories, wireframes, architecture) into the current submission's Project Board and Wiki. It extracts anchor objects and deterministically scores lexical coverage against the repo's issues, PRs, and discussion corpus. These cross-lab traceability edges enable queries like "which teams failed to trace feature X across all three labs?"

Benchmark Harness & Provenance Tracking

A regression testing framework runs the grader against a fixed multi-team suite before full-class execution. Each run records model name, prompt hash, evidence hash, runtime, output hash, and route-level metadata. Prompt changes or model swaps are compared against stable benchmarks without relying on intuition — drift is detected automatically.

Mermaid-First Evidence Extraction

Mermaid.js code blocks are detected during GitHub Wiki ingestion (before LLM evaluation), persisted on the repository record, and carried into the grading snapshot for direct rendering in the Evidence Tab. This decouples evidence capture from scoring — the system stores what it found before deciding how to grade it.

SVG Blueprint Rasterization

SVG diagram assets encountered during blueprint evaluation are rasterized to PNG before being sent to multimodal vision models. The pipeline first attempts CairoSVG, then falls back to resvg-py, ensuring SVG-reliant teams are not silently penalized due to missing system libraries.

Identical-Penalty Consolidation

Before the scorer computes final deductions, the validator stage consolidates duplicate penalties that share the exact same source quote. This prevents mechanical double-counting when the extractor produces overlapping findings for the same observation.

DB-Backed Review Persistence

Review state is no longer session-only. Grading snapshots are persisted as GradingRun documents in MongoDB; reviewer overrides are stored in Review (latest state) and ReviewRevision (append-only history). The Streamlit UI restores saved review state on load and autosaves committed changes.

Canvas CSV Export

The pipeline's output boundary is a Canvas-ready CSV with columns Student_Email, Score, Feedback. The export is generated on demand from the current reviewed (and potentially calibrated) state, preserving the HITL principle that AI does not directly post grades to Canvas.

Cloud-Native DevOps

The grader is Dockerized, deployed on Kubernetes with PersistentVolumeClaims for grading artifacts, and exposed via a Cloudflare Zero-Trust Tunnel. Secrets (Gemini, GitHub, MongoDB, Figma API keys) are injected via Kubernetes Secrets. A CI/CD workflow auto-publishes the Docker image to a container registry on main-branch merges.

Cohort-Wide Penalty Bank

The calibration pass produces a canonical penalty bank that summarizes every unique penalty type across all teams — its fail rate, standard deduction, and calibrated deduction. This is a class-level derived artifact: more useful for instructors and future semesters than any single team's snapshot. In a knowledge-graph architecture, each canonical penalty becomes a node with edges to affected teams, rubric sections, and prior runs.

Architecture: Nested DAG Pipeline

The grader is structured as a Nested Directed Acyclic Graph (DAG) of five independent stages that execute sequentially, each opening and closing its own MongoDB session. A post-hoc calibration phase aggregates penalty candidates globally, applies semantic clustering, and statistically thresholds class-wide failure rates before the HITL triage UI.

Architecture: Stage Separation (Claims vs. Reality)

A core architectural philosophy: Report Analysis extracts what students claim they built, while Code Analysis builds a semantic map of what the repository actually contains — completely blind to the student's claims. Cross-Examination then reconciles the two with deterministic heuristics and a two-pass Gemini audit. This mirrors the distinction between ephemeral workspace artifacts and the authoritative persistence layer in a knowledge system.

Architecture: Five-Phase Storage Migration

All five migration phases from disk-based caches to MongoDB are implemented. MongoDB is now the active runtime store for runtime metadata, historical promises, repo asset signals, extracted submission bundles, and per-team audit payloads. Old disk caches are archived or downgraded to regenerable export artifacts.

Architecture: Dual-LLM Routing

The system routes by stage: local LLMs (vLLM/Ollama) handle Code Analysis high-throughput commit summarization; cloud Gemini handles Report Analysis extraction, Cross-Examination reasoning, and Rubric Scoring synthesis. The switch prevents cluster GPU throughput from competing with cloud reasoning quality.

Architecture: Deep Context Pass (Two-Pass Cross-Examination)

Cross-Examination's opt-in Pass 2 sends the full codebase snapshot to a frontier model's large context window. Pass 1 uses condensed commit summaries (local LLM output); Pass 2 confirms, revokes, or adds findings by reading actual source code — discovering cross-file relationships and import chains that summaries miss. The pattern maps to local → global search in a GraphRAG system.

Architecture: Forensic Dossier as Primary Artifact

The pipeline's ultimate output is a Markdown forensic dossier — not just a numeric grade. The dossier narrates what the team claimed, what the repo shows, and how the grader reconciled the two, with per-claim evidence traces and academic metrics (effort score, churn rate, procrastination index, balance/Gini coefficient, PR-to-Push ratio).

Architecture: Academic Metrics as Sanity Layer

Five metrics are computed deterministically from Git data — no LLM involvement for raw numbers, providing a hallucination-free anchor:

  1. Balance Index (Gini Coefficient) — Measures contribution equity across team members. A balanced team resembles a "Walking Skeleton" ideal; a highly skewed distribution flags a "Hero Rescue" risk profile.
  2. Procrastination Index — Quantifies temporal concentration of commits near deadlines, identifying teams that front-load work versus those that cram.
  3. Code Churn Rate — Measures lines added relative to modified/deleted, revealing scoping instability and rework.
  4. PR-to-Push Ratio — Captures code review hygiene: teams that push directly to main vs. those that follow structured pull-request workflows.
  5. Total Effort Score — Aggregate metric combining volume, consistency, and distribution of work.

An LLM Academic Synthesis step provides a 2–3 sentence interpretation, but the metrics themselves are pure formula. A constrained generation pattern strictly limits the LLM to interpreting only the supplied numbers, never introducing new data or hallucinating context.

Architecture: Heuristic Override with LLM Rubric Scores

Rubric Scoring uses a Synthetic Auditor approach: heuristic baseline scores are computed first, then Gemini reviews actual claims vs. code reality and can override scores with a reasoning trace. Deterministic template and late penalties are always re-applied on top of the LLM override — the LLM never sets the final number unilaterally.

The Synthetic Auditor detects seven penalty categories across two severity levels: Hallucination, Missing Evidence, Contribution Imbalance, AI Unreviewed Boilerplate, Poor Commit Messages, File Dump, and Scope Drift. Detection uses a two-pass Gemini audit — Pass 1 operates on condensed commit summaries for broad coverage, while an opt-in Pass 2 performs a deep 1M-token context scan with smart file selection to catch cross-file relationships and import chains that summaries miss.

Architecture: Author Alias Resolution

Cross-Examination resolves git author names to canonical student names using a two-layer strategy: (1) roster identity mappings (GitHub username, full name, substring fallback) and (2) email-based enrichment matching commit email local parts against roster email local parts. This catches aliases with no textual overlap in name fields.

Architecture: Concurrency Stack

Code Analysis uses a three-layer semaphore for throughput: across repos via async gathering, within a repo via an in-flight cap, and within a commit via per-file parallel map calls. Peak theoretical in-flight can reach dozens of concurrent operations, achieving 97–99% GPU utilization.

Architecture: Phase-Gated Rollout

The system uses dual schema versions during soft launch: a top-level schema version stays stable while per-stage schema versions track newer payloads. Legacy fields remain as compatibility projections. Adapters ensure backward compatibility to maintain the HITL dashboard during migration.

Architecture: LLM Academic Synthesis (Constrained Generation)

The Academic Metrics LLM prompt strictly constrains output to 2–3 sentences interpreting five numeric metrics. The LLM is prohibited from introducing new data or hallucinating context beyond the supplied numbers. This constrained generation pattern enforces structure, limits scope, and never lets the LLM invent data.

Supplementary Reading