Post-Hoc Grading Calibration: A Psychometric Framework for AI-Assisted Assessment¶

1. The Rationale for Algorithmic Empathy in Automated Grading¶

In high-stakes academic environments, the transition from rigid, deterministic autograding to AI-assisted assessment often risks introducing "construct irrelevant variance"—noise in the data where student scores are influenced by factors other than the intended learning outcomes. Traditional autograders are historically punitive, docking points for missing requirements without accounting for class-wide performance trends. To mitigate this, we have implemented a strategic post-hoc calibration model centered on "Algorithmic Empathy." This shift recognizes that when a vast majority of a cohort fails a specific metric, the failure often indicates systemic flaws in the rubric or assignment instructions rather than widespread student incompetence.

The key differentiator between raw AI assessment and calibrated grading lies in the transition from an automated "finality" to a Candidate-Only mode. In this paradigm, raw AI findings serve as an initial extraction snapshot, identifying potential issues with high fidelity. Calibration then applies a psychometric lens to these findings, adjusting a "suggested deduction bank" for instructor review. This ensures the grading system addresses instructional-level gaps rather than punishing students for navigating ambiguous prompts. By identifying these systemic hurdles, we prepare the dataset for rigorous statistical normalization and semantic clustering.

2. Phase I: Semantic Consolidation and Canonical Penalty Identification¶

To maintain pedagogical integrity, the system must normalize varied Large Language Model (LLM) outputs to ensure that teams penalized for the same conceptual error receive identical suggested deductions. In a standard assessment of 39 teams, the Lab4Grader generates hundreds of distinct natural language findings. Without consolidation, the system might suggest disparate point values for identical mistakes simply due to variations in how the AI phrased the observation.

The "Semantic Clustering Pass" utilizes a "Mixed Routing" architecture—employing gemini-3.1-flash-lite-preview for high-speed structured extraction and gemini-3-flash-preview for nuanced scoring. This process transforms 402 candidate observations into 33 Canonical Clusters:

1. Initial Extraction Snapshot: The grader executes across the cohort, saving raw evidence into JSON snapshots.
2. Semantic Clustering: A secondary model (e.g., gemini-3-flash-preview) ingests the raw findings to identify semantically identical mistakes.
3. Standardization: Issues are grouped into "Canonical Penalties," assigned a global identifier and a baseline deduction value.
4. Mapping: The system generates a calibrated_penalties.json file, mapping every affected team to these unified clusters for frequency analysis.

This consolidation provides a structured dataset for the application of psychometric Item Difficulty indices.

3. Phase II: The Item Difficulty Index (\(p\)) and Statistical Thresholding¶

With normalized data, we apply the psychometric concept of "Item Difficulty" (\(p\)) to automated grading. In this framework, \(p\) serves as a proxy for assignment clarity, functioning as a measure of Differential Item Functioning (DIF) across the entire cohort. If a requirement exhibits an exceptionally high failure rate, it signals that the "item" (the requirement) is not effectively discriminating between high and low performers, but is instead acting as a class-wide obstacle.

We segment suggested penalties into three primary "Calibration Zones" based on the failure rate across the 39-team cohort:

• The Baseline (\(p < 40\%\)): These requirements are considered "fair" and well-aligned with instructor intent. The standard suggested deduction is maintained without modification.
• The Reduction Zone (\(p = 40\% - 75\%\)): Items in this range are classified as "surprisingly difficult." This frequency suggests the rubric prompt may have been ambiguous. The system applies a 50% global reduction to the suggested penalty severity to reflect this uncertainty.
• The Waiver Zone (\(p > 75\%\)): A failure rate this high indicates a systemically flawed requirement. A prime example was the "ERD Missing" cluster, which recorded a \(p\) of ~0.8974. The system issues a 0-point "Waiver" for these items while retaining the feedback in the report to preserve the pedagogical learning loop.

These thresholds ensure the grading system adapts to the actual performance of the Spring 2026 cohort, prioritizing fairness over mechanical adherence to a flawed rubric.

4. Operational Workflow: From Extraction to Calibrated Snapshot¶

The operational pipeline is strictly decoupled to ensure data integrity and prevent "mechanical double-counting," where a single student error might trigger multiple redundant deductions.

The Grading Lifecycle follows a three-stage progression: 1. Initial Extraction Snapshot: Generation of raw JSON findings containing evidence bundles (Wiki text, READMEs, and Mermaid diagrams). 2. Calibration Run: Execution of calibrate_penalties.py to produce calibrated_penalties.json, applying \(p\)-index calculations to the canonical clusters. 3. Application Pass: A script that populates the Candidate-Only suggested deduction bank, replacing raw values with normalized and scaled deductions.

Operational Case Study: SVG Rasterization Recovery During the initial run, the system encountered a systemic failure where SVG blueprint files were unreadable due to environment library issues (CairoSVG). Recognizing this as a technical hurdle rather than a student failure, the workflow allowed for an "Empathetic Rerun." By shifting to resvg-py for fallback rasterization, we selectively reran affected teams (e.g., S26-CSE-302, 307), recovering their blueprint scores and ensuring no student was penalized for a library-level runtime error.

This workflow culminates in the Human-in-the-Loop (HITL) Streamlit Dashboard. The dashboard presents TAs with a "Withheld Findings" panel, where low-confidence items are surfaced for adjudication rather than being automatically scored, and every suggested penalty is displayed with its "calibrated context" (e.g., "65% of class failed this — Penalty reduced").

5. Empirical Impact: Class-Wide Grade Distribution and Equity Analysis¶

Comparing raw distributions to calibrated outcomes validates the necessity of this methodology. The calibration pass effectively transforms the grade distribution from a punitive, low-mean state into one that accurately reflects the cohort's genuine effort and mastery.

Calibration Impact Summary: * Mean Score Shift: The student mean score rose from a punitive 13.24 (Raw) to a defensible 28.52 (Calibrated). * Bin Compression: The "0-9.99" score bin, which initially contained 66 students, was completely eliminated. The majority of teams (35 of 39) moved into the 20-40 point range. * Base-Score Lift: Specific teams saw significant lifts; S26-CSE-337, for instance, received a +28.82 lift after calibration adjusted for high-prevalence blueprint gaps. * The Single Strongest Signal: The waiving of the "ERD Missing" cluster (89.74% fail rate) was the primary driver of equity, acknowledging a class-wide instruction-level gap. * Traceability Pass: The calibration is further anchored by a "Traceability Pass," ensuring that Lab 4.0 issues reflect the contracts established in Lab 2 and Lab 3, thereby reducing "drift" between project phases.

The Bottom Line: Calibration transformed the final grade distribution from a punitive curve into an equitable reflection of effort. By neutralizing construct irrelevant variance, the system provides a suggested baseline that is statistically grounded and fully defensible during instructor review.

6. Closing: The Future of Defensive and Transparent Assessment¶

The implementation of this framework elevates the Lab 4.0 Grader from a simple assessment script to a "Provenance-Aware Grading Platform." By integrating psychometric calibration and mixed-model routing, the system achieves a level of fairness and technical durability that manual grading struggles to maintain at scale.

The long-term benefits of this architecture are substantial: reduced TA fatigue through automated consistency checks, increased student transparency via the retention of feedback for waived items, and a scalable model for project-based assessment. Ultimately, while AI provides the efficiency of high-volume data extraction, the Human-in-the-Loop agency remains the final authority, ensuring that algorithmic speed never supersedes educational integrity.