System Architecture Specification: Lab 4 Grader Pipeline¶

1. Architectural Vision and System Objectives¶

The Lab 4 Grader represents a strategic pivot from traditional, rigid autograding to a multi-stage, human-in-the-loop (HITL) evaluation platform. In a high-stakes capstone environment, evaluating open-ended engineering artifacts requires more than pass/fail logic; it demands "empathetic AI grading." Our vision was to build a system that achieves consistency across 39 teams while maintaining total transparency. By decoupling evidence extraction from final score adjudication, we ensure that every deduction is defensible, evidence-backed, and calibrated against class-wide performance.

The system was realized through five core architectural milestones that systematically reduced fragility:

• Phase 1: Mermaid-First Extraction: Integrated Mermaid.js block detection to transform text-native architecture into first-class visual evidence.
• Phase 2: Benchmark Harness and Provenance: Implemented a regression suite tracking prompt and evidence hashes, ensuring pipeline changes were measurable and repeatable.
• Phase 3: Blueprint Routing: Specialized evaluation logic to distinguish between visual mockups, ERDs, and system architecture, routing each to optimized model profiles.
• Phase 4: Feature Graph Traceability: Established Lab 4.0 as a technical "contract" against prior work, validating feature coverage from Labs 2 and 3.
• Phase 5: Staged Gemini Prompting: Decomposed monolithic prompts into Extractor, Validator, and Scorer stages to maximize explainability and minimize hallucination.

This architectural foundation begins with a robust data ingestion layer, designed to transform unstructured GitHub artifacts into immutable evidence snapshots.

2. Data Ingestion and Evidence Transformation Layer¶

Decoupling data ingestion from evaluation is a strategic necessity to ensure system resilience against API rate limits and to facilitate offline re-grading. This immutable evidence snapshot serves as the idempotent input for all subsequent AI evaluation stages.

Multi-Repo Team Handling¶

Student teams frequently create multiple repositories (e.g., S26-CSE-334 vs. S26-CSE-334-8). The system employs a ranking heuristic to select the "canonical" repository, prioritizing the richest data sources for evaluation.

Specialized Extraction and Visualization¶

The pipeline utilizes specialized routines for non-textual evidence. It detects Mermaid.js blocks within Markdown and utilizes a dual-fallback SVG rasterization pipeline to prepare blueprints for multimodal analysis. While CairoSVG is the primary rasterizer, we implemented a resvg-py fallback to resolve production environment errors where the native libcairo.so.2 shared library was missing. This ensures that visual artifacts are consistently transformed into PNG blueprints for the vision models.

Once raw evidence—including the now fully-operational readme_text found in 52 out of 57 repositories—is captured, it is transformed into a structured MongoDB document for the grading orchestrator.

3. MongoDB Schema and Persistence Model¶

The database functions as both the "Ground Truth" store for GitHub artifacts and the "Durable History" for human review checkpoints.

Raw Evidence Collections¶

The system organizes raw data into four primary collections with optimized lookup indexes:

• repositories: The primary metadata store. Critical fields include wiki_text, readme_text, image_urls, and diagram_image_paths to unblock vision grading.
• issues: Keyed by (repo_name, number). Tracks comment_sync_complete to ensure all planning history is captured.
• pull_requests: Tracks merged status and consolidated comments from review threads and issue-style discussions.
• discussions: Captures high-value design rationale occurring outside of traditional task tracking.

Review Persistence Architecture¶

To support collaborative grading, we implemented three layers of generated state: 1. GradingRun: A durable record of the AI’s initial evaluation for a specific team and snapshot version. 2. Review: The latest materialized state reflecting human overrides and final score adjustments. 3. ReviewRevision: An append-only history of every autosave and manual checkpoint.

This multi-tiered model allows the system to restore a reviewer’s exact state across sessions while providing a complete audit trail of grade evolution.

4. Staged LLM Evaluation and Model Routing¶

We transitioned from "Monolithic Prompting" to "Stage-Decomposed Evaluation" to minimize LLM hallucinations. By breaking tasks into Extractor, Validator, and Scorer stages, we ensure that every deduction is cross-referenced against the rubric and backed by an explicit source quote.

Mixed-Model Routing Strategy¶

To optimize for cost, latency, and reasoning depth, we route tasks to specific Gemini profiles using JSON-schema-constrained responses via Pydantic (GeminiEvaluation schema) at a low temperature (0.1) for stability.

• gemini-3.1-flash-lite-preview (Text Extraction): Our default for Action Plan and structured extraction. This model reduced runtime for text extraction from 780s to just 33s (a 95% reduction in latency).
• gemini-3-flash-preview (Multimodal Evaluation): Used for evaluating blueprints and visual artifacts, providing necessary visual reasoning for mockups and schemas.
• gemini-3.1-pro-preview (Complex Adjudication): Reserved for resolving ambiguous evidence or deep reasoning tasks.

Withheld Findings Logic¶

To maintain defensibility, the system identifies "low-confidence" findings where evidence is ambiguous. These findings are surfaced in the "Withheld Findings" UI panel rather than automatically affecting scores. This ensures that AI-generated uncertainty does not penalize students without human verification.

5. Feature Graph Traceability and Cross-Lab Contracts¶

Lab 4.0 serves as a "contract" against Labs 2 and 3. We moved beyond "issue counting" to "feature coverage validation," ensuring the current backlog reflects the features promised in earlier iterations.

Traceability Normalization and Retrieval¶

The Anchor Normalization process transforms Lab 2 User Stories (Markdown) and Lab 3 Remediation Gaps (eval_architecture.json and eval_wireframes.json) into anchors for a searchable corpus. The retrieval model in app/traceability_retrieval.py uses a scoring hierarchy:

• Explicit Coverage: Exact quote matches in titles or bodies.
• Partial Coverage: Keyword overlap or high token similarity.
• Uncertain/None: No lexical or semantic bridge found.

This matrix allows reviewers to spot "drift" where teams have abandoned their architectural plans or ignored critical remediation feedback.

6. Post-Hoc Calibration and the "Penalty Bank"¶

Our pedagogical philosophy centers on "Item Analysis." By analyzing penalty clusters across the cohort, we prevent "mechanical double-counting" and ensure class-wide fairness.

Statistical Threshold Strategy¶

We use the Item Difficulty Index (\(p\)) to calibrate deductions. If a failure rate is too high, it indicates a flaw in the rubric or instruction rather than student negligence.

Case Study: The ERD Waiver. During the April reruns, the "Technical Blueprints: ERD Missing" cluster showed an 89.74% fail rate. This was the strongest class-wide signal that the requirement was flawed, leading to a global waiver of the penalty.

Candidate-Only Workflow¶

In "Candidate-Only" mode, the AI findings are presented as a suggested "Penalty Bank" for TAs. During full-class analysis, the system generated 402 candidate observations across 39 teams. This approach allows TAs to adjudicate a suggestion bank of approximately 10–16 items per team, ensuring that AI suggestions are vetted before affecting final grades.

7. Operational Infrastructure and Deployment¶

The system has transitioned from local scripts to a cloud-native, highly available Kubernetes (MicroK8s) deployment to support the full teaching staff.

Kubernetes Deployment Stack¶

We utilize the following components for security and durability: * PersistentVolumeClaim (PVC): Ensures grading snapshots and human overrides survive pod restarts. * Secret Management: Securely handles API keys (Gemini, GitHub, MongoDB). * Cloudflare Zero-Trust Tunnel: Provides secure access to the dashboard at https://lab40.mpisman.app. * Docker Image: The pipeline is packaged as mpisman/lab40-grader:latest.

Batch Runner Resilience¶

The "Batch Runner" includes defensive mechanisms such as try/except isolation per team and incremental JSON flushes. These features prevent mid-run LLM inference crashes from losing progress, allowing for seamless resumes during large-scale grading tasks.

This comprehensive blueprint ensures the Lab 4 Grader provides a consistent, empathetic, and technologically superior assessment of complex student engineering projects.