From Commit to Canvas: The Lab 4.0 Grading Journey¶

1. The Big Picture: "Sprint 0" and the Grading Mission¶

In the lifecycle of a senior capstone project, Lab 4.0 serves as the "Inception Phase" or "Sprint 0." Before a single line of production code is written, teams must align on their technical vision and operational foundation. The mission of this lab is to establish three core pillars of student submission:

• Infrastructure: Establishing the DevOps environment, including private GitHub repositories, enabled Wikis, and functional Project Boards.
• Action Plan: Defining the roadmap through Milestones, Iteration planning (Steel Thread and MVP), and a refined Issue Backlog.
• Blueprints: Creating the "technical contract" via high-fidelity UI mockups and backend architecture definitions (ERDs and API specifications).

To grade these multi-faceted artifacts fairly and at scale, we utilize a pipeline that balances automated rigor with human empathy.

Pedagogical Philosophy: Empathetic AI Grading Traditional autograders act as rigid, binary judges. Our system is designed to mimic the empathy of a seasoned instructor. Through the use of "Item Analysis," the system determines if a mistake was made by a single team or the entire cohort. If more than 75% of the class misses a requirement, the system identifies a potential flaw in the instruction or rubric ambiguity. It then applies a global waiver, ensuring students are guided by feedback rather than punished for systematic instructional gaps.

To understand how a project transforms into a grade, we must first look at how the raw evidence is gathered.

2. Phase 1: The Evidence Room (Data Ingestion)¶

The journey begins with the transition from GitHub's live environment to a structured MongoDB database. This decoupling ensures that the grading engine operates on a stable "snapshot" of evidence, free from the latency of external API calls.

The Primary Data Collections¶

Expectation Management Note: Based on recent data ingestion reports, Wiki text remains the richest source of grading input. While the schema includes a field for README text, current ingestion shows it is often a placeholder (0 characters); TAs should prioritize Wiki-hosted documentation during review.

Specialized Extraction Techniques¶

Technical evidence often hides behind complex formats. We employ "Specialized Extraction" to ensure no student work is overlooked: * Mermaid.js Detection: The system scans Wikis for text-based diagram code and renders it as visual evidence for the grader. * SVG-to-PNG Rasterization: To ensure Vision models can interpret vector graphics, we use a resvg-py fallback. This was specifically implemented because the standard cairosvg library often fails due to a missing libcairo.so.2 runtime in cloud environments. * Remote Image Discovery: The pipeline identifies images hosted in GitHub user-attachments, downloads them locally, and caches them for AI "Vision" analysis.

Once the evidence is securely stored in the database, the AI "Brain" begins its analysis.

3. Phase 2: The Staged AI Evaluation (The Brain)¶

Rather than utilizing a monolithic prompt—which often leads to hallucinations or skipped requirements—the system employs Stage-Decomposed Evaluation. This process breaks the analysis into three distinct roles: 1. Extractor: Identifies raw facts (e.g., "Identified 5 issues assigned to 'Alice'"). 2. Validator: Cross-references facts against the rubric "contracts." 3. Scorer: Determines the final deduction and assigns a Confidence Score.

Gemini Model Routing¶

We route tasks to specific models based on the Gemini Model Routing Benchmark to optimize for both speed and multimodal accuracy: * [x] Action Plan Extraction (Text): gemini-3.1-flash-lite-preview (Optimized for fast, low-cost structured text extraction). * [x] Blueprints Evaluation (Vision): gemini-3-flash-preview (Supports multimodal "sight" for analyzing diagrams and mockups). * [x] Complex Adjudication: gemini-3.1-pro-preview (Utilized for high-stakes reasoning and resolving low-confidence findings).

Feature Graph Traceability Lab 4.0 is not evaluated in isolation. The system performs a traceability pass that checks Lab 4 evidence against the "contracts" established in prior labs. It specifically ingests Lab 2 markdown (User Stories/Acceptance Criteria) and Lab 3 evaluation JSON (Architecture/Wireframe results) to ensure the team's current backlog reflects the features they originally promised to build.

Raw scores from the AI are only the first draft; the system then applies a "fairness filter" across the entire class.

4. Phase 3: The Fairness Filter (Post-Hoc Calibration)¶

To ensure consistency, the system performs Item Analysis. It uses "Semantic Clustering" to group identical AI findings from all 39 teams. For example, if multiple agents describe a "missing database primary key" in different words, they are clustered into a single Canonical Penalty.

The Calibration Zones¶

The system calculates the Item Difficulty Index (\(p\)) for every requirement. If the failure rate (\(p\)) exceeds certain thresholds, the system automatically softens the penalty:

With the scores calibrated for fairness, the final decision-making power returns to the teaching staff.

5. Phase 4: Human-in-the-Loop (The Control Tower)¶

The final decision-maker is always the TA. The Streamlit HITL UI serves as the primary workspace where automated findings are reviewed and finalized.

The TA Workspace: High-Leverage Actions¶

1. Evidence Inspection: TAs use popovers to view the exact evidence the AI saw—including rendered Mermaid diagrams, Wiki quotes, and cached mockup images.
2. Deduction Overrides: TAs maintain final agency, using sliders and checkboxes to adjust scores or reject AI findings.
3. Final Export: Once the "Candidate Queue" is reviewed, the TA generates a Canvas-ready CSV with integrated HTML feedback.

Candidate-Only Mode¶

To maintain high standards without "autograding traps," the system operates in Candidate-Only Mode. This functions as a Reviewer-Assist feature. Because the AI now surfaces highly granular findings—such as specific UML notation errors (e.g., "UML Arrow Abuse") or traceability gaps—these are presented as suggestions. A TA must manually determine if a finding warrants a formal deduction based on the team's specific context.

This collaboration between automated extraction and human judgment ensures every student receives a grade that is both data-driven and empathetic.

6. Summary of the End-to-End Execution Flow¶

For the teaching staff, the workflow moves from raw data to a finalized grade through the following versioned steps:

• Ingestion: Raw GitHub evidence (Wiki, PRs, Issues) is synced to MongoDB; Mermaid and SVG artifacts are processed.
• Preparation: The batch runner collapses multi-repo teams and selects the "best" repository based on asset volume (Wiki text/Image paths).
• AI Evaluation: Staged Gemini models (-preview versions) generate findings for the Action Plan and Blueprints.
• Calibration: Semantic clustering groups raw findings into Canonical Penalties; fail rates (\(p\)) are calculated and thresholds applied.
• HITL Review: TAs use the Streamlit Dashboard to approve or override candidates.
• Persistence: All edits are saved to ReviewRevisions. This is an append-only history that serves as a durable audit trail, allowing every human adjustment to be versioned and recovered.
• Export: The reviewed state is compiled into a final CSV for Canvas upload.