Lab 4.0 Grading Logic: A Guide to Automated & AI-Assisted Evaluation¶

1. Introduction: The Two Pillars of Assessment¶

The Lab 4.0 grading system is engineered to evaluate the "Inception Phase" of your software project with a level of rigor and transparency that matches professional industry standards. To achieve this, the architecture rests on two distinct pillars: Deterministic Rules and Qualitative Evaluations.

As an AI Assessment Architect, my goal is to ensure that while our systems utilize cutting-edge Large Language Models (LLMs), the final grade remains defensible and grounded in evidence. Deterministic rules provide mathematical certainty for infrastructure, while qualitative evaluations use staged AI reasoning to offer nuanced feedback on your project roadmap.

While binary rules ensure the baseline repository requirements are met, the qualitative evaluation provides the nuanced, professional-grade project planning feedback required for the Capstone.

2. Infrastructure Grading: The Deterministic Foundation¶

The "Infrastructure" category (Total: 5 points) is the baseline of your digital workspace. The system performs a series of binary checks on your GitHub repository to ensure a professional team environment is established.

These checks are scored with 100% mathematical certainty. The bot scans for: * Private Repository Status: Ensuring the project is not public, protecting both student and partner intellectual property. * Wiki Feature Enabled: Verifying the primary documentation hub is accessible. * Discussions Feature Enabled: Confirming the platform for team-wide design rationale and coordination is active.

Because these are factual repository settings, if a feature is off, points are deducted automatically without AI intervention. This ensures the foundation of the project is structurally sound before we analyze the contents.

Once the digital workspace is verified, the system shifts from checking settings to analyzing the substance of your project's roadmap.

3. The Action Plan: Qualitative Text Analysis¶

The "Action Plan" category (Total: 15 points) evaluates the logical depth of your goals and team structure. To maintain algorithmic transparency, we utilize a Gemini AI pipeline (specifically the gemini-3.1-flash-lite-preview model) utilizing a three-stage Extractor-Validator-Scorer logic.

This pipeline is designed to prevent "hallucinations" by decomposing the reasoning process: 1. Fact Extraction: The AI identifies raw facts within your Wiki and README (e.g., "The team has assigned the role of Database Lead to Alice"). 2. Rubric Validation: The system checks these facts against the specific assignment requirements. 3. Scoring & Evidence: The AI identifies Candidate Penalties. It only suggests a penalty if it can find an exact source_quote from your text as proof.

Crucially, the system operates in a "Candidate-only" mode. The AI does not unilaterally dock points; instead, it populates a suggestion bank for your TA to review. The qualitative evaluation focuses on: * Problem Statement & Goals: Clarity of motivations and well-defined requirements. * Team Roles Clarity: Distinct and assigned responsibilities for every member. * Action Plan Quality (Detail/Depth): Whether GitHub issues and milestones are descriptive artifacts or mere placeholders.

The AI does not just count words; it looks for the logical depth of your planning artifacts.

4. Blueprints: Multimodal Vision Evaluation¶

The "Blueprints" category (Total: 30 points) acts as the visual contract for your application. We employ "Route-Aware" grading, where the system detects the difference between file types (like PNG/Figma) and code-based diagrams (like Mermaid).

The system utilizes specialized multimodal vision models (gemini-3-flash-preview) to "see" your diagrams. Through Mermaid-aware evidence extraction, the bot parses text-based charts and rasterizes SVGs into PNGs to ensure the model can analyze them accurately. Evidence is routed into two paths:

• Visual Mockups (15 pts): High-fidelity UI flows. The system specifically searches for "Critical Path" screens and the inclusion of loading, error, and success states.
• Technical Architecture (15 pts): Entity-Relationship Diagrams (ERDs) and system flows. The AI checks for technical rigor, such as proper crow's-foot notation in database schemas.

To protect students from cascading failures in visual interpretation, a Deduction Cap is applied: deductions cannot exceed the total points for this section.

Because visual interpretation can be subjective, the system includes an additional layer of statistical protection.

5. The "Empathy" Engine: Post-Hoc Calibration¶

To ensure fairness, we have engineered an "Empathy" engine that uses Post-Hoc Calibration. This mimics a human instructor's ability to recognize if a specific requirement was misunderstood by the entire class.

The system uses Semantic Clustering to group roughly 150 individual AI findings into Canonical Penalties. We then calculate the Item Difficulty Index (\(p\)) for each penalty: * Standard (Fail Rate < 40%): The requirement was well-understood; the candidate deduction remains at standard weight. * Reduced (Fail Rate 40% - 75%): The requirement may have been ambiguous. The penalty severity is automatically halved (50% reduction). * Waived (Fail Rate > 75%): The requirement was systematically misunderstood. The deduction is set to 0.0 points, though the feedback is preserved for your learning.

Case Study: During calibration, the "ERD Missing" penalty cluster showed a 90% fail rate. The engine automatically moved this to the "Waived" category, ensuring no student was unfairly punished for an artifact the entire cohort found confusing.

This calibration ensures that if the entire class finds a requirement ambiguous, you are not unfairly punished for it.

6. Technical Traceability: The Cross-Lab Contract¶

The system performs a "Feature Graph Traceability" pass. This is a check against your previous work to ensure your project is evolving consistently. Using lexical retrieval, the system searches for "anchors"—keywords and features established in your earlier labs—within your current Lab 4 wiki and issues.

This pass explicitly links: * Lab 2: User stories and core requirements. * Lab 3: Architecture diagrams and wireframe evaluations.

While this pass is currently informational only (0 points), it serves as a "warning system" to notify you of "project drift," ensuring the product you planned in February is the same one you are building in April.

Traceability ensures that the project you planned in February is the same one you are building in April.

7. Human-in-the-Loop (HITL) & Final Export¶

The AI is an assistant, not the final judge. The process concludes with a Human-in-the-Loop phase via a specialized Streamlit Dashboard. Here, TAs and Instructors review all "Candidate Penalties" and a "Withheld Findings" panel where low-confidence AI thoughts are surfaced for human adjudication.

Before your grade is "Canvas-ready," several manual steps occur: 1. TA Review: Humans inspect any "Low-Confidence" findings or complex visual artifacts. 2. Manual Overrides: TAs can accept, adjust, or delete AI-suggested deductions. 3. Final Export: Only after human verification is a final CSV generated for the Gradebook.

This hybrid approach—combining deterministic precision, staged AI analysis, and human oversight—maximizes both grading accuracy and pedagogical fairness.