Skip to content

Outcomes & Findings

Running a 114+ student capstone course provides a wealth of data. Here's what we've learned across semesters.

Grading Fairness

The Individual Multiplier Works

The Individual Multiplier — a formula that differentiates individual contributions within team projects using a 40% TA / 50% Peer / 10% Self split — has produced measurable improvements in grading fairness.

  • The quadratic penalty function means minor peer friction (e.g., a student rated slightly below average) results in negligible grade impact, while severe abandonment accelerates penalties rapidly
  • A 10% contribution threshold ensures that penalties are only applied when a student contributes less than 10% of the team's total meaningful changes
  • The 40% TA weight serves as a circuit breaker against team collusion or interpersonal conflict

AI-Assisted Grading Consistency

The staged LLM evaluation pipeline has demonstrated: - No catastrophic failures: The lowest Lab 4.0 score was 26/50 — the safety net of partial credit prevents students from being completely derailed by a single misconfiguration - Identifiable bottlenecks: Score clustering at 40/50 revealed a specific deployment pipeline challenge that became a targeted teaching intervention - Anti-pattern detection: The pipeline automatically flags hardcoded credentials, race conditions, mock data arrays, and non-functional stubs — patterns that manual grading frequently misses

The Intentional Negative Skew

The grade distribution is deliberately left-skewed — teams building working systems score higher. However, we've learned that:

A negative skew without rigorous individual differentiation represents a quality control failure — it permits low-effort participants to exit with the same credentials as high-contributing peers.

The combination of team-level scoring with the Individual Multiplier addresses this directly.

Team Formation

Algorithmic Teams Outperform Random Assignment

Data from the OptiTeam pipeline shows: - More balanced contributions: Algorithmically-formed teams show lower Gini coefficients in work distribution - Higher satisfaction: Student survey scores are higher for algorithmically-assigned teams compared to self-selected or random teams - Better skill complementarity: Teams formed with SkillNet-inferred competencies show more diverse technical approaches

The NP-Hard Reality

Student team formation is genuinely NP-Hard — as our class size has grown from ~60 to 114+, brute-force approaches became computationally infeasible. The ILP-based optimization solves in minutes what would take classical computers longer than the universe to enumerate.

Student Learning

The "v0.1" Flip

Requiring students to complete an initial version of workshop tasks at home before lab sessions has: - Reduced lecture lag — students arrive with context - Lowered cognitive load during lab — NASA-TLX measures show frustration dropping from 7-8 (individual setup) to 4-5 (team coordination) - Increased workshop productivity — lab time is spent building, not waiting for environment setup

Changelogs Improve Understanding

Replacing unstructured prose reports with changelogs has had a measurable impact:

"Traditional reports were a nightmare to grade due to unstructured, sprawling nature prioritizing quantity of prose over quality of technical evidence."

Students now think in terms of Added, Changed, and Fixed — categorizing work trains them to evaluate the nature of their contributions rather than padding word counts.

The Dunning-Kruger Effect in Git

One finding that surprised us: confidence in Git commands does not equal competence in collaborative workflows. The PR-only mandate — where all changes flow through pull requests with peer review — has been essential for developing genuine version control proficiency.

Infrastructure & Scalability

Graduate-Proofing

All tools are designed with a Bus Factor of 1 — the application layer is a high-value conduit, but the ultimate source of truth is always a standard format (Google Sheets, CSV, or similar). If any tool becomes unmaintainable, the workflow can "eject" back to standard practices with all data preserved.

From 3,000 to 50,000 Minutes

A single inefficient CI/CD workflow could decimate the shared pool of GitHub Actions minutes. The transition to the GitHub Campus Program (Enterprise-tier) increased capacity from 3,000 to 50,000 monthly minutes, with SAML SSO eliminating FERPA concerns tied to personal accounts.

Ongoing Research

Several findings are being developed into publications:

  • The Empirical Triangulation Framework for correlating subjective survey data with objective repository metadata
  • SkillNet's GNN-based competency inference and its application beyond team formation
  • The agentic autograding architecture as a model for AI-assisted assessment at scale
  • The "Transfer-then-Adapt" privacy methodology for training educational AI systems on industrial data

What's Next

Every semester generates new questions. Current directions include: - Expanding cross-lab feature traceability — can we automatically verify that requirements from Lab 1 are implemented by Lab 4? - Improving the calibration pass with more sophisticated Item Difficulty Index models - Applying the team formation pipeline to other courses and institutions - Publishing formal results on the correlation between AI-predicted and human-assigned scores

  • Capstone Survey — Source of the longitudinal data powering these outcome analyses.
  • GradeFlow — Pipeline that will make these analyses reproducible across semesters.
  • Team Evaluation — Peer evaluation data informing the individual differentiation findings.
  • T4 + TA Copilot — Task tracking metrics correlated with team success outcomes.