Skip to content

GradeFlow

GradeFlow is a visual, no-code platform for building AI-assisted grading pipelines. Think of it as "LangFlow for grading" — instructors design assessment workflows by connecting modular components (extractors, validators, scorers, calibrators) in a visual DAG interface, without writing any code.

Motivation

After deploying AI autograders across multiple CSE120 semesters at UC Merced, we learned that every course and assignment has unique grading requirements. A one-size-fits-all autograder simply doesn't work for open-ended capstone projects. The Lab 4.0 autograding experiments taught us that assessment can be decomposed into composable stages — but each course needs to define its own evaluation criteria, penalty structures, and feedback templates. Instructors shouldn't need to write Python or understand LLM prompting to do it.

GradeFlow is the synthesis of everything learned from those experiments into a platform for the broader teaching community.

Core Concepts

Grading Pipelines as Visual DAG Workflows

GradeFlow's core abstraction makes grading criteria, evidence collection, and scoring composable and visual. An educator drags nodes — Document Ingest, Repository Scanner, Claim Verifier, Rubric Judge, Effort Metrics — onto a canvas and connects them, mirroring LangFlow's insight for LLM chains but renaming everything for the educator domain.

Rubric Criteria as Structured Data, Not Prompts

Rubrics are encoded as structured data (YAML/JSON arrays of criteria objects), not buried inside LLM prompts. Each criterion (Participation, Listening, Feedback, Cooperation, Time Management, Contribution, Quality) has a 4-level descriptive scale. Students see the same rubric before submitting, and educators can version-control, share, and adapt rubrics as configurable artifacts decoupled from execution code.

Sigmoid (Logistic) Scoring for Phase Transitions

Peer evaluation scoring uses a sigmoid (logistic) function instead of polynomial/quadratic curves to model competency thresholds as phase transitions. The sigmoid provides natural top-end immunity (minor peer-score disagreements produce negligible multiplier changes) and a bottom-end floor (scores below threshold asymptotically approach zero). This frames evaluation as a phase transition between "Contributing Member" and "Free-Rider" rather than a linear good-to-bad scale — educators tune just two parameters: the inflection point (where multiplier = 0.5) and the steepness (how aggressive the cliff is).

LLM as Optional Enhancer, Not Default Engine

Most grading tasks default to deterministic scoring — test pass/fail, file presence, git activity metrics. LLM nodes are optional enhancers for soft criteria like "quality of explanation" or "depth of reflection." The educator describes "good" in plain text; the platform translates that into a prompt internally. This establishes a tiered approach: deterministic rules form the foundation, LLM-prompt templates add nuance where needed, and the platform mediates between them.

Evidence Types as a Fixed Menu

Evidence types — commit history, report claim, test results, file exists, code pattern — form a bounded enumeration that educators configure in a spreadsheet-like rubric table. This constrains the design space and provides deterministic defaults for common assessment patterns.

Node SDK / Plugin Architecture

Three plugin types emerge: - Evidence collectors — scrape external sources (GitHub Actions, Canvas, Figma) - Scorers — regex, AST, LLM, test-runner algorithms - Output adapters — export to Canvas, Gradescope, CSV

Each node has a plain-English description and hides LLM routing behind a Quality/Speed slider. The plugin SDK defines the contract between pipeline components and the platform.

Pedagogical Phrase Transition Model

The logistic model frames assessment as a phase transition rather than linear scaling. The inflection point moves students between competency bands, with the steepness parameter controlling how much peer-score disagreement the system tolerates before flipping a student's status. This is both mathematical (the sigmoid formula) and pedagogical (the rationale for why sigmoid beats polynomial scoring).

Workflow-Level Concerns: Dry-Run, Schema, Serialization

The prioritized build sequence mirrors the Lab 4.0 architecture: 1. Formalize a DAG workflow schema (JSON/YAML) 2. Create the Node SDK base class 3. Build dry-run/preview mode so educators can test pipelines 4. Build rubric configuration UI 5. Build the Canvas/LMS bridge

The existing Lab 4.0 codebase becomes the reference implementation and first plugin bundle.

Current Status

GradeFlow is in Prototyping stage, building directly on the five-phase pipeline architecture (B → A → C → D → E) and post-hoc calibration developed in the Lab 4.0 experiments. The Nested DAG architecture — sequential stages with isolated MongoDB sessions — is the runtime model that GradeFlow's visual builder will orchestrate.