Skip to content

Test Forge: AI-Assisted Assessment Generation

Test Forge introduces an end-to-end, flexible workflow for automated assessment generation that directly addresses the limitations of existing static or black-box test generators. Rather than treating question generation as a one-shot black-box process, Test Forge builds a collaborative, pedagogically-informed partnership between instructors and AI systems.

Core Pipeline

The system begins with direct ingestion of instructor-provided course materials — lecture slides, textbook excerpts, problem sets, and supplementary readings. These materials are processed through a retrieval-augmented generation (RAG) pipeline that first extracts key topics and their relative importance from the source content. The RAG architecture ensures that generated questions remain grounded in the specific material the instructor intends to assess, reducing hallucination and improving relevance compared to open-domain generation approaches.

Once topics are extracted, Test Forge provides fine-grained control over question difficulty and cognitive distribution through Bloom's Taxonomy. Instructors can specify not just the number of questions, but how they should be distributed across cognitive levels — from basic recall ("Remember") through higher-order thinking ("Evaluate," "Create"). This enables alignment between assessment design and learning objectives, a feature absent from most automated generation tools.

Instructor-in-the-Loop Feedback

A distinguishing feature of Test Forge is its dynamic feedback loop. Generated questions are not final outputs but draft proposals that instructors can review, edit, accept, or reject. The system incorporates this feedback to refine subsequent generation cycles, effectively creating a continuous improvement loop.

This interactive refinement addresses a critical gap in automated assessment: instructors possess pedagogical domain knowledge that current AI systems cannot replicate. By placing the instructor in the loop, Test Forge ensures that questions meet real classroom standards for clarity, fairness, and curriculum alignment while benefiting from AI's ability to rapidly generate diverse question variants at scale.

Technical Architecture

Built on FastAPI with MongoDB (via the Beanie ODM) for persistent storage, Test Forge provides both a REST API and a CLI interface (via Typer) for managing question banks, assessment configurations, and feedback data. The system is containerized with Docker for straightforward deployment. The generic CRUD API design allows the system to be extended to new question types and assessment modalities without architectural changes.

Research Contributions

  • Demonstrates that RAG + LLM pipelines can produce pedagogically-valid assessment items when grounded in instructor-provided material.
  • Provides an architectural template for instructor-in-the-loop systems that respect pedagogical expertise while leveraging AI generation capability.
  • Integrates Bloom's Taxonomy as a controllable dimension of AI output, not just a post-hoc classification label.
  • Test Forge — Implements the AI-driven assessment generation pipeline described here.
  • Test Forge App — Frontend application for the Test Forge engine.
  • Skill Networks — SkillNet uses similar LLM extraction techniques but applied to skill relationships rather than assessment items.
  • Team Formation — OptiTeam shares the NL-to-formal-model pipeline pattern.