Skip to content

Test Forge

Test Forge

Test Forge is an AI-based sample testing engine that ingests course materials, generates custom questions, and helps instructors understand how students are learning. It uses LLMs, RAG, and analytics to turn raw content into structured assessments.

The goal is to give educators a data-driven, instructor-in-the-loop way to create practice exams and assignments.

Where This Project Fits

flowchart LR
  Instructor[Instructor] --> TFApp[Test Forge App]
  TFApp --> TF[Test Forge<br/>Question Engine]
  TF --> Psephos[Psephos<br/>Survey Engine]
  Psephos --> Students[Students]

  UMACS[UMACS<br/>Auth Service] --> TFApp
  UMACS --> TF
  UMACS --> Psephos
  • Test Forge App is the frontend where instructors configure and review questions
  • Psephos stores questions and delivers them as surveys/quizzes
  • UMACS provides auth for Test Forge, Test Forge App, and Psephos

RAG Pipeline & Question Generation

Test Forge's core pipeline operates in several stages:

  1. Ingestion & Chunking: Course materials (PDFs, slides, repos) are ingested and split into semantically meaningful chunks
  2. Topic Extraction: Key topics and their relative importance are automatically extracted from the material
  3. Vector Embedding: Chunks are embedded and stored in Qdrant for retrieval-augmented generation
  4. Question Synthesis: An LLM (via Ollama or cloud providers) generates questions grounded in the retrieved context

Bloom's Taxonomy Controls

Questions can be configured by cognitive level across Bloom's Taxonomy — from simple recall (Remember) through synthesis and evaluation (Create). This allows instructors to control the difficulty distribution of an assessment, ensuring a balanced mix of lower-order and higher-order thinking skills.

Instructor-in-the-Loop Workflow

Rather than treating AI-generated questions as final, Test Forge implements a feedback loop:

  • Instructors can approve, edit, or reject each generated question
  • Accepted edits feed back into the model as few-shot examples or fine-tuning data
  • A running performance log tracks question quality metrics (discrimination, difficulty) over time
  • This creates a collaborative process where the AI improves with instructor guidance, rather than operating as a black box

What You Could Work On

  • Build pipelines that ingest and chunk course materials (PDFs, slides, repos)
  • Design and evaluate question generation strategies and Bloom's taxonomy controls
  • Implement instructor feedback loops (approve/edit/reject) and track model performance
  • Experiment with RAG architectures, vector stores, and evaluation metrics
  • Integrate Test Forge with downstream systems (Psephos, UniPoll, LMSes)

Core Concepts & Tech

  • Backend: Python, FastAPI, MongoDB
  • AI stack: LangChain, Qdrant (vector DB), LLMs via Ollama or cloud providers
  • Key ideas: RAG with contextual retrieval, Bloom's taxonomy controls, instructor-in-the-loop feedback, difficulty modeling

See the linked wiki for deeper design docs and scope discussions.