Skip to content

SkillNet: Graph-Based Skill Representation

SkillNet addresses a fundamental challenge in CS education: how to model the complex, interconnected landscape of computer science skills as a computable graph. Moving beyond static hierarchical taxonomies, SkillNet constructs rich skill ontologies that capture prerequisite relationships, co-occurrence patterns, and conceptual proximity — enabling downstream applications in team formation, learning-path construction, and curriculum design.

SkillNet's architecture is implemented on Neo4j Enterprise on Kubernetes, using Causal Clustering with the routing-aware neo4j:// protocol to avoid leader-bottleneck contention in read-heavy workloads. Graph algorithms — PageRank, Louvain community detection, and FastRP embeddings — run in-database via the Neo4j GDS (Graph Data Science) library, eliminating serialization overhead for skill graphs with tens of thousands of nodes.

Taxonomy-to-Ontology Paradigm Shift

Traditional skill representation relies on fixed taxonomies — parent-child is_a relationships that are easy to build but limited in expressiveness. SkillNet embraces a shift toward graph-based ontologies that model rich, multi-type interconnections: requires, precedes, co-occurs_with, and — most critically — prerequisite_of. This transforms a flat skill list into a network amenable to computational analysis through graph theory and network science.

Small-World Skill Networks

Skill graphs exhibit small-world properties: high clustering (skills within specialized domains like "web development" form dense neighborhoods) combined with low characteristic path length (any two skills are reachable via few hops). SkillNet validates constructed graphs using the omega (ω) metric, which compares clustering to a lattice baseline and path length to a random baseline, producing a clean measure of "small-world-ness." A high ω confirms the graph is fit-for-purpose for traversal-based queries.

This property is not merely an interesting structural artifact — it is a quality gate. Before running downstream analyses, SkillNet validates the constructed graph using Neo4j's Graph Data Science library to compute clustering coefficients and path lengths. A graph that fails to exhibit small-world properties signals poor construction quality, prompting a revisit of entity linking or edge extraction before proceeding.

LLM-Based Edge Extraction with Chain-of-Thought Auditing

The core mechanism for populating the skill graph is LLM-based edge extraction with chain-of-thought (CoT) auditing. Given a source document (documentation, README, syllabus), the LLM identifies skill relationships and provides a quoted justification from the source text before emitting the relationship type. Each extraction carries a confidence score (0–100).

The pipeline enforces a confidence-based triage policy:

  • High-confidence extractions (>90): Auto-approved with periodic random sampling for accuracy auditing.
  • Low-confidence extractions: Flagged for human-in-the-loop manual review.

This produces an auditable, academically rigorous data pipeline — critical for establishing trust in LLM-generated knowledge graphs.

Two-Phase Extractor Pattern

The extraction process operates in two phases:

  1. Bootstrapper (Phase 1): An LLM/NER pipeline that mass-processes text into graph triples — dumb but scalable. This phase extracts explicit skill mentions and relationships from source material.
  2. Conceptual Extractor (Phase 2): A GNN-powered service that uses graph embeddings to infer conceptually related skills not explicitly mentioned in any single document. This phase enriches the graph with latent connections that emerge from the overall topology.

CSO as Seed Scaffold

The Computer Science Ontology (CSO) provides approximately 14,000 CS topics with 162,000 semantic relationships as the initial node dictionary and hierarchical scaffold. CSO contributes hierarchical (superTopicOf) and synonym (relatedEquivalent) relations but notably lacks prerequisite_of edges — precisely the relationship type that SkillNet's LLM extraction pipeline is designed to infer from GitHub dependencies, job board co-occurrences, and documentation.

GNN Embeddings for Downstream Applications

Graph Neural Networks operate on the constructed graph to learn dense vector embeddings per skill node by aggregating neighborhood features. These embeddings enable three high-value downstream applications:

  • Skill extraction from project descriptions: Vector similarity search from a "project vector" to related latent skills.
  • Learning-path construction: Shortest-path traversal in embedding space between a student's current skill set and their target skills.
  • Team formation: Optimization that maximizes cosine similarity between student skill vectors and project requirement vectors.

Data Sources and Infrastructure

SkillNet ingests from multiple heterogeneous sources — GitHub repositories (explicit dependency edges via package.json), job boards (co-occurrence edges), and documentation (semantic similarity via text embeddings). Entity linking normalizes raw mentions (e.g., "JS," "Javascript," "ECMAScript") to canonical nodes using CSO as the authoritative anchor dictionary.

The "Prerequisite Edge" as Core Semantic Relationship

The most valuable — but hardest to infer — edge type in skill graphs is prerequisite_of: a directed dependency expressing that one skill must be mastered before another (e.g., Python → is_prerequisite_for → Deep Learning). This goes far beyond co-occurrence, which captures correlation but not causation. Prerequisite Chain Learning (PCL), traditionally applied to formal curricula, is adapted in SkillNet to noisy, implicit signals extracted from GitHub dependency graphs and job-board skill co-listing patterns. Inferring these directed edges reliably is SkillNet's crown-jewel challenge and the primary motivation for the validation and confidence mechanisms described below.

PCL Framework: Prerequisite Chain Learning in the Wild

The Prerequisite Chain Learning (PCL) framework operationalizes prerequisite inference by treating package.json dependency lists as "skills-as-ingredients" proxies — the libraries a project depends on reveal latent prerequisite structure between the skills used in that project. SkillNet's PCL pipeline uses Inductive GraphSAGE GNNs trained on these dependency graphs to learn a scoring function that predicts directed prerequisite_of edges between skill nodes. The system runs on a Kubernetes-native, KEDA-scaled pipeline orchestrated by Argo Workflows, enabling cost-effective, burst-parallel processing when large corpuses of GitHub repositories are ingested.

Partial GPU Offloading and QLoRA Distillation

Running a 70B-parameter LLM (e.g., Llama 3.3 70B) for offline batch edge extraction at scale presents a practical throughput bottleneck: the full corpus would take approximately 75 days on a single RTX 5090 (32 GB VRAM). SkillNet addresses this through two complementary strategies:

  1. Partial GPU offloading — Loading roughly 30 GB of model layers into VRAM and offloading the remaining ~10 GB to system RAM. Inference slows to ~5–10 seconds per document but is acceptable for offline batch pipelines where throughput, not latency, is the optimization target.
  2. QLoRA distillation — Fine-tuning a smaller 32B student model (DeepSeek-RQ, Qwen 3) on the 70B teacher's chain-of-thought outputs using QLoRA. The student targets a 3–5× throughput improvement while maintaining high extraction quality, making the pipeline practical for continuous ingestion.

Small-World Validation as Graph-Quality Gate

While small-world topology is a structural hallmark of well-formed skill networks, SkillNet also uses it as a quality gate before any downstream GNN or traversal pipeline runs. After the graph is constructed, Neo4j's Graph Data Science library computes clustering coefficients and characteristic path lengths to derive the omega (ω) metric. A poorly constructed graph — one with fragmented entity linking, spurious edges, or missing prerequisite relations — will fail to exhibit the expected high clustering and low path length. A low ω score triggers a halt: entity linking and edge extraction are revisited before GNN training proceeds. This diagnostic gate ensures that downstream analyses operate on a structurally sound graph.

Confidence Scoring and Human-in-the-Loop for LLM Extraction Trust

LLM-generated knowledge graphs carry inherent hallucination risk. SkillNet addresses this with a structured confidence and audit pipeline:

  • Each extraction is forced into structured JSON carrying a confidence score (0–100) and a quoted justification from the source text.
  • High-confidence extractions (>90): Auto-approved with periodic random sampling for statistical accuracy measurement.
  • Low-confidence extractions: Flagged for human-in-the-loop manual review.
  • Audit metadata (confidence score, reviewer status, justification quote) is stored alongside each extracted triple, maintaining an academically rigorous, auditable data pipeline.

This system is critical for establishing institutional trust in LLM-generated knowledge graphs — a prerequisite for deployment in real curriculum design and team-formation workflows.

Trafilatura over BeautifulSoup for Text Extraction

For crawling CSO-linked resource pages (Wikipedia, DBpedia, topic pages) and documentation sites, SkillNet uses Trafilatura over BeautifulSoup. Trafilatura consistently outperforms in stripping boilerplate (navigation menus, ads, sidebars) and returning clean, structured Markdown. Preserving Markdown headings and lists is critical for LLM prompt quality in the downstream chain-of-thought extraction pipeline: cleaner input directly translates to higher-confidence relationship extractions.

Redis as URL Queue and MongoDB as Document Store in the Crawl Pipeline

The crawl pipeline follows a queue-based architecture:

  • Redis holds the "to-do" URL frontier, enabling broker-based job distribution.
  • Crawler workers pull URLs from Redis, fetch the page, extract text with Trafilatura, and persist the result to MongoDB.
  • MongoDB serves as the durable document store, decoupling storage from crawling.

This decoupling supports arbitrary parallelism — whether via multiprocessing on a single machine or KEDA-scaled Kubernetes pods — and cleanly separates the crawling ETL stage from the subsequent triple-extraction stage that ingests into the knowledge vault.

Kubernetes-Native Crawling with KEDA Autoscaling

The crawl infrastructure runs on Kubernetes with KEDA (Kubernetes Event-driven Autoscaling) monitoring the Redis URL queue depth. KEDA scales crawler pods from zero to the required count when a large batch of URLS arrives and back down to zero when the queue drains. This makes resource usage match workload exactly — ideal for burst-style scraping where tens of thousands of URLs need processing after a new data source is added, followed by long idle periods. The approach ensures cost-effective, elastic data collection.

Argo Workflows for Pipeline Orchestration

Complex multi-stage pipelines — seed CSO, crawl linked URLs, extract text, run LLM edge extraction, generate embeddings, load to graph database — are modeled as Directed Acyclic Graphs (DAGs) in Argo Workflows. Argo handles retries, parallel stage execution, artifact passing, and observability natively. Each stage in the DAG corresponds to a well-defined transformation in the Vault-to-Workbench pipeline, with provenance tracked across steps.

Partial GPU Offloading for 70B Models on Consumer Hardware

Running a 70B LLM (e.g., Llama 3.3 70B) for offline batch edge extraction presents a resource challenge: the model requires approximately 40 GB at 4-bit quantization, exceeding the 32 GB VRAM of a single RTX 5090. SkillNet's solution is partial GPU offloading — loading roughly 30 GB of layers into VRAM and offloading the remaining ~10 GB to system RAM. Inference is slower (~5–10 seconds per document) but entirely acceptable for offline batch pipelines where throughput, not latency, is the optimization target. For setups where even partial offloading is too slow, 32B-parameter models (DeepSeek-RQ, Qwen 3) serve as drop-in fallbacks.

Neo4j GDS Library for In-Database Graph Analytics

Neo4j's Graph Data Science (GDS) library is SkillNet's primary analytics engine. GDS runs graph algorithms — PageRank, Louvain community detection, clustering coefficient, FastRP embeddings — directly inside the database, eliminating costly data serialization and transfer. With an educational Neo4j Enterprise license, the GDS library is unthrottled (no 4-core limit), making it viable for skill graphs with tens of thousands of nodes. GDS sits between the raw graph data and the GNN layer: it produces centrality scores, community assignments, and structural embeddings that feed into downstream learning-path and team-formation applications, while also powering the small-world validation gate described above.

Neo4j Architecture: Causal Clustering and Routing Protocol

The database layer runs Neo4j Enterprise on Kubernetes configured with Causal Clustering via the official Helm chart. In this topology, all cluster members can accept writes (no single-leader bottleneck), and the routing-aware neo4j:// protocol directs read requests to any member while pinning causal sessions to ensure read-your-writes consistency. This is particularly suited to SkillNet's workload profile: bursty batch ingestion (triple extraction from crawled documents) with sustained analytical queries (GDS algorithms, shortest-path lookups for learning-path and team-formation applications). The cluster decouples write throughput from read scalability, allowing the GDS analytics layer to saturate multiple cores across cluster members without contending with the ingestion pipeline.

  • SkillNet — Direct implementation of the skill graph construction and GNN analysis framework.
  • Skill Extractor — LLM-based tool for extracting skill requirements from project descriptions.
  • AI-Assisted Assessment — Test Forge uses related LLM extraction techniques.
  • Team Formation — OptiTeam consumes skill graph embeddings for team composition.

Supplementary Reading