Why LLMs Need Structured Knowledge

Large language models contain a surprising amount of factual information baked into their parameters during pre-training. Ask GPT-4 who discovered Radium and it will answer correctly. Ask it what the capital of France is and it will not hesitate. This parametric knowledge is impressive, but it comes with three fundamental limitations that make it insufficient for production systems.

First, knowledge becomes outdated. A model trained on data up to a certain cutoff date cannot know what happened after that date. No amount of prompting can change this: the information simply is not there. Second, hallucinations are hard to control. When a model does not know something, it tends to confabulate plausible-sounding answers rather than admitting ignorance. This is not a bug that can be patched; it is an emergent property of how these models are trained. Third, multi-fact reasoning is unreliable. Even when a model has all the relevant facts in its parameters, chaining them together in a single inference step is error-prone. The model may correctly know that Radium is used in cancer treatment, and separately that Marie Curie discovered Radium, but fail to connect these facts reliably when answering a question that requires both.

Because of these limitations, most modern LLM systems rely on external knowledge sources that can be updated, verified, and queried with precision. Two architectural patterns dominate this space: knowledge bases and knowledge graphs. Although these terms are often used interchangeably, they represent fundamentally different ways of organizing information, and those differences have significant implications for what kinds of questions an LLM system can answer reliably.

Knowledge Bases

A knowledge base (KB) is a structured repository of facts, typically implemented as a relational database, document store, or vector index. Information is stored as discrete records (rows in a table, documents in a collection, or vectors in an embedding space) without explicit representation of relationships between records.

The simplest form of a knowledge base is a relational table where each row encodes a fact about an entity. Consider the following example, which stores facts about people, places, and things:

From the perspective of LLM systems, knowledge bases are most commonly used as retrieval sources in Retrieval-Augmented Generation (RAG) pipelines. The system retrieves relevant rows or documents before passing them as context to the LLM. This setup works well for document retrieval and simple fact lookup. However, it does not naturally represent relationships between entities. The table above tells us that Radium is used for cancer treatment and that Marie Curie worked in physics, but it does not encode the connection between these two facts. To answer a question like "which physicist discovered an element used in cancer treatment?", a knowledge base would require multiple independent lookups and manual reasoning to connect the results.

Knowledge Graphs

A knowledge graph (KG) is a graph-structured representation of knowledge in which entities are nodes and relationships between entities are typed, directed edges. Information is stored as triples of the form (head, relation, tail), forming a network that can be traversed to answer relational queries.

Formal Definition

Formally, a knowledge graph 𝒢 is defined as a set of triples:

𝒢 = {(𝒉, 𝒓, 𝒕) | 𝒉, 𝒕 ∈ ℰ, 𝒓 ∈ ℛ}

where ℰ is the set of entities (nodes) and ℛ is the set of relation types (edge labels). Each triple (𝒉, 𝒓, 𝒕)  asserts that entity 𝒉 stands in relation 𝒓  to entity 𝒕. For example:

(Marie Curie, discovered, Radium)
(Marie Curie, spouse, Pierre Curie)
(Radium, used_for, Cancer treatment)
(Radium, element_of, Periodic table)

These triples form a directed graph where traversing edges corresponds to following logical relationships. The same information that required four separate table rows in a knowledge base is now encoded as a connected subgraph that can be queried with a single path expression.

Figure 1. Left: a knowledge base stores facts as isolated rows in a table, with no explicit representation of how entities relate to one another. Right: a knowledge graph encodes the same facts as a network of typed relationships, making connections between entities first-class citizens of the data model.

The key insight is that in a knowledge graph, relationships are data, not just a consequence of querying. The edge discovered between Marie Curie and Radium is as much a fact as the entities themselves. This seemingly small distinction has profound implications for what kinds of questions can be answered efficiently.

LLMs Building Knowledge Graphs
Traditionally, building a knowledge graph required substantial manual effort: domain experts had to design an ontology, annotators had to label entities and relationships in text, and engineers had to maintain the graph as facts changed. This high construction cost was the primary reason enterprises historically preferred simpler knowledge bases.

This calculus changed dramatically in 2024–2025. LLM-driven knowledge graph construction has reached production maturity, allowing organizations to automatically extract structured knowledge from unstructured text at scale.[10] The process operates in three stages:

1. Entity extraction. The LLM identifies named entities (people, organizations, concepts, locations) in raw text and assigns them to types.

2. Relation extraction. The LLM identifies typed relationships between extracted entities and expresses them as triples (𝒉, 𝒓, 𝒕).

3. Knowledge fusion. Duplicate entities are merged (e.g., "Marie Curie" and "M. Curie" resolve to the same node), conflicts are resolved, and the new triples are integrated with existing graph data.
   

Figure 2. The LLM-driven knowledge graph construction pipeline. Raw text is processed through entity extraction, relation extraction, and knowledge fusion stages, all orchestrated by language models, to produce a queryable knowledge graph. Tools like Neo4j's LLM Knowledge Graph Builder and LangChain's graph transformer module make this accessible without deep graph expertise.^[11]

^{Organizations adopting this approach have reported 300–320% ROI on knowledge graph construction projects, driven primarily by the reduction in manual annotation costs.[9] The traditional argument that knowledge graphs are "too expensive to build" is weakening rapidly.}

Why Graphs Matter for LLM Reasoning

The structural advantage of knowledge graphs becomes most apparent when answering questions that require multi-hop reasoning: queries whose answers cannot be found in a single fact but require traversing a chain of relationships.

Consider the query: "Which scientist discovered an element used in cancer treatment?" Answering this requires two inference steps:


​
In a knowledge graph, this corresponds to traversing a path of length two. The graph structure makes this path explicit and efficiently queryable. In a standard knowledge base, answering the same question requires at least two independent lookups (one to find what is used for cancer treatment, and another to find who discovered that thing), followed by a join operation that the retrieval system may not be designed to perform.

Figure 3. A multi-hop reasoning path for the query "Which scientist discovered an element used in cancer treatment?" The answer requires traversing two edges in the knowledge graph. Each intermediate node (Radium) serves as a bridge between the query and the answer. Vector RAG, which retrieves chunks independently, cannot reliably traverse this path.

This graph-traversal capability enables three properties that are increasingly important in production LLM systems:

• Multi-hop retrieval. The system can follow chains of relationships to find answers that span multiple facts.

• Structured reasoning. The path through the graph provides an explicit reasoning trace, not just a retrieved chunk.

• Explainable answers. The system can show why it reached a conclusion by exposing the traversal path to the user.

Knowledge Retrieval for LLMs

Modern LLM systems use three distinct retrieval strategies, each suited to different types of queries. Understanding when to use each is central to designing effective GenAI architectures.

Figure 4. The three primary retrieval strategies for LLM systems, shown as parallel pipelines. Each strategy has a distinct data flow, query mechanism, and trade-off profile.

1. Vector Retrieval

The dominant approach in RAG systems is embedding-based similarity search. Documents are chunked, embedded into a high-dimensional vector space using a language model, and stored in a vector index. At query time, the query is embedded using the same model, and the top-k most similar chunks are retrieved using approximate nearest-neighbor search.

Formally, given a query <i>q</i> and a corpus of document chunks {d₁, d₂, ..., dₙ} with embeddings {e₁, e₂, ..., eₙ}, the retriever returns:

\( \text{top-}k = \arg\max_{i} \ \text{top-}k \ \cos(E(q), e_i) \)

where E(·) is the embedding function and cos(·,·) is cosine similarity.This approach is fast, scalable, and works well for semantic similarity queries. Its limitations are discussed in the next section.

2. Knowledge Base Retrieval

For structured data, SQL-based or NoSQL lookup provides deterministic, schema-bound retrieval. A natural language query is parsed into a structured query (often using an LLM as a text-to-SQL translator), which is then executed against a relational or document database.

This approach is reliable for fact lookup and returns exact, verifiable answers. However, it requires a predefined schema, cannot traverse relationships across tables without explicit joins, and becomes unwieldy for queries that span multiple entity types.

3. Knowledge Graph Retrieval

Graph-based retrieval enables path traversal across a network of typed relationships. The typical pipeline begins with entity linking (mapping terms in the query to nodes in the graph), followed by graph traversal to collect a relevant subgraph, which is then passed as context to the LLM.

This single query traverses two hops and returns the answer directly. The graph structure makes the reasoning path explicit and auditable. Some systems also use Text2Cypher, an LLM that translates natural language queries into Cypher graph queries, to make this accessible without requiring users to write graph query language.

Why Vector RAG Fails for Multi-Hop Reasoning

Vector retrieval is the workhorse of most RAG systems, but it has a fundamental architectural mismatch with multi-hop reasoning tasks. As one 2026 analysis put it bluntly:

The core problem is what researchers call semantic density failure: the embedding space collapses relationships that are logically distinct but semantically similar. Four specific failure modes have been documented in production systems:^[5]

A knowledge graph avoids these failures because it stores explicit, typed relationships rather than geometric proximity. The path Radium → used_for → Cancer treatment cannot be lost to chunking or embedding drift: it is a first-class fact in the graph. This is the core architectural argument for graph-based retrieval in systems that need to answer relational questions reliably.

Note
This does not mean vector retrieval is useless. For semantic similarity queries such as "find documents about climate policy", vector search remains the best tool. The failure modes above are specific to relational reasoning tasks. The practical implication is that production systems should use both, routing queries to the appropriate retrieval strategy based on their type.

GraphRAG: The Emerging Standard

The most significant development in knowledge-augmented LLM systems over the past two years is GraphRAG (Graphs + Retrieval Augmented Generation), a technique introduced by Microsoft Research in early 2024 and open-sourced in July 2024^[2] GraphRAG has become the reference architecture for combining LLMs with knowledge graphs, and understanding it is essential for anyone working on production RAG systems in 2026.

GraphRAG is a retrieval-augmented generation technique that builds a knowledge graph from unstructured text using LLMs, applies community detection algorithms to cluster entities into thematic groups, generates LLM summaries of each community, and uses these summaries to enable both entity-level and corpus-level question answering.

The Pipeline

GraphRAG operates in two phases: an offline indexing phase that builds the knowledge graph and community summaries, and an online query phase that uses them to answer questions.

During indexing, the system processes a corpus of unstructured text through the following steps:

1. Entity and relation extraction. An LLM reads each document and extracts entities and typed relationships, producing a set of triples (𝒉, 𝒓, 𝒕).

2. Knowledge graph construction. The extracted triples are assembled into a graph, with deduplication and conflict resolution applied.

3. Community detection. The Leiden algorithm^[14] partitions the graph into communities: clusters of densely connected entities that tend to share a common theme or domain.

4. Community summarization. An LLM generates a natural language summary of each community, capturing the key entities, relationships, and themes it contains.

Figure 5. The GraphRAG pipeline. The indexing phase (top) builds a knowledge graph and community summaries offline. The query phase (bottom) routes incoming queries to one of three search strategies (Local Search for entity-level queries, Global Search for corpus-wide synthesis, and DRIFT Search for hypothesis-driven exploration) before passing the retrieved context to the LLM.

The key innovation of GraphRAG is the community summary. Standard RAG can answer questions about specific entities (local search), but it struggles with questions that require synthesizing information across an entire corpus, for example: "What are the main themes in this collection of research papers?" Community summaries enable global search: by querying the summaries of all communities, the system can answer corpus-wide questions that no single retrieved chunk could support.

At query time, GraphRAG routes each query to one of three search modes:

- Local Search. For entity-level queries, the system links the query to specific nodes in the graph and retrieves their local neighborhood: connected entities, relationships, and associated text chunks.

- Global Search. For corpus-wide synthesis queries, the system queries the community summaries and aggregates the results using an LLM.

- DRIFT Search (introduced November 2024). A hybrid mode that begins with a global search to form hypotheses and then uses local search to verify and refine them, combining the breadth of global search with the precision of local search.

Variants and the Cost Problem

The original GraphRAG had a significant adoption barrier: indexing a large dataset could cost up to $33,000 in LLM API calls, because every entity and community required LLM processing during the offline phase.^[6] This made it impractical for most teams outside of large enterprises.

Microsoft Research addressed this with LazyGraphRAG (November 2024), which defers community summarization until query time rather than precomputing it. This reduces indexing costs by 10–90% while maintaining competitive accuracy, making production deployment viable for a much wider range of use cases.

GraphRAG is now available as an open-source library on GitHub and is integrated into Microsoft Discovery, an agentic platform for scientific research built on Azure.^[2]

KB vs KG in GenAI Systems

With the above context established, we can now compare knowledge bases and knowledge graphs directly across the dimensions that matter most for LLM system design.

In practice, the choice between knowledge bases and knowledge graphs is rarely binary. Most production systems combine both, using knowledge bases for high-volume semantic retrieval and knowledge graphs for relational reasoning and explainability. The question is not "which one?" but "which one for which query type?"

Practical Architecture: Hybrid Retrieval

The most effective production architectures in 2025-2026 are hybrid systems that combine vector retrieval, knowledge graph traversal, and structured database lookup. Rather than choosing a single retrieval strategy, these systems route each query to the most appropriate backend based on its type.^{[7] [8]}

Figure 6. A hybrid retrieval architecture combining vector search, knowledge graph traversal, and SQL lookup. A query router (typically an LLM classifier) directs each incoming query to the appropriate retrieval backend. Results from all active backends are assembled and re-ranked before being passed to the LLM for generation.

The query router is the critical component. It classifies each incoming query and dispatches it to one or more retrieval backends:

For teams building these systems, the practical tooling landscape includes: Neo4j and Amazon Neptune for graph storage; Pinecone, Weaviate, and Meilisearch for vector search; LangChain's graph transformer and Neo4j's LLM Knowledge Graph Builder for automated graph construction; and the open-source Microsoft GraphRAG library for the full GraphRAG pipeline.

Agentic RAG and the Future

The most forward-looking development in this space is Agentic Graph RAG, where autonomous AI agents dynamically orchestrate retrieval strategies rather than following a fixed pipeline.^[4] This represents the convergence of everything discussed in this post: knowledge bases, knowledge graphs, and vector retrieval all become tools that an intelligent agent selects from based on the nature of the question.

Agentic Graph RAG is a retrieval architecture in which an LLM-based agent dynamically decomposes complex queries into sub-queries, selects appropriate retrieval tools (vector search, graph traversal, SQL lookup, web search) for each sub-query, iteratively refines its retrieval based on intermediate results, and synthesizes a final answer with an explicit reasoning trace.

Figure 7. The Agentic Graph RAG architecture. An LLM-based orchestrator dynamically selects from a toolkit of retrieval strategies (vector search, graph traversal, SQL lookup, and web search) based on the nature of each sub-query. The knowledge graph also serves as long-term memory across sessions. Results are synthesized into a final answer with a reasoning trace.

Unlike static GraphRAG, which follows a predetermined pipeline, agentic systems can:

- Decompose complex questions into a sequence of simpler sub-queries, each handled by the most appropriate retrieval tool.

- Iterate on retrieval. If the first retrieval attempt returns insufficient context, the agent can reformulate the query or try a different backend.

- Maintain persistent memory by writing newly discovered facts back to the knowledge graph, enabling reasoning that spans multiple sessions.

- Produce reasoning traces that explain not just the answer but the sequence of retrieval and reasoning steps that produced it.

A NeurIPS 2025 workshop titled "Knowledge Graphs & Agentic Reasoning" (NORA) was dedicated entirely to this convergence, signalling that it is now a mainstream research direction.^[12] The KG-RAG-Agent framework (EAMCON 2025) demonstrated that combining knowledge graphs with agentic retrieval improved multi-hop question answering accuracy significantly over both pure vector RAG and static GraphRAG.^[13]

Key Takeaways

• Knowledge bases store structured facts as records; they are fast and scalable but cannot natively represent relationships between entities.

• Knowledge graphs encode facts as typed triples  (𝒉, 𝒓, 𝒕) making relationships first-class data and enabling multi-hop reasoning through graph traversal.

• Vector RAG fails for relational reasoning tasks due to chunk fragmentation, semantic collapse, temporal blindness, and identifier failure. It remains the best tool for semantic similarity queries.

• GraphRAG (Microsoft Research, 2024) is the current reference architecture for combining LLMs with knowledge graphs. LazyGraphRAG reduces indexing costs by 10–90%, making production deployment practical.

• LLMs can now build knowledge graphs automatically from unstructured text, removing the primary historical barrier to KG adoption.

• Hybrid architectures that combine vector search, graph traversal, and SQL lookup behind a query router are the dominant production pattern in 2025–2026.

• Agentic Graph RAG is the emerging frontier: LLM agents that dynamically orchestrate all retrieval strategies, with the knowledge graph serving as persistent, structured memory.

References