Vector search tells you which chunks are similar to your query. GraphRAG tells you how the entities in your corpus relate to each other. Those are different questions, and most teams reach for the graph before confirming they’re actually asking the second one.

The Question Multi-Hop Retrieval Can’t Answer

The BM25 vs. Vector Search post in this series established a decision framework for choosing between keyword and semantic retrieval — and the Agentic RAG post showed how iterative retrieval loops handle multi-hop queries by chaining multiple retrieval passes together. Both posts share an assumption worth examining directly: that the unit of retrieval is the chunk, and that relationships between facts are something the LLM reconstructs at generation time, from whatever chunks happen to land in context together.

That assumption breaks down on a specific, common class of query: questions whose answer depends on traversing a relationship, not matching a similarity. “Which suppliers does our highest-risk vendor share ownership with?” “What’s the chain of approvals that led to this incident?” “Which engineers have worked on every service that touched this outage?” None of these are well-served by retrieving the top-K most similar chunks — the answer isn’t in any single chunk, similar or not. It exists in the structure connecting multiple entities across the corpus, and flat retrieval has no representation of that structure to retrieve.

This is the problem GraphRAG was built to solve: replace or augment chunk-based retrieval with a knowledge graph — entities as nodes, relationships as edges — that the system can traverse to answer structural and relational questions a similarity search cannot.

GraphRAG has had a rough public history. Microsoft’s original 2024 implementation produced genuinely strong accuracy gains but came with an indexing cost that made it impractical for most teams — figures in the tens of thousands of dollars for large corpora were widely reported, enough to scare off adoption outside well-funded research teams. A 2026 analysis found that 72% to 80% of enterprise RAG implementations are failing to reach production, and graph construction overhead was a recurring contributor — teams found that knowledge extraction pipelines hallucinate entities and relationships, producing brittle structures that need expensive manual correction, and that graph construction consumes compute budget before a single question gets answered.

Two years on, the picture has changed. This post covers what GraphRAG actually buys you, the benchmark evidence for when that trade-off is worth it, how the cost problem got solved, and a concrete decision framework for choosing between flat retrieval and graph retrieval — because for most corpora, the honest answer is still flat retrieval, and knowing why is as valuable as knowing how to build the graph.

Article context: This post extends the retrieval series. The BM25 vs. Vector Search post established the decision framework for keyword vs. semantic retrieval — GraphRAG is best understood as a third option in that same framework, not a replacement for the other two. The Agentic RAG post’s multi-hop query planning is the alternative path to the same problem GraphRAG addresses; Section 6 covers directly when to reach for one over the other.

Table of Contents

1. What a Knowledge Graph Actually Adds

A knowledge graph represents your corpus as entities (nodes) and the relationships between them (edges), extracted from the unstructured text during ingestion rather than left implicit in prose. A document that says “Acme Corp acquired Northwind Logistics in 2023, and Northwind’s former CEO now sits on Acme’s board” gets decomposed into structured facts: (Acme Corp)-[ACQUIRED]->(Northwind Logistics), (Northwind Logistics)-[HAD_CEO]->(Person X), (Person X)-[SITS_ON_BOARD_OF]->(Acme Corp).

This decomposition is the entire value proposition, and it’s worth being precise about what it does and does not buy you.

What it buys you: the ability to traverse relationships that span multiple documents or multiple mentions within documents, without relying on those relationships co-occurring in a single retrievable chunk. The board-membership fact above might be one sentence in a press release from 2023; the acquisition might be described in an entirely different filing from a different year. Flat retrieval would need both chunks to land in the same context window for the LLM to connect them — and it would need to “get lucky” that a semantically-similar-to-the-query search surfaces both. Graph traversal doesn’t depend on luck: it follows the edge.

What it does not buy you: better answers to questions that are genuinely about a single fact in a single place. If the question is “what was Acme’s revenue in 2023,” that’s a flat lookup — the graph adds extraction cost and traversal complexity for zero benefit, because there’s no relationship to traverse. Knowledge-based GraphRAG extracts detailed knowledge graphs from the corpus using entity recognition and relation extraction, offering fine-grained, domain-specific information — but fine-grained relational structure is wasted machinery on a query that was never relational to begin with.

The architectural distinction worth holding onto: vector and BM25 retrieval answer “what text is relevant to this query.” Graph retrieval answers “what is connected to this entity, and how.” These are genuinely different retrieval primitives, not two implementations of the same idea at different sophistication levels — which is why the right mental model is not “GraphRAG is RAG, but better.” It’s “GraphRAG is a different retrieval primitive, applicable when your queries are about relationships rather than content.”

2. The Benchmark Evidence — When GraphRAG Helps and When It Doesn’t

Hype-cycle claims about GraphRAG accuracy gains have been common since Microsoft’s original announcement, and some are well-supported, some are cherry-picked from favorable benchmarks, and most don’t specify the query class the gains apply to. The GraphRAG-Bench evaluation — a benchmark purpose-built to test graph retrieval against flat retrieval across stage-specific evaluation metrics that provide granular insight into how well a GraphRAG model performs at each phase of its pipeline rather than relying solely on end-to-end accuracy — gives a more honest picture than most vendor claims, precisely because it isolates whether failures stem from flawed knowledge graph construction, suboptimal retrieval, or weak reasoning, rather than treating the whole pipeline as a black box.

The pattern that emerges across this and related benchmarks is consistent: GraphRAG’s advantage is concentrated in multi-hop and relational query classes, and close to nonexistent — sometimes negative once you account for the extraction cost — on single-fact lookup queries. Graph-aware retrieval mechanisms enable multi-hop reasoning and context-preserving knowledge acquisition specifically because they capture entity relationships and domain hierarchies explicitly rather than leaving them to be reconstructed at generation time.

What this means for your corpus: before building any graph infrastructure, characterize your actual query distribution. A practical audit:

QUERY_TYPE_CLASSIFIER_PROMPT = """Classify whether this question requires relational reasoning
(traversing a connection between entities) or single-fact lookup (one piece of information,
no relationship traversal needed).

Question: {question}

Respond with JSON: {{"type": "relational" | "single_fact", "entities_involved": [...], "reasoning": "..."}}"""

# Run this classifier against a sample of 200+ real production queries before building anything.
# If relational queries are under 15% of traffic, GraphRAG's benchmark advantage will not
# materialize at your system's actual query mix — the corpus-wide extraction cost still applies
# to 100% of documents regardless of what fraction of queries benefit.

This audit is the single highest-leverage thing you can do before committing to graph infrastructure. Teams that skip it consistently report disappointing results — not because GraphRAG doesn’t work, but because their query distribution was never the distribution the benchmarks were measuring.

3. Graph Construction — Where the 2024 Cost Problem Lived

The original GraphRAG cost complaint was specific: extracting a knowledge graph from a large corpus requires running an LLM extraction pass over every document, identifying entities and relationships, then a second LLM pass to summarize clusters of related entities into hierarchical community summaries (Microsoft’s approach used the Leiden community-detection algorithm for this clustering step). Both passes scale with corpus size and both use LLM calls priced per token — which is why Microsoft’s GraphRAG sparked intense interest in 2024, but its $33K indexing cost for large datasets made it impractical for most teams.

The extraction pipeline, conceptually:

ENTITY_EXTRACTION_PROMPT = """Extract entities and relationships from this text chunk.
Entities: people, organizations, products, locations, events.
Relationships: how entities connect (acquired, employs, located_in, caused, depends_on).

Text: {chunk_text}

Respond with JSON:
{{
  "entities": [{{"name": "...", "type": "...", "id": "..."}}],
  "relationships": [{{"source": "entity_id", "target": "entity_id", "type": "...", "evidence": "quote"}}]
}}"""

# This prompt runs once per chunk during ingestion — for a 10,000-document corpus
# chunked at ~500 tokens each, that's tens of thousands of LLM calls before
# a single user query has been answered. This is the cost Microsoft's 2024
# implementation incurred at scale, and it is unavoidable in any approach
# that uses an LLM for extraction — the question is how much you can reduce
# the per-chunk cost and how much of the corpus actually needs it.

Two compounding costs sit inside this pipeline. The extraction cost scales linearly with corpus size and runs regardless of query patterns — you pay it for every document whether or not future queries ever touch that document’s entities. The community summarization cost is worse: hierarchical clustering followed by LLM-generated summaries at each level of the hierarchy means the summarization cost scales not just with corpus size but with how richly connected the entity graph turns out to be — a densely interconnected corpus produces more communities, more summary levels, and more summarization calls than the same volume of sparser, less-connected text.

This is also where the quality risk lives, not just the cost risk. Extraction is an LLM call making a judgment about whether two mentions refer to the same entity, whether a sentence implies a relationship or merely a co-occurrence, and how to type that relationship. Every one of those judgments can be wrong, and wrong extractions don’t fail loudly — they sit in the graph as confidently-wrong edges that future traversals will treat as ground truth. Knowledge extraction pipelines hallucinate entities and relationships, creating brittle graph structures that require expensive manual correction.

4. How 2026 Implementations Cut the Cost

The research and tooling landscape spent the two years since Microsoft’s release attacking this cost problem from multiple angles simultaneously, and the combination of techniques is why GraphRAG is viable for mid-sized teams in 2026 in a way it wasn’t in 2024.

Selective extraction, not corpus-wide extraction. The single highest-leverage fix: run entity/relationship extraction only on documents likely to contain relational content, identified by a cheap pre-classification pass, rather than every document uniformly. Most corpora are not uniformly relational — financial filings and organizational documents tend to be entity-dense, while procedural documentation and FAQs tend not to be. Classifying first and extracting selectively avoids paying extraction cost on the fraction of the corpus that will never benefit from it.

Lightweight extraction models for the first pass. Use a fast, cheap model for the bulk extraction pass and reserve the expensive model for ambiguous cases the cheap model flags with low confidence — the same cheap-model-first pattern used for query routing in the agentic RAG post, applied to ingestion instead of query time.

Relation-free or lightweight graph construction. A relation-free graph construction method for efficient GraphRAG was released in late 2025 specifically to reduce construction overhead — building entity co-occurrence structure without running full relationship-typing extraction on every edge, then layering relationship typing only where a downstream query actually needs to traverse a specific edge type. This defers cost from ingestion time to query time, where it’s incurred only for graphs that get used.

Hybrid extraction — classical NLP plus LLM, not LLM alone. Recent work demonstrates that careful engineering of classical NLP techniques can match modern LLM-based approaches while enabling practical, cost-effective, and domain-adaptable retrieval-augmented reasoning at scale — meaning named-entity recognition and dependency parsing (cheap, deterministic, decades-old NLP techniques) handle the bulk of entity identification, with LLM calls reserved for relationship typing and disambiguation where classical methods are weakest. This dual-extraction approach cuts the LLM call volume substantially versus full LLM-based extraction.

The combined effect of these techniques is reported in the 10–90% cost reduction range depending on corpus characteristics and which techniques are stacked — a meaningful enough shift that GraphRAG moved from “research-budget-only” to “evaluate seriously for production” for teams with genuinely relational query distributions.

5. Graph Retrieval — Traversal Patterns That Matter

Once the graph exists, retrieval means traversing it for a given query — and the traversal pattern matters as much as the construction quality. Three patterns cover most production use cases.

Local search — entity-centric traversal. Given a query that names or implies a specific entity, retrieve that entity’s node, its directly connected neighbors, and the source text that justified each edge. This is the graph equivalent of a targeted lookup and handles questions like “who are Acme’s board members” — start at the Acme node, traverse SITS_ON_BOARD_OF edges inward, return the connected Person nodes plus their source evidence.

Global search — community-level traversal. For broad, corpus-wide questions (“what are the major themes in our compliance filings this year”), traverse the hierarchical community summaries rather than individual entities — the pre-computed cluster summaries from the construction phase answer aggregate questions without re-reading the entire corpus at query time. This is the pattern that justifies the community summarization cost from Section 3; if your queries never need this aggregate view, that cost was wasted.

Multi-hop path traversal. For genuinely relational questions spanning more than one edge — “which suppliers does our highest-risk vendor share ownership with” — traverse from the vendor node, through ownership edges, to connected entities, then back out through their supplier relationships. This is the traversal pattern that single-pass and even agentic flat retrieval cannot replicate, because it depends on the graph’s explicit edge structure, not on text similarity.

# Local search: targeted entity lookup
async def local_search(entity_name: str, graph_client, hop_depth: int = 1) -> dict:
    entity_node = await graph_client.find_entity(entity_name)
    neighbors = await graph_client.get_neighbors(entity_node.id, depth=hop_depth)
    return {
        "entity": entity_node,
        "neighbors": neighbors,
        "evidence": [n.source_text for n in neighbors],   # always carry provenance forward
    }

The evidence field above is not optional decoration — it is the traceability that flat retrieval gives you for free (the chunk you retrieved is its own evidence) and that graph retrieval must construct deliberately, because the answer to a traversal query is a derived structure, not a quoted passage. Every edge in your graph should carry a pointer back to the source text that justified it, and every traversal result should surface that provenance alongside the structural answer. Skipping this is the single most common reason GraphRAG systems lose the faithfulness properties the LLM Evaluation in Production post’s RAGAS faithfulness metric depends on — faithfulness checking needs a context to check claims against, and “the graph said so” is not a checkable context unless the source text travels with the traversal result.

6. GraphRAG vs. Agentic Multi-Hop Retrieval

The Agentic RAG post solved multi-hop questions a different way: iterative retrieval, where the reflection agent identifies a gap and issues a new retrieval query to fill it, repeating until sufficient context accumulates. This is a legitimate alternative to graph traversal for the same underlying problem class, and the two approaches have real trade-offs against each other that are worth stating plainly rather than treating GraphRAG as a strict upgrade.

Agentic multi-hop retrieval pays its cost at query time, incrementally, only for queries that actually need it — there is no corpus-wide preprocessing investment, and the system degrades gracefully (worse latency, not wrong answers) if the multi-hop reasoning required is more complex than anticipated. Its weakness is that it depends on the LLM correctly inferring which follow-up query would close the gap, which is a soft, probabilistic process — two runs of the same complex multi-hop question can take different reasoning paths and surface different evidence.

GraphRAG pays its cost at ingestion time, once, for the whole corpus — and in exchange gets deterministic traversal: the same multi-hop question, traversed the same way, returns the same connected entities every time, because the relationship structure is explicit rather than inferred fresh on each query. Production AI systems require deterministic reasoning, traceability, and live context, none of which flat vector search provides reliably — and that determinism argument extends to agentic flat retrieval too, since its multi-hop reasoning is still inference, not traversal.

The practical decision rule: if your multi-hop queries are occasional, varied in shape, and tolerant of some answer-quality variance, agentic retrieval is the lower-cost, lower-commitment choice. If your multi-hop queries are frequent, follow a small number of recurring relational patterns (ownership chains, approval chains, dependency chains), and need consistent, auditable answers — particularly in compliance, risk, or audit contexts where “the system gave a different answer last time” is itself a problem — the graph’s one-time construction cost buys you a retrieval primitive that agentic inference structurally cannot guarantee: the same query, traversed the same way, every time.

7. The Hybrid Architecture — Graph and Vector Together

In production, GraphRAG is almost never a replacement for vector or BM25 retrieval — it is a third retrieval tool selected by a router, exactly the role vector and keyword search played against each other in the BM25 post’s decision framework. Most real queries are not purely relational; they have a relational component and a content-similarity component, and the strongest production architectures route to graph traversal, vector search, or both, depending on query shape.

HYBRID_ROUTER_PROMPT = """Classify retrieval strategy for this query.

Query: {query}

- GRAPH_ONLY: purely relational — asks about connections, ownership, hierarchy, "who is related to X"
- VECTOR_ONLY: purely content-similarity — asks about facts, definitions, explanations
- HYBRID: needs both — e.g. "summarize the risk profile of companies connected to X"

Respond with JSON: {{"strategy": "GRAPH_ONLY" | "VECTOR_ONLY" | "HYBRID", "reasoning": "..."}}"""

async def hybrid_retrieve(query: str, strategy: str, graph_client, vector_client) -> dict:
    if strategy == "GRAPH_ONLY":
        return await local_search(extract_entity(query), graph_client)
    if strategy == "VECTOR_ONLY":
        return {"chunks": await vector_client.search(query)}
    # HYBRID: traverse the graph for structure, then vector-search within the
    # neighborhood it surfaces — narrowing the similarity search to a relevant
    # subset rather than the whole corpus
    graph_result = await local_search(extract_entity(query), graph_client)
    neighborhood_ids = [n.source_doc_id for n in graph_result["neighbors"]]
    chunks = await vector_client.search(query, filter_doc_ids=neighborhood_ids)
    return {"graph_context": graph_result, "chunks": chunks}

The hybrid path’s value is specific: it uses the graph to narrow the search space to a structurally relevant neighborhood, then uses vector similarity to find the most content-relevant passages within that neighborhood — combining “what’s connected” with “what’s similar” rather than choosing one. This is meaningfully better than either retrieval primitive alone for queries that genuinely have both a relational and a content dimension, and it’s the architecture pattern most production GraphRAG deployments converge on once they move past the “graph vs. vector” framing and into “graph and vector, routed.”

8. Production Failure Modes Specific to Graphs

Beyond the extraction-cost and hallucinated-edge risks already covered, graph-based retrieval introduces failure modes that flat retrieval simply doesn’t have, because a graph is a stateful, evolving structure rather than an immutable set of chunks.

Entity resolution drift. “Acme Corp,” “Acme Corporation,” and “Acme” need to resolve to the same node, or the graph fragments into disconnected partial views of the same entity — and the fragmentation gets worse, not better, as the corpus grows and more documents introduce more name variants. Entity resolution needs ongoing maintenance, not a one-time pass at construction.

Stale edges from a changing world. A SITS_ON_BOARD_OF relationship extracted from a 2023 document is true until it isn’t, and unlike a chunk (which is just “this is what the document said, dated then”), a graph edge implicitly reads as a current-state assertion unless every edge explicitly carries its as-of date and your traversal logic respects it. Corpus refresh strategies for graphs need an edge-invalidation policy, not just new-document ingestion.

Community summary cascade. Because community summaries are built hierarchically from the entity graph, a correction to the underlying entity extraction — fixing one hallucinated edge — invalidates every community summary built above it in the hierarchy. Unlike a flat vector index, where re-indexing one corrected chunk is a local, contained operation, correcting a graph error can mean re-running summarization at multiple hierarchy levels above it.

The GraphRAG Decision Checklist

Before committing to knowledge graph infrastructure:

  • Query distribution audited — relational vs. single-fact queries classified across 200+ real production queries; relational share quantified, not assumed
  • Relational query share justifies the investment — extraction cost applies to the whole corpus regardless of what fraction of queries are relational
  • Selective extraction in place — only documents likely to contain relational content go through the expensive extraction pass
  • Cheap-model-first extraction pipeline — lightweight model handles bulk extraction; expensive model reserved for low-confidence cases
  • Every edge carries source provenance — traversal results include the originating text, not just the structural fact
  • Entity resolution has an ongoing process — not a one-time construction-phase pass; new name variants get merged as the corpus grows
  • Edges carry as-of dates — traversal logic respects temporal validity rather than treating every edge as a current-state assertion
  • Hybrid router implemented — graph, vector, and hybrid paths selected per query, not graph-only or vector-only as a blanket policy
  • Agentic multi-hop retrieval evaluated as the alternative — for occasional, varied relational queries, before committing to graph construction cost
  • Correction cascade understood — a fix to one extraction error’s blast radius across dependent community summaries is known before it happens in production, not discovered then

Closing: The Graph Is a Cost Center Until Proven Otherwise

The honest framing for GraphRAG in 2026 is not “GraphRAG vs. RAG” as a single architectural choice made once. It’s a third retrieval primitive, with a real one-time construction cost and real ongoing maintenance burden, that earns its place only when a measurable fraction of your actual query traffic is asking questions a similarity search structurally cannot answer — questions about how entities connect, not which text resembles a query.

The benchmark evidence supports graph retrieval clearly for that query class. It does not support reaching for a knowledge graph as a default upgrade path for every RAG system that feels limited, and the 2024 cost history is a useful corrective against treating it as one: a $33K indexing bill is what happens when a powerful tool gets applied to a corpus that never needed it.

Audit the query distribution first. If the relational share is small, the agentic multi-hop pattern from the prior post in this series gets you most of the benefit at a fraction of the commitment. If the relational share is significant and the relationships are recurring, auditable patterns rather than one-off curiosities, the graph’s deterministic traversal and one-time construction cost become the better trade — and the cost-reduction techniques from Section 4 make that trade considerably more affordable than it was when Microsoft’s original implementation set the field’s expectations in 2024.

📌 Key Takeaway

GraphRAG answers a different question than vector or keyword retrieval: not “what text is similar to this query” but “what is connected to this entity, and how.” That makes it valuable specifically for relational, multi-hop query classes — and close to worthless, while still carrying full extraction cost, for single-fact lookups. Audit your actual query distribution before building graph infrastructure; the benchmark gains are real but concentrated in a query class that may be a small fraction of your traffic. Where it does apply, treat construction cost as a budget to actively manage — selective extraction, cheap-model-first pipelines, and hybrid classical-NLP-plus-LLM extraction are what moved GraphRAG from a $33K research expense in 2024 to a viable production investment in 2026. And route to it alongside vector and keyword search, not instead of them — most real queries have a content dimension, a relational dimension, or both.

Further Reading: Edge et al. — From Local to Global: A Graph RAG Approach to Query-Focused Summarization (Microsoft Research, 2024), Peng et al. — Graph Retrieval-Augmented Generation: A Survey (arXiv, 2024), Wang et al. — GraphRAG-Bench: When to use Graphs in RAG (arXiv, 2025), FalkorDB — GraphRAG SDK Documentation, Neo4j — Knowledge Graph Construction Guide