RAG, GraphRAG, or something else: choosing without overbuilding

The same progression appears in many conversations about AI products. First, we connect a model to a few documents. Then somebody says we need RAG. Soon after that, GraphRAG arrives, usually next to a diagram full of nodes, and the solution starts growing before anyone has measured whether the original search worked badly.

I prefer to start with a less exciting but much more useful question: what information does the answer need, and where does that information actually live?

If the answer is in 1 passage of 1 document, conventional RAG is usually enough. If we need to reconstruct relationships spread across hundreds of documents or understand the themes of an entire collection, a graph may help. If the data already lives in PostgreSQL, I probably want to query PostgreSQL. And if all the relevant material fits comfortably inside the context window, I may not need an index at all.

GraphRAG is not the next version of RAG. It is another way to build context for another class of questions.

The example I will use

Imagine a restaurant group that has accumulated 4 kinds of information:

• recipe sheets, allergen information, and front-of-house and kitchen procedures;
• reviews, complaints, and reports written by each shift manager;
• supplier invoices, catalogues, and delivery incident reports;
• sales, bookings, and stock in a database.

We want to build an internal assistant. These 4 questions look similar because they are written in natural language, but they do not ask for the same work:

1. How should we prepare the kitchen for a guest with a sesame allergy? The answer is in a specific procedure.
2. Which suppliers, dishes, and restaurants repeatedly appear in complaints about unavailable dishes? We need to connect entities and facts scattered across the collection.
3. How many dinner covers did the city-centre restaurants serve yesterday, and what was the average spend? We need an exact calculation over current, structured data.
4. Compare these 6 supplier proposals I have just attached. The document set is small, temporary, and fits in context.

I would solve each one differently. Making the same vector store answer all 4 is a good way to build a convincing demo and an unreliable product.

What RAG actually does

The paper that introduced RAG combined a model’s parametric memory with retrievable external memory. In a current application, the name usually describes a broader pipeline:

documents -> chunks -> index -> retrieval -> context -> answer

During ingestion, we split documents into chunks, preserve metadata, and build an index. When a question arrives, we retrieve the most promising chunks and pass them to the model with instructions to answer and cite its sources.

The index does not have to be purely vector based. Embeddings work well when the question and the source share meaning but not wording. Lexical search, such as BM25, is often better for identifiers, error names, versions, and exact terms. In practice, I like to start with hybrid search, apply metadata filters, and use a reranker when needed before filling the context window.

RAG fits the sesame question very well. The user may write “stop an allergy reaching the plate” while the manual calls it the “allergen and cross-contamination protocol”. Semantic search can bridge that vocabulary gap and return the right section.

Possible RAG questions and answers

These answers are invented, but they show the shape I would expect from RAG over the restaurant documents: short, specific, and linked to a source we can open.

Question	Potential answer
What should I do when a guest reports a sesame allergy?	Follow protocol AL-04, notify the kitchen manager, and add the alert to the booking. [Allergen manual, p. 3]
Does the house cake contain nuts?	Yes. Its recipe lists almonds and warns of possible cross-contamination. [Recipe REC-31]
What is the procedure when a cold room exceeds its limit?	Isolate the affected product and record the temperature and time before deciding whether to discard it. [Procedure COLD-02]

When I would choose conventional RAG

RAG is a good starting point when:

• the answer is usually contained in a few passages;
• the document collection changes and we want to update knowledge without retraining the model;
• showing the original source matters;
• questions are local: a policy, procedure, specification, or individual case;
• we can filter by permissions, product, version, language, or date before retrieving text.

Product documentation, knowledge bases, manuals, internal policies, and support systems usually start here.

Why simple RAG fails

Generation gets most of the attention, but many failures happen earlier. If we do not retrieve the evidence, the model cannot use it.

A chunk may say “the limit increases to 200” while losing the plan name, version, or date from 2 pages earlier. An embedding may return 10 similar passages and miss the only one containing an exception. Semantic search may treat an exact identifier as noise. And an arbitrary top_k = 5 may be too small for a comparison and too large for a simple lookup.

Before building a graph, I would try these things in order:

1. preserve titles, hierarchy, date, version, and permissions as metadata;
2. split documents according to their structure, not only every fixed number of tokens;
3. combine semantic and lexical search;
4. rewrite ambiguous queries, apply filters, and rerank the results;
5. add document context to each chunk;
6. measure retrieval with real questions.

Anthropic’s Contextual Retrieval is a useful example of step 5. It prepends a short explanation of each chunk’s place in the document before building the semantic and lexical indexes. It does not turn RAG into GraphRAG. It simply avoids losing important information when the document is cut apart.

Quite often, the quality jump is there: fixing representation and retrieval instead of changing the whole architecture.

What GraphRAG adds

Conventional RAG looks for chunks similar to the question. That works for local questions, but it struggles with global ones such as “which patterns appear across these 3 years of complaints and shift reports?”. No individual passage contains the answer because the answer is a property of the collection.

The GraphRAG approach published by Microsoft Research first builds an intermediate representation. Simplified, it looks like this:

documents
  -> entities + relationships + claims
  -> communities of related entities
  -> summaries for each community
  -> local, global, or mixed search
  -> answer

This enables 2 particularly useful kinds of query:

• Local: start from an entity and traverse its neighbours. For example, “which dishes, suppliers, and restaurants are connected to availability problems with this ingredient?”.
• Global: combine community summaries to answer something about the entire collection. For example, “which operational patterns recur across customer complaints?”.

Possible GraphRAG questions and answers

Here, the answer does not come from 1 passage. It is assembled by connecting complaints, shift reports, dishes, ingredients, suppliers, and restaurants. The numbers are still fictional.

Question	Potential answer
What connects the complaints about unavailable dishes?	9 of 14 complaints involve dishes supplied by the same vegetable supplier and are concentrated in the City Centre and North restaurants.
Which pattern recurs on Friday evenings?	Late Friday deliveries are connected to missing ingredients, menu changes, and more complaints during dinner service.
What would happen if we stopped using this supplier?	6 dishes across 3 restaurants depend on its products, and 2 have no alternative supplier recorded.

Microsoft also maintains DRIFT Search, which mixes community information with local search and follow-up questions. The distinction matters more than the names: sometimes we need precision around an entity; at other times we need to widen the view and synthesise the collection.

When I would choose GraphRAG

I would consider GraphRAG when several of these conditions are true:

• relationships between entities are part of the answer, not decorative metadata;
• the facts we need are distributed across many documents;
• users ask global, discovery, or thematic analysis questions;
• users want to navigate from a person, restaurant, supplier, dish, ingredient, or event to its connections;
• the document collection is stable and valuable enough to justify more expensive indexing;
• we can evaluate the extracted entities, relationships, and summaries.

Fraud investigation, threat intelligence, literature analysis, due diligence, organisation networks, and operational analysis across a restaurant group are reasonable candidates. “Chat with 40 PDFs” is not automatically one.

The graph creates a new class of problems too

GraphRAG does not discover the true structure hidden inside the documents. It builds an interpretation of that structure.

That introduces difficult decisions. City Centre Restaurant, City Centre Branch, and Central Location may be the same entity or 3 different ones. A relationship may be implicit, temporary, or negated. The model may omit it or extract it incorrectly. If a community summary flattens an important exception, that error flows into global answers.

There is a real operational cost too. Standard GraphRAG indexing uses models to extract entities and relationships, summarise descriptions, and generate community reports. Its own documentation offers a faster method because graph extraction accounts for roughly 75% of the standard indexing cost. When the collection changes, we must decide what to recompute, how to version it, and how to preserve permissions and provenance through the derived structure.

A model-generated graph should not quietly become the company’s source of truth either. I would always retain the path from every entity, relationship, and summary to the original text. The graph helps find and organise evidence; the evidence still lives in the sources.

If the domain already has explicit and reliable relationships, such as suppliers, ingredients, dishes, restaurants, and purchase orders, I would start from those tables and events instead of asking a model to reconstruct them from prose.

When to use something else

The useful decision is not only RAG versus GraphRAG. Nearby problems are often better solved with neither.

1. Long context for a small, temporary document set

If the purchasing manager attaches 6 supplier proposals and wants a 1-off comparison, I would pass them directly to the model as long as they fit comfortably in its context window. I avoid permanent ingestion, preserve complete order and structure, and the system disappears when the request is finished.

This does not mean stuffing every possible document into every prompt. More context increases cost and latency, and relevant details can get buried. For small collections, though, the simplicity is hard to beat.

2. SQL, APIs, or tools for structured data

“How many dinner covers did the city-centre restaurants serve yesterday, and what was the average spend?” is not a semantic similarity question. It is a filter and an aggregation.

I would have the model produce a call to a validated tool or use a semantic layer with defined metrics. The result should come from the database, not from chunks of dashboards indexed weeks ago. That gives us current numbers, deterministic operations, and the authorisation rules the system already understands.

RAG can retrieve the internal definition of a “served cover”; SQL calculates how many there were and their average spend. The architectures can be composed.

Possible SQL questions and answers

On this route, I do not want an approximate synthesis. I want the exact result of an authorised query over bookings, sales, or stock.

Question	Potential answer
How many dinner covers did the city-centre restaurants serve yesterday?	345 covers, with an average spend of €31.40.
What was the no-show rate this week?	23 of 412 bookings, or 5.6%.
Which ingredients are below their minimum stock level?	7 ingredients. Salmon, rice, and sesame oil are the most urgent.

3. Classic search when the user wants to find, not generate

To find invoice INV-1842, a specific supplier reference, or a legal phrase, an inverted index with filters may be everything we need. I would not add a model when the correct result is a list of links.

Lexical search is also an excellent component inside RAG. “Classic” does not mean obsolete.

4. Fine-tuning for behaviour, not as a document store

Fine-tuning can teach a format, tone, taxonomy, or repetitive task. It is not my first choice for storing policies that change every week, returning citations, or enforcing document-level permissions. Updating that knowledge requires preparing data, training, and evaluating another model version, and it still does not give us a verifiable source.

We can combine both: fine-tuning to make the model respond with the correct structure and retrieval to provide current facts.

5. A workflow or agent for multi-step questions

Some questions do not need a more sophisticated index. They need a process. “Find last quarter’s complaints about unavailable dishes, check which suppliers delivered late during those weeks, and inspect current ingredient stock” requires searching, querying data, following references, and possibly revising the plan.

A workflow can alternate between search, SQL, APIs, and document reading. I would give it clear limits: allowed tools, a budget, a maximum number of steps, and mandatory evidence. Autonomy without observability is another way to hide errors.

A table for choosing without falling in love with the architecture

Dominant need	First option	Example	Main cost or risk
Find a few relevant passages	Hybrid RAG	“Which protocol applies to an allergy?”	Poor chunking and retrieval
Synthesise patterns across a document collection	GraphRAG or hierarchical summaries	“Which complaint causes recur?”	Indexing, extraction, and maintenance
Traverse relationships between entities	Explicit graph or GraphRAG	“Which dishes and restaurants depend on this supplier?”	Entity resolution and false relationships
Calculate over current data	SQL, API, or tool	“How many covers did we serve yesterday?”	Query validation and permissions
Analyse a few documents once	Long context	“Compare these 6 supplier proposals”	Cost, latency, and diluted attention
Find identifiers or exact text	Lexical search	“Find INV-1842”	No automatic synthesis
Change output format or behaviour	Fine-tuning	“Classify tickets with this taxonomy”	Dataset quality and model regressions
Combine several sources and steps	Workflow or agent	“Investigate complaints and supplier delays”	Latency, cost, and process control

These are not mutually exclusive boxes. A serious system may use a simple router: SQL for metrics, RAG for procedures, and GraphRAG only for global questions. What I try to avoid is sending every question through the most expensive path.

How I would build it from small to large

I would start with 30 or 50 real questions written by the people who will use the system. For each one, I would record the expected answer, the required sources, and the operation type: lookup, comparison, aggregation, relationship, or global synthesis.

Then I would build the smallest baseline that might work. For documentation, that would be hybrid search, good metadata, and citations. I would measure these separately:

• retrieval: does the required evidence appear in the results?;
• answer: is each claim supported by that evidence?;
• citations: do they point to the place that actually supports the claim?;
• operation: latency, cost, freshness, and permission enforcement.

If local retrieval fails, I would fix chunks, filters, query handling, and reranking. If global questions still require manually reading and joining dozens of results, I would test hierarchical summaries or GraphRAG on that subset. If the answer should be an exact number, I would remove that route from retrieval and connect it to structured data.

The evaluation needs negative cases too. The system should be able to say “I do not have enough evidence”, distinguish between 2 similarly named products, and refuse to use a document the user cannot open. A fluent answer with an irrelevant source is still a failure.

The rule I try to remember

RAG retrieves local evidence. GraphRAG organises relationships and helps synthesise global structure. SQL and APIs calculate over real state. Long context removes infrastructure when the problem is small. Fine-tuning changes how the model behaves; it does not keep a library of facts current.

I would not choose GraphRAG because RAG feels insufficiently advanced. I would choose it when I can point to important questions that require relationships or a global view and show that the baseline cannot answer them. Until then, a good retrieval system with metadata, hybrid search, reranking, citations, and a decent evaluation set is usually a much more useful foundation.

The right architecture is not the one containing the most AI. It is the one that takes the question to the right source with the fewest opportunities to be wrong.

Thanks for reading, Hack the Planet!