This guide is aimed at CIOs, R&D directors, AI leads and architects who need to choose an architecture for an AI project on a documentary corpus in a regulated sector. It does not claim that one is always better than the other. It provides the criteria that make the difference.

First: what are we talking about?

RAG (Retrieval-Augmented Generation): a system that retrieves relevant passages from a corpus of documents (via vector embeddings), then asks an LLM to formulate a response based on those passages.

Knowledge Graph: a system that models the entities of a domain (substances, components, bodies, regulations, decisions) and the relationships between them in graph form. Queries traverse this graph and return facts traced back to their source.

These are not two ways of doing the same thing. They are two different philosophies of what an information system should do.

What RAG does well

RAG is a mature architecture, quick to deploy, and powerful for certain use cases:

  • Documentary synthesis: "Summarise the findings of these 50 reports on this subject" → RAG excels
  • Open-ended questions on large corpora: "What does the company broadly say about X?" → correct
  • Reformulation and explanation: "Explain this paragraph in plain language" → very good
  • Monitoring and summarisation: "What's new in the last 20 publications?" → effective
  • Time-to-market: software deployment in a few weeks on an existing corpus

What RAG cannot do

The limitations of RAG are not bugs. They are structural characteristics of the architecture.

Precise traceability: impossible by design

An LLM merges information from multiple passages to generate a response. It cannot guarantee that each element of the response comes from a specific source document, a precise section, a dated version. In an ALCOA+ context (pharma), this is sufficient to disqualify the architecture: "Attributable" and "Original" are not satisfied.

Precision on similar terms: uncontrollable

Similar terms ("hepatotoxicity", "hepatic toxicity", "hepatic effects") will be grouped by vector similarity. This grouping is sometimes correct, sometimes not. On regulatory nomenclatures (MedDRA, REACH, ICD), this approximation generates errors that are difficult to detect.

Cross-session consistency: not guaranteed

Ask the same question twice to a RAG system: you may get different answers. On a regulatory corpus where "is this substance authorised?" must have an identical answer regardless of the session, this is incompatible.

Hallucination on specialist corpora: underestimated

On specialist technical corpora, the proportion of partially incorrect responses is significantly higher than on general corpora — because the model fills in what it cannot find with what "seems consistent" from its general training. Architectural targets aim for <5% error rates, but this is not guaranteed without structural controls.

What the Knowledge Graph does well

  • Precise multi-criteria queries: "All substances X with property Y, updated since date Z, in category W" → seconds, without risk of error
  • Exact traceability: every fact is timestamped, attributed to a source document + section + version
  • Guaranteed consistency: the same query returns the same result
  • Complex relationships: "Who decided what, why, and under what conditions, in 2017?" → the graph traverses relationships
  • Regulatory compliance: ALCOA+, ITAR, audit requirements → satisfied structurally
  • Cross-source querying: combining data from heterogeneous sources (DMS + CMMS + external regulatory database)

What the Knowledge Graph does not do

  • It does not generate natural language text — it produces structured results (an LLM can reformulate downstream, on a validated result)
  • It does not deploy in 2 weeks — modelling a regulatory domain requires 2 to 4 months for a first scope
  • It does not answer open-ended questions — "tell me everything interesting on this subject" → it responds to structured queries
  • It requires business modelling — someone must define what a "derogation" or a "compensating condition" is

Cases where the two are combined

There is no rule against combining them. Some architectures use the graph as the source of truth for precise facts, and the LLM for formatting or synthesising results. The graph responds "here are the 14 precedents matching your query, with their exact source". The LLM reformulates: "Here is a structured summary of these 14 precedents."

What does not work: giving the LLM responsibility for the regulatory response, while relying on the graph as a simple context source. The architecture must be clear about who is responsible for the answer.

The 5-question decision framework

Question 1 — What is the nature of tolerable errors?

"We can review and correct" → RAG acceptable
"An error could trigger a regulatory rejection / non-conformity" → Knowledge Graph

Question 2 — Is traceability of every response required?

No → RAG possible
Yes, by an external standard (ALCOA+, GxP, nuclear authority audit, medicines agency inspection) → Knowledge Graph mandatory

Question 3 — Are questions mostly open-ended or precise?

Open-ended ("tell me what you know about…") → RAG suitable
Precise and multi-criteria ("all X with property Y since date Z") → Knowledge Graph

Question 4 — Is your data confidential / subject to ITAR?

No → cloud RAG possible
Yes → on-premise mandatory. The graph is architecturally compatible; cloud RAG solutions are disqualified.

Question 5 — What is the expected value horizon?

Results in 2–4 weeks on a limited scope → RAG
Queryable infrastructure across 10–20 years of documentation → Knowledge Graph

Questions to ask your vendors

If you are evaluating an AI architecture for a regulatory corpus, ask your vendors these questions:

  • "If I ask the same question twice, do I get the same answer both times?"
  • "For each element of the response, can you cite the source document, the exact section, and the validation date?"
  • "Where is inference processed — in a third-party cloud or on-premise?"
  • "How does your system handle two substances with similar names?"

The answers reveal the real architecture behind the demonstration.