Detecting Hallucinations in LangGraph Self-RAG: How to Build a Hallucination Grader

Stopping AI from Making Things Up: Building a Hallucination Grader with LangGraph and Self-RAG

Have you ever asked an AI a question, and it just… made something up? This common problem is called “hallucination,” and it can make AI answers unreliable. Imagine asking for medical advice, and the AI gives you completely wrong information!

This is why self-RAG hallucination detection LangGraph is so important. We want our AI assistants to be truthful and base their answers on facts. In this guide, we’ll learn how to build a special tool called a hallucination grader to catch these made-up answers.

We will use something cool called Self-RAG, which helps AI get information from real sources. Then, we’ll combine it with LangGraph, a powerful tool for building AI systems that can make smart decisions. You will learn how to ensure your AI’s answers are always honest and helpful.

What are AI Hallucinations and Why Do They Happen?

Think of an AI like a very smart student who loves to talk. Sometimes, even the smartest student might guess or fill in gaps if they don’t have all the information. This is what an AI does when it “hallucinates.” It creates text that sounds real but isn’t based on any actual facts.

These made-up answers can be a big problem, especially when accuracy matters. For example, if you’re building an AI for customer service, you don’t want it giving false details about your products. That’s why spotting and fixing hallucinations is a super important job for AI builders.

AI models are trained on tons of data from the internet. Sometimes, they learn patterns that allow them to generate very convincing-sounding but incorrect information. They don’t “know” they are wrong because they don’t have a built-in truth-checker without our help.

How Self-RAG Helps Reduce Hallucinations

To make AIs more truthful, we often use a method called Retrieval-Augmented Generation, or RAG. This means the AI first looks up information in a trusted database before it writes an answer. It’s like asking a librarian for facts before writing an essay.

Self-RAG is an even smarter version of this process. Instead of just looking up information once, Self-RAG asks itself questions about the retrieved facts and about its own generated answer. It’s like having a little debate with itself to ensure everything is correct. This internal checking makes it much better at producing grounded generation.

With Self-RAG, the AI system doesn’t just retrieve documents; it also critically evaluates them and its own response. It can ask, “Does this document actually support my answer?” or “Is my answer fully explained by the facts I found?” This adds a layer of self-correction that regular RAG might miss. You can learn more about building RAG applications in general here: Build RAG Applications with LangChain Vector Store in 2026.

Why We Still Need Hallucination Detection Even with Self-RAG

Even with Self-RAG’s clever self-checking, there’s always a small chance an AI might still make a mistake. No system is perfect, and sometimes the retrieved information might be unclear or incomplete. This is where our special hallucination grader comes into play.

Think of it as a final faithfulness check before the AI’s answer is shown to you. This grader acts like a strict editor, reviewing the AI’s final answer against the original facts. Its job is to confirm that every part of the answer is supported by the retrieved information.

This extra step ensures that you can truly trust the AI’s output. It builds confidence in your AI system, knowing that you’ve put safeguards in place to catch any lingering errors. So, even with advanced techniques like Self-RAG, a dedicated detection system is a must-have for robust applications.

Introducing the Hallucination Grader

A hallucination grader is essentially a program that checks if an AI’s answer is true to its sources. It’s designed to compare the AI’s generated text with the information it was given. If the answer introduces new, unsupported facts, the grader flags it as a hallucination.

The main goal of this grader is to perform a faithfulness check. This means it evaluates how “faithful” the AI’s answer is to the retrieved documents. It’s not just about grammar or spelling; it’s about the factual accuracy and groundedness of the statements.

Imagine you have a document stating, “The cat is on the mat.” If the AI says, “The cat is on the rug,” that would be a hallucination if the document never mentioned a rug. The hallucination grader would identify this mismatch.

Building Your Hallucination Grader with LangGraph

LangGraph is an amazing tool for creating complex AI systems because it lets you build “graphs” of different AI steps. It’s like drawing a flowchart for your AI, showing exactly what it should do at each point. This makes it perfect for adding a hallucination grader to your Self-RAG system. You can learn more about multi-step AI agents with LangGraph here: LangGraph StateGraph: Building Multi-Step AI Agents.

The Core Idea: Grounded Generation

Our main goal for the hallucination grader is to ensure grounded generation. This means every piece of information in the AI’s final answer must be directly supported by the documents it used. No guessing, no making things up, just solid facts.

We achieve this by having an LLM judge review the answer and the sources. The LLM judge will act as our impartial truth-teller. It will scrutinize the relationship between the facts and the generated response.

Components of the Grader

To build our grader, we’ll need a few key pieces:

1. The Retriever: This part finds relevant documents from your knowledge base.
2. The Generator: This is the main AI that creates an answer based on the retrieved documents.
3. The LLM judge (Our Grader): This special AI’s job is to read the original documents and the generated answer. It then decides if the answer is faithful to the documents.
4. The Decider (using LangGraph conditional edge): This part uses the judge’s decision to figure out what to do next. Should we accept the answer, or try again?

Using LangGraph for Flow Control

LangGraph makes it easy to define these steps and the connections between them. We can set up a “graph” where one step leads to the next, like a chain. The most exciting part is using a LangGraph conditional edge. This is like a “fork in the road” for our AI system. Based on the grader’s output, we can tell the system to go one way (if the answer is good) or another (if it’s a hallucination).

This allows for dynamic and intelligent workflows. If the hallucination grader finds an issue, we don’t just stop. We can try to regenerate the answer, fetch different documents, or even ask the user for clarification. This adaptive behavior is what makes LangGraph so powerful for self-RAG hallucination detection LangGraph.

Step-by-Step Implementation with LangGraph

Let’s walk through how to set this up using LangGraph.

1. Setting up the Graph: First, we need to define our graph. This is like drawing the blueprint for our AI’s decision-making process. We’ll use StateGraph from LangGraph, which helps manage the information as it flows between different steps.

2. Defining Nodes: Each step in our AI’s journey is called a “node.” Here are the nodes we might define for self-RAG hallucination detection LangGraph:

• retrieve: This node will get relevant documents for our question.
• generate: This node will use the retrieved documents to create an initial answer.
• grade_hallucinations: This is our hallucination grader. It uses an LLM judge to check the answer against the retrieved facts.
• grade_answer_quality: (Optional but good) This node could check if the answer is helpful, relevant, and well-written, in addition to being factual.
• decide_action: This node looks at the grading results and decides what the graph should do next.

3. Defining Edges: Edges tell LangGraph how to move from one node to another. We’ll have regular edges and LangGraph conditional edge types.

• From retrieve to generate: Always happens after retrieving documents.
• From generate to grade_hallucinations: The generated answer immediately goes for a faithfulness check.
• From grade_hallucinations to decide_action: The grade determines the next step.

4. The Grading Logic (Faithfulness Check): This is the heart of our hallucination grader. Inside the grade_hallucinations node, we’ll use another LLM to perform the faithfulness check. This LLM judge will get both the original question, the retrieved documents, and the AI’s generated answer.

It will then be prompted to carefully compare the answer with the documents. The LLM judge will tell us if the answer is fully supported by the documents. If any part of the answer cannot be found or logically inferred from the documents, it’s a hallucination.

Practical Example: Building the Grader

Let’s imagine you’re building an AI for a company’s internal knowledge base about their products. You want to ensure the AI never invents features that don’t exist.

Setting up the Tools

First, we’ll need to set up our Large Language Model (LLM) and a retriever. We’ll use LangChain for the components and LangGraph for orchestrating them.

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_community.document_loaders import WebBaseLoader
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import OpenAIEmbeddings
from langchain_openai import ChatOpenAI
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.runnables import RunnablePassthrough

from langgraph.graph import StateGraph, END

import os

# Set up environment variables (replace with your actual API key)
os.environ["OPENAI_API_KEY"] = "your_openai_api_key_here"

# 1. Initialize our LLM
llm = ChatOpenAI(model="gpt-4o", temperature=0)

# 2. Set up a simple retriever (for demonstration, using a fake document)
# In a real scenario, you'd integrate with a full vector store like Weaviate
# and possibly hybrid search. See:
# [LangChain Weaviate Hybrid Search for Scalable RAG]({% post_url 2026-04-05-langchain-weaviate-hybrid-search-scalable-rag %})
# [Build RAG Applications with LangChain Vector Store in 2026]({% post_url 2026-02-05-build-rag-applications-langchain-vector-store-2026 %})

class FakeRetriever:
    def get_relevant_documents(self, query: str):
        if "product A features" in query.lower():
            return [
                {"page_content": "Product A has a battery life of 12 hours and is waterproof up to 1 meter. It comes in black or white.", "source": "product_manual.pdf"},
                {"page_content": "Product A's charging time is 2 hours. It is compatible with iOS and Android devices.", "source": "tech_specs.html"},
            ]
        elif "product B" in query.lower():
            return [
                {"page_content": "Product B features a 15-inch display and 16GB RAM. It's designed for creative professionals. Price: $1500.", "source": "product_page_B.html"},
            ]
        return [{"page_content": "No relevant information found.", "source": "none"}]

retriever = FakeRetriever()

# 3. Define the prompt for our main generator
generator_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant. Answer the user's question based ONLY on the provided context. If you cannot find the answer, state that you don't know."),
        ("human", "Context: {context}\nQuestion: {question}"),
    ]
)

# 4. Define the prompt for our Hallucination Grader (LLM Judge)
grader_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a strict judge. Your task is to perform a faithfulness check. Given the retrieved documents and a generated answer, determine if the answer is fully supported by the documents. Rate it as 'yes' if it is fully supported, 'no' if it contains information not found in the documents (a hallucination), or 'partially' if only some parts are supported. Provide a brief reason."),
        ("human", "Retrieved Documents: {documents}\nGenerated Answer: {answer}\n---"),
        ("human", "Is the answer faithful to the documents? Answer 'yes', 'no', or 'partially' and explain why."),
    ]
)

# Define chain for generation and grading
generate_chain = generator_prompt | llm | StrOutputParser()
grader_chain = grader_prompt | llm | StrOutputParser()

Building the State and Nodes

We need a StateGraph to manage the conversation flow. The state will hold all the information relevant to our self-RAG hallucination detection LangGraph process.

from typing import List, Dict, Any, Literal

# Define our graph state
class GraphState(Dict):
    question: str
    documents: List[str]
    answer: str
    hallucination_grade: Literal["yes", "no", "partially"]
    hallucination_reason: str
    num_retries: int = 0

# Node 1: Retrieve documents
def retrieve(state: GraphState):
    print("---RETRIEVE DOCUMENTS---")
    question = state["question"]
    # In a real app, 'retriever' would query a vector store
    docs_objects = retriever.get_relevant_documents(question)
    documents = [doc["page_content"] for doc in docs_objects]
    return {"documents": documents, "question": question}

# Node 2: Generate answer
def generate(state: GraphState):
    print("---GENERATE ANSWER---")
    question = state["question"]
    documents = state["documents"]
    context = "\n\n".join(documents)
    answer = generate_chain.invoke({"context": context, "question": question})
    return {"answer": answer, "question": question, "documents": documents}

# Node 3: Grade Hallucinations (LLM Judge)
def grade_hallucinations(state: GraphState):
    print("---GRADE HALLUCINATIONS (LLM JUDGE)---")
    documents = state["documents"]
    answer = state["answer"]

    grade_output = grader_chain.invoke({"documents": "\n\n".join(documents), "answer": answer})
    
    # Simple parsing: expect "yes", "no", or "partially" at the start
    grade_lower = grade_output.lower()
    if grade_lower.startswith("yes"):
        grade = "yes"
    elif grade_lower.startswith("no"):
        grade = "no"
    elif grade_lower.startswith("partially"):
        grade = "partially"
    else:
        grade = "no" # Default to 'no' if parsing fails, for safety

    reason_start = grade_output.find(":") + 1 if ":" in grade_output else 0
    reason = grade_output[reason_start:].strip()

    print(f"Hallucination Grade: {grade} - Reason: {reason}")
    return {"hallucination_grade": grade, "hallucination_reason": reason, "documents": documents, "answer": answer}

The LangGraph Conditional Edge: Deciding What to Do Next

This is where the magic of LangGraph conditional edge happens. Based on the hallucination_grade from our LLM judge, we can direct the flow.

# Node 4: Decide action based on grade
def decide_action(state: GraphState):
    print("---DECIDE ACTION---")
    hallucination_grade = state["hallucination_grade"]
    num_retries = state["num_retries"]

    if hallucination_grade == "yes":
        print("---Decision: Answer is faithful, END---")
        return "end_success"
    elif num_retries < 2: # Allow up to 2 retries
        print("---Decision: Hallucination detected, RETRY GENERATION---")
        # Increment retry count for the next iteration
        return "retry_generation"
    else:
        print("---Decision: Hallucination detected, max retries reached, END WITH FAILURE---")
        return "end_failure"

Assembling the Graph

Now, let’s put all these pieces together using LangGraph.

workflow = StateGraph(GraphState)

# Add nodes
workflow.add_node("retrieve", retrieve)
workflow.add_node("generate", generate)
workflow.add_node("grade_hallucinations", grade_hallucinations)
workflow.add_node("decide_action", decide_action)

# Set entry point
workflow.set_entry_point("retrieve")

# Add edges
workflow.add_edge("retrieve", "generate")
workflow.add_edge("generate", "grade_hallucinations")

# Add conditional edge from grade_hallucinations to decide_action
workflow.add_edge("grade_hallucinations", "decide_action")

# Define conditional transitions from decide_action
workflow.add_conditional_edges(
    "decide_action",
    lambda state: state["hallucination_grade"] == "yes", # Condition for success
    {"end_success": END} # If successful, end
)
workflow.add_conditional_edges(
    "decide_action",
    lambda state: state["hallucination_grade"] != "yes" and state["num_retries"] < 2, # Condition for retry
    {"retry_generation": "generate"} # If not faithful and retries left, go back to generate
)
workflow.add_conditional_edges(
    "decide_action",
    lambda state: state["hallucination_grade"] != "yes" and state["num_retries"] >= 2, # Condition for failure
    {"end_failure": END} # If not faithful and max retries, end with failure
)

# Compile the graph
app = workflow.compile()

Running the Self-RAG Hallucination Detection LangGraph

Let’s test our hallucination grader with a few examples.

Example 1: Faithful Answer

print("\n--- Running Example 1: Faithful Answer ---")
inputs = {"question": "What is the battery life and color options for Product A?", "num_retries": 0}
for state in app.stream(inputs):
    print(state)

# Expected output should show grade: 'yes' and then END.

Expected flow:

1. retrieve finds documents about Product A.
2. generate creates an answer based on these documents.
3. grade_hallucinations uses the LLM judge to confirm the answer matches the documents.
4. decide_action sees “yes” and sends it to END.

Example 2: Hallucination Detected (and Retried)

Let’s modify our generate node or prompt slightly to simulate a hallucination, or simply ask a question that might lead to one with limited context. For this example, let’s assume the LLM might incorrectly add a feature not in the documents.

Since our FakeRetriever is deterministic, we’ll simulate the LLM’s hallucination by asking a question that requires information not in the provided fake documents, but that an LLM might confidently guess.

print("\n--- Running Example 2: Hallucination Detected and Retried ---")
# This question asks for a detail not in the provided context for Product A,
# forcing the LLM to potentially hallucinate if it doesn't strictly follow
# "ONLY on the provided context." Our grader should catch this.
inputs_hallucination = {"question": "What is the warranty period for Product A and its main competitor?", "num_retries": 0}
for state in app.stream(inputs_hallucination):
    print(state)

# Expected output:
# - retrieve finds docs for Product A (no warranty info).
# - generate might guess or state it doesn't know (if good). If it guesses warranty,
#   grade_hallucinations will catch it.
# - grade_hallucinations should output 'no' or 'partially'.
# - decide_action will see 'no'/'partially' and trigger 'retry_generation'.
# - The graph goes back to 'generate' with num_retries=1.
# - (Hopefully) on retry, LLM is more careful or different path.
# - If still hallucinating after max retries, ends with failure.

In a real Self-RAG system, the generate node might have internal thought processes that lead to regeneration upon initial grading. Our simplified example focuses on the external hallucination grader.

Implementing a Faithfulness Check

The core of our hallucination grader is the faithfulness check. This check ensures that the AI’s answer is truly grounded generation. It means every statement in the answer must have direct evidence from the retrieved documents.

How to Define “Faithful”

An answer is “faithful” if all its claims can be verified by the provided context. If the AI says “X is true” and you can point to a sentence in the retrieved documents that confirms X, then it’s faithful. If the AI says “Y is true” but Y is nowhere in the documents, then it’s unfaithful (a hallucination).

It’s not about making exact copies. The AI can rephrase information or combine facts, as long as the underlying information is there. The LLM judge needs to be smart enough to understand meaning, not just exact word matches.

Using an LLM Judge for This

An LLM judge is essentially another powerful language model specifically tasked with evaluating the quality of text. For our faithfulness check, we give the LLM judge three things:

1. The original question.
2. The set of retrieved documents.
3. The answer generated by our main AI.

The LLM judge then acts as an expert critic. It compares the answer to the documents, sentence by sentence or claim by claim. It will tell us if the answer is factually consistent with the sources.

Example Prompt for the LLM Judge

The prompt for our LLM judge is crucial for its performance. Here’s a more detailed example of what you might use:

grader_detailed_prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are an AI judge specialized in performing faithfulness checks for RAG systems. Your goal is to critically evaluate if a generated answer is fully supported by the provided context documents. Follow these steps:\n"
                   "1. Read the user's original question.\n"
                   "2. Read through ALL provided 'Retrieved Documents' carefully.\n"
                   "3. Read the 'Generated Answer'.\n"
                   "4. For each distinct claim or piece of information in the 'Generated Answer', determine if it can be directly verified or logically inferred from the 'Retrieved Documents'.\n"
                   "5. Do NOT use any outside knowledge. Stick strictly to the provided documents.\n"
                   "6. Output your judgment in JSON format with two keys: 'grade' (string: 'yes', 'no', or 'partially') and 'reason' (string: a brief explanation detailing specific unsupported claims if 'no' or 'partially').\n"
                   "   - 'yes': Every claim in the generated answer is fully supported by the documents.\n"
                   "   - 'no': The generated answer contains significant claims that are not supported by the documents (hallucinations).\n"
                   "   - 'partially': Some claims are supported, but others are not, or the answer misses crucial supported information that was expected."),
        ("human", "Question: {question}\nRetrieved Documents: {documents}\nGenerated Answer: {answer}\n---"),
    ]
)

# You would then integrate this into your LangGraph node:
# grader_chain_detailed = grader_detailed_prompt | llm | JsonOutputParser()
# Inside grade_hallucinations node:
# grade_output = grader_chain_detailed.invoke({"question": question, "documents": "\n\n".join(documents), "answer": answer})
# This would require a custom output parser or a smarter way to handle JSON.
# For simplicity, we used StrOutputParser and basic string parsing earlier.
# For custom output parsing, see: [LangChain Custom Output Parser Tutorial]({% post_url 2026-03-31-langchain-custom-output-parser-tutorial %})

This detailed prompt guides the LLM judge to be very precise in its evaluation, making our faithfulness check more reliable.

Leveraging LangGraph for Dynamic Detection

One of the greatest strengths of LangGraph in self-RAG hallucination detection LangGraph is its ability to create dynamic workflows. This means your AI system can change its behavior on the fly, based on the results of the hallucination grader.

How LangGraph Conditional Edge Helps Reroute or Retry

As we saw in our example, the LangGraph conditional edge is crucial here. If our LLM judge determines that the generated answer is a hallucination (i.e., hallucination_grade is “no” or “partially”), we don’t just stop. The conditional edge allows us to define alternative paths.

Common actions when a hallucination is detected include:

• Retry Generation: Go back to the generate node, perhaps with a modified prompt, different retrieved documents, or specific instructions to be more cautious.
• Re-retrieve: If the initial documents were insufficient, the system could go back to the retrieve node with a refined query or by using different search parameters.
• Alert Human: For critical applications, the system might flag the answer for human review instead of attempting a retry.
• Return a “Cannot Answer”: If multiple retries fail, it’s better to admit “I don’t know” than to provide a false answer.

Example of Rerouting if a Hallucination is Detected

In our decide_action node, we built a simple rerouting mechanism:

• If hallucination_grade is “yes,” the process ENDs successfully.
• If hallucination_grade is “no” or “partially,” and num_retries is less than a limit, the graph transitions back to the generate node. This effectively makes the AI try to create a better, more faithful answer.

This retry mechanism is a powerful way to enhance the reliability of your self-RAG hallucination detection LangGraph system. It allows for automated self-correction, minimizing the chances of incorrect information reaching the user.

Testing and Refining Your Grader

Building a hallucination grader is one thing; making sure it works perfectly is another. Testing and refining are crucial steps to ensure your self-RAG hallucination detection LangGraph system is reliable.

How to Evaluate the Hallucination Grader

You need to evaluate both the grader itself and how it improves your overall Self-RAG system.

1. Grader Accuracy: Prepare a dataset of AI answers, some of which are known to contain hallucinations and some that are completely faithful. Manually label each answer as “faithful” or “hallucinated.” Then, run your hallucination grader on this dataset and compare its output to your manual labels. This will tell you how accurate your LLM judge is.
2. System Improvement: After integrating the grader with retries, measure how often your overall system produces faithful answers compared to before. Does it reduce the rate of hallucinations? Does it increase the number of “I don’t know” responses, which might be a good thing if it prevents false information?

Importance of Good Test Data

High-quality test data is your best friend here. Your test questions and documents should cover a wide range of topics and complexities. Include cases where:

• Documents provide clear answers.
• Documents are ambiguous or incomplete, forcing the AI to potentially guess.
• Questions are out of scope for the documents.

This variety will help you understand the strengths and weaknesses of your hallucination grader and refine your LLM judge’s prompt. You can even use human evaluators to create a “gold standard” for your test data, ensuring the labels are as accurate as possible.

Beyond Basic Detection

Once you have a functional self-RAG hallucination detection LangGraph system, you can think about more advanced features.

Measuring Grounded Generation Percentage

Instead of just a “yes” or “no,” you might want a percentage of how much of the answer is grounded generation. An LLM judge could go sentence by sentence and rate each one, giving you a detailed breakdown. This could lead to a “groundedness score” from 0-100%.

For example, the grader might say: “80% of the answer is supported. The claim about ‘product being green’ is not found in the documents.” This gives a much richer understanding of the answer’s quality.

More Advanced Grading Metrics

Beyond simple faithfulness, you could introduce other grading criteria:

• Completeness: Does the answer address all parts of the question based on the available documents?
• Relevance: Is all information in the answer relevant to the question?
• Conciseness: Is the answer brief and to the point without unnecessary detail?
• Bias Detection: Does the answer show any unwanted bias present in the source documents or introduced by the LLM?

Implementing these advanced metrics would involve expanding your LLM judge’s prompt and potentially adding more nodes to your LangGraph workflow. Each additional metric could have its own conditional edge, allowing for even more nuanced decision-making within your Self-RAG agent. You might even find custom output parsers useful for handling complex grading outputs: LangChain Custom Output Parser Tutorial.

Conclusion

Building reliable AI systems that you can trust is incredibly important. By combining the power of Self-RAG with the flexible orchestration of LangGraph, you can create robust self-RAG hallucination detection LangGraph workflows.

You’ve learned how to design a hallucination grader using an LLM judge to perform a crucial faithfulness check. This ensures that your AI’s answers are always based on grounded generation, preventing it from making things up. With LangGraph conditional edges, your system can dynamically react to detected hallucinations, leading to more accurate and dependable AI responses.

Remember, the goal is not just to generate answers, but to generate truthful and helpful answers. By implementing these techniques, you’re taking a big step towards building AI applications that users can truly rely on.