AI Hallucination Prevention Method – How to Block Incorrect Answers with Prompts
AI makes up things it doesn't know. It coldly analyzes how much can be prevented through prompting.
Introduction: AI Got It Wrong, But Only I Didn't Know
In early March, preparing for the Industrial Hygiene Supervisor exam, I asked an AI to solve past exam questions.
The results were shocking. The AI gave the wrong answer. That wasn't the problem. The explanation for the wrong answer was perfectly convincing. It explained why this option was correct, cited relevant laws and regulations, and derived the answer from certain principles—all seamlessly, confidently, and plausibly.
Suspicious, I directly provided the correct answer.
"No, based on my analysis, option ○ is correct. Option × is wrong for these reasons."
The AI insisted. It claimed the wrong answer was correct and that the answer I provided was actually incorrect.
Finally, I uploaded both the problem file and the answer key file simultaneously. I asked it to analyze them.
It was still wrong.
This three-step experience perfectly reveals the essence of LLM hallucination. In this piece, I completely deconstruct why this happens and how much you can prevent it with prompts.
1. What Exactly Is Hallucination?
Hallucination is the phenomenon where an AI generates information that isn't factual as if it were true. But this definition is too simple.
The real problem is that the AI doesn't know it got it wrong. Lying and hallucination are different. Lying is knowing the truth and concealing it. Hallucination, from the AI's perspective, is completely true. The model doesn't think it's wrong.
This is what makes hallucination particularly dangerous. False information doesn't look false.
Types of Hallucination
Hallucination broadly falls into three categories.
1. Factual Hallucination
The AI invents non-existent people, dates, events, or papers. It's when a quote starting with "According to Harvard research in 2019..." is purely invented. It can generate sources, authors, journal names, publication years, and DOIs perfectly.
2. Reasoning Hallucination
Each step looks logically sound, but the conclusion is wrong. The Industrial Hygiene Supervisor exam case falls into this category. The AI cited legal clauses, analyzed the options, and presented arguments—but started from an incorrect premise. The form of logic was there, but the content was wrong.
3. Context Conflict Hallucination
The hardest to understand. Even when the user provides correct information directly in the context, the model ignores it and insists on the wrong answer based on training data. This is why it failed even when I uploaded the answer key file.
2. Why Does Hallucination Occur: How LLMs Work
To prevent hallucination, you need to understand why it happens.
LLM is a "next token prediction machine"
An LLM doesn't understand text. It predicts the next most plausible word (token) given a context. This prediction is based on statistical patterns encoded in hundreds of billions of parameters.
When asked "What's the criteria for noise-induced hearing loss in the Industrial Hygiene Supervisor exam?", the model generates the most plausible text that follows this question. Here, "plausible" means not factual but what came after similar patterns in the training data.
If the training data lacks relevant regulations or contains older standards more frequently, the model confidently outputs the older standard. It has no way of knowing it's wrong.
Temperature and Certainty: Unrelated to Hallucination
There's a common misconception: lowering temperature reduces hallucination. It's partially true, but misses the core issue.
Temperature controls output diversity. Lowering it produces more decisive and consistent answers. However, if the model learned incorrect information with high probability, lowering temperature to 0 will just make it output that wrong information more confidently.
Why It Insists Even When You Correct It: Parametric Knowledge vs. Context
This is the most important part.
When processing text, LLMs use two sources of information:
- Parametric Knowledge: Information encoded in hundreds of billions of weights during training. What's "remembered" inside the model.
- Contextual Knowledge: Information provided by the user in the current conversation. What's in the prompt.
When these conflict, should the model always prioritize context? Theoretically, yes. But in reality, no.
Especially when the model has learned something strongly—information that repeatedly appeared in training data—parametric knowledge can dominate over contextual knowledge. When the AI insisted despite being told the correct answer, it was because of this conflict. The model's internal "memory" was stronger than the user's input.
Why It Failed Even With Two Files: Attention Mechanism Limits
The failure when uploading both the problem file and answer key simultaneously has a different cause.
Processing both files simultaneously dramatically increases context length. The LLM's attention mechanism cannot process all information equally in long contexts. Especially, cross-referencing between two documents—"the answer to this problem is X from that answer key"—is a far more complex task than it seems.
This is also known as the "Lost in the Middle" phenomenon—as context length increases, middle information is lost. This was confirmed experimentally in Liu et al. 2023. Models tend to handle the beginning and end of context well but miss information in the middle.
3. Practical Techniques to Reduce Hallucination With Prompts
Now, the practical part. You can't completely prevent hallucination with prompts. But techniques to meaningfully reduce it exist.
Technique 1: Force Explicit Uncertainty
The simplest yet effective technique. Instruct the model to explicitly express uncertainty.
Rules:
- Answer only with confident facts
- Explicitly state "I'm not certain" when unsure
- Answer "I don't know" if you lack information
- Mark if speculation is included as "This is speculation"
Effectiveness and limitations of this prompt:
Effectiveness: The model is induced to explicitly express uncertainty. Especially on general knowledge questions, "I'm not certain" responses increase.
Limitation: It's less effective for incorrect information the model has learned strongly. If it's wrong but confident, the model doesn't consider itself "uncertain." This is why instructing "say you don't know if you don't know" didn't help with the exam question. The model thinks it's right.
Technique 2: Scope Restriction
Clearly limit the areas the model can answer.
You must answer based only on the [provided documents] below.
If the document doesn't contain information, answer "This information is not in the provided document."
Never use knowledge outside the documents.
Effectiveness: Significantly reduces hallucination when the model operates within the scope of provided context. It's also the basic prompt pattern for RAG systems.
Limitation: The model doesn't perfectly follow this rule. Especially when document content conflicts with training data, it sometimes secretly mixes in internal knowledge. Completely blocking this is impossible with prompts alone.
Technique 3: Self-Verification Prompt
Have the model verify its own answer after generating it.
Step 1: Answer the question
Step 2: Review the generated answer
Step 3: Ask yourself of each claim: "Is this factual? Is this verifiable?"
Step 4: If there are uncertain parts, mark them and modify the answer if needed
Effectiveness: Often improves accuracy over simple CoT. When the model critically reviews its output, it sometimes catches obvious errors.
Limitation: Again, incorrect information the model believes strongly passes through self-verification. When you ask "Is what I just said correct?", the model is confident it is.
Technique 4: Requesting Sources—But There's a Trap
"Tell me the source too" is frequently used to prevent hallucination. However, this is a double-edged sword.
When you request sources, the model generates them. It doesn't verify whether those sources actually exist. It can plausibly generate paper titles, author names, journal names, publication years, and DOIs. This is called citation hallucination.
Bad approach:
"Explain ○○ and tell me the source"
→ Model may generate fake sources
Better approach:
"Explain ○○. If you don't have a reliable source,
don't create one—mark it as 'Source verification needed'"
The best approach is not to ask the model if sources matter, but to find reliable sources yourself. AI is not a search engine.
Technique 5: Separation of Roles
Have the model separate responder and verifier roles. Call it twice, or specify both roles in the prompt.
[First Response Role]
Answer the given question to the best of your ability.
[Verification Role]
Critically review the above response. Point out factual errors, logical leaps,
and uncertain claims. Suggest corrections if needed.
Using the same model for verification means the verifier has the same biases, so effectiveness is limited. But it's better than not doing it. In real production systems, different models are sometimes used as verifiers.
4. What Prompts Absolutely Cannot Prevent
Time to be honest.
Strongly Learned Errors
Incorrect information that the model has learned from training data tens of thousands of times is very difficult to correct with prompts. Especially when older standards, pre-revision regulations, and past common knowledge are learned far more extensively than new information.
In the Industrial Hygiene Supervisor exam case, related laws and standards are continuously revised. Content revised after the model's training data cutoff is unknown to the model. And since the old standards are learned in far more text, even when told the latest standards, the model tends to answer with the old ones.
Information After Knowledge Cutoff
Every LLM has a knowledge cutoff date for training data. Information after that date is not in the model. When asked about information it doesn't have, it doesn't always answer "I don't know." It pretends it does.
Highly Specialized Domains
Domains extremely specialized compared to general knowledge—latest regulations in specific fields, treatment protocols for rare diseases, standards for specialized industries—have insufficient training data. The less data, the higher the hallucination probability.
5. Then How Should You Use It: Practical Judgment Criteria
If you can't completely prevent hallucination, how should you use AI?
Use it for verifiable tasks.
When asking it to write code, you can run and verify it. When asking it to structure an essay, you can logically review it. When asking it to help translate, you can compare it with the original. Trusting AI output without verification for tasks where you can't verify results is dangerous.
Verify factual information with original sources.
Check the law the AI cited in the National Law Information Center. Find the research paper it mentioned directly. AI is a starting point, not an end point.
Don't use AI as a specialist replacement.
Exam preparation, medical judgment, legal interpretation, financial decisions. Trusting AI output without verification in these areas is dangerous. AI is a tool that assists specialists, not replaces them. Especially in areas where the cost of being wrong is high.
Choose models that acknowledge hallucination.
Latest models are being improved to more actively use expressions like "This isn't in my training data," "Verification is needed." But this is still imperfect. When choosing models, consulting hallucination benchmarks is an option.
6. Fundamental Solution: RAG
There are limits to reducing hallucination with prompt techniques. The real solution is to inject the necessary facts directly so the model doesn't need to fabricate unknown ones.
This is the core idea of RAG (Retrieval-Augmented Generation).
RAG retrieves relevant content from a trustworthy document database when a user question comes in, puts that content in the context, then instructs the model: "Answer based only on these documents." The model doesn't need to fabricate facts itself.
Returning to the Industrial Hygiene Supervisor exam example, an ideal structure looks like this:
1. User: "What's the criteria for noise-induced hearing loss workers' compensation?"
2. System: Search latest Industrial Accident Compensation Insurance Act + Ministry of Employment and Labor notices DB
3. Retrieved original text → injected into context
4. Model: "According to the provided documents..." (answers based on original text)
The model doesn't need to speculate based on training data; it looks at verified, latest documents and answers. It addresses the fundamental cause of hallucination: "absence of information."
Specific RAG implementation and prompt design are covered in detail in Episode 8.
Closing: Understand Hallucination Rather Than Fear It
Everyone has experienced AI confidently giving a wrong answer. After that experience, reactions split two ways.
One is abandoning or severely limiting AI use, saying "I can't trust AI." The other is "understanding why this happens and using it accordingly."
Hallucination is a structural characteristic of LLMs. It's not a bug but part of the design. It happens because a system that predicts the next token is optimized for "generating something plausible" rather than "staying silent when uncertain."
Understanding this helps you use AI better. Use it for verifiable tasks, find originals for fact-checking, force uncertainty expression with prompts, but know it's imperfect.
Knowing the tool is the first step to using it properly.
Episode 7 covers treating prompts like code. Variables, templates, reusable prompt architecture design—the point where prompt engineering evolves from simple writing to system design.
