Introduction: AI Is Convenient. But "Careless" Accidents Happen Easily
Generative AI (such as ChatGPT) is active across a wide range, from writing, research, and summarization to coding support. On the other hand, risks such as hallucination (plausible lies), information leakage, and bias directly lead to real trouble depending on how you use it.
In this article, we break down difficult topics as much as possible and bring countermeasures useful in the field down to a "you can do it from today" level.
Risk 1: Hallucination (AI's "Plausible Misinformation")
What kind of problem occurs?
Hallucination is the phenomenon of AI confidently saying wrong things. It is especially likely in the following situations.
- Latest information (outside the model's training range, very recent news)
- Specialized domains (law, medicine, finance, security)
- When a source is required (papers, statistics, systems, terms)
- Proper nouns (people, companies, product names, clause numbers, model numbers)
Common "accident examples"
- Presenting nonexistent papers or URLs as "references"
- Asserting laws or systems while still in old content
- Misleading by arbitrarily and "plausibly" filling in internal rules
Countermeasure: Make AI a "draft craftsman," not a "responder"
Hallucination countermeasures are, roughly speaking, "do not make the AI's answer the final answer." Specifically, the following work.
1) Ask for the basis (source) as a set
Put "always include the basis" and "write uncertain points as uncertain" into the prompt.
Example: "Answer in the order conclusion then basis then how to verify. Always attach a source URL or primary information (the official document name). For unclear points, don't guess; write 'unknown.'"
2) Have it convert into a "form that is easy to verify"
Rather than having it assert in prose, having it turn things into a checklist of verification items increases safety.
- Decompose claims into bullets
- Attach a "needs verification" flag to each claim
- Have it propose where to verify (official sites, terms, primary sources)
3) Lean toward RAG (internal data search) or citation-based operations
If you use it for work, a mechanism that goes and fetches the correct information source is important rather than dumping everything on the model. For example, RAG (Retrieval-Augmented Generation) searches internal documents and a knowledge base and builds answers using those results as the basis.
Concrete options used include Azure AI Search, Amazon Kendra, Elasticsearch, OpenSearch, or search infrastructure integrated with Notion/Confluence/Google Drive.
4) Build human review in as a "process step"
What is strong in operations is to make review criteria explicit.
- Numbers, dates, and proper nouns are checked in principle
