What is RAG? A Practical Approach to Letting Your Own Knowledge Answer
RAG (Retrieval-Augmented Generation) is a mechanism that makes a generative AI (LLM) search through in-house documents and knowledge, and uses the results as the basis to generate answers. It is more attractive because it is faster, cheaper, and safer to create a company-specific AI than retraining the model.
In short, RAG is composed of two steps:
- Retrieval: Search for fragments of internal data related to the question
- Generation: Use the found fragments as evidence for the LLM to generate the answer
The key is that the areas where LLMs tend to "pretend to know" (policies, product specifications, contracts, FAQs, procedures, etc.) are where RAG works best. Conversely, it is not suitable for applications that require accurately answering information that does not exist in internal data.
The Big Picture: The Standard RAG Architecture
A typical RAG flow looks like this:
- Data collection (PDF / Confluence / Notion / SharePoint / Google Drive / DB, etc.)
- Preprocessing (OCR, remove unnecessary parts, normalization, metadata tagging)
- Chunking (split text into suitably sized chunks)
- Embeddings (vectorize chunks)
- Store in Vector DB (make searchable)
- Search (semantic search + filtering, re-ranking if needed)
- Prompt composition (feed the search results as evidence to the LLM)
- Answer generation (format citations, evidence, notices)
- Evaluation & Monitoring (accuracy, hallucination, leakage, cost, latency)
Tip 1 for Avoiding Failures: Define Use Cases by Question Type
RAG is not a universal solution, so defining the question type at the outset increases success rate. A common approach is to classify into three types as follows:
- Search-type: Find the relevant portion and summarize (e.g., "What are the rules for expense reimbursement?")
- Procedure-type: Break down internal procedures into steps (e.g., "What is the initial response during an outage?")
- Decision-support-type: Use policies or specs as evidence to branch by condition (e.g., "Is this case eligible for a return?")
Starting with the "Search-type" is a classic approach. It's easy to evaluate and helps gain internal trust.
Tip 2 for Avoiding Failures: Data Preparation Should Not Be Overdone for AI
It's common to think "perfectly cleanse the data first...," but for RAG it pays to start small with high-value data. Preferred priorities are as follows:
- FAQs and inquiry history (the questions themselves become search keys)
- Regulations and manuals (clear evidence and easy to evaluate)
- Product specifications and release notes (the more often updated, the more effective RAG)
- Minutes and chat logs (lots of noise, so defer)
If you have many PDFs, OCR quality is the bottleneck. If scan PDFs are included, first ensure that you can extract text.
