The Real Challenge of RAG
A simple RAG architecture alone is not enough. Building a robust system always boils down to two main challenges: putting the right information into the vector database, and retrieving the right chunks out of it based on a user's request.
Putting clean data into the vector DB is completely in our hands. For example, when converting documents like PDFs into chunks, we have to make sure they don't contain unnecessary XML, messy code, or binary data. We can easily clean this up programmatically using regex or by simply using better parsing libraries.
Now, the real challenge is pulling the right chunks based on the query. While building a RAG architecture, we must always keep one thing in mind: user requests are often vague, incomplete, or confusing. This inevitably creates a problem when the system tries to pull relevant chunks.
To solve this, we use a pre-retrieval approach known as Query Transformation (or Query Translation). There are several powerful methods that can improve a user's raw query, allowing the system to perform a rich similarity search and get the perfect context for our final LLM.
1. Query Rewriting
This is a straightforward method for improving the user's input. The raw query goes to a smaller, faster LLM that rewrites the user's "lazy" question into a highly structured search query.
User asks: "What was the name of that thing Apple launched yesterday?"
The Problem: The database doesn't know what "that thing" is, and the word "yesterday" is relative.
The Rewrite: "What new hardware or software products did Apple announce on [insert current date]?"
Conclusion: By using this approach, the vector DB can easily find the relevant matching content because the new query contains multiple concrete, semantic keywords.
2. Multi-Query
Users sometimes ask multiple things in a single question, making it too specific or complex. Therefore, the RAG system will struggle to find a single relevant chunk that answers everything at once. To fix this, the user's query is given to an LLM that breaks it down into multiple sub-queries. The system then searches the vector DB for each sub-query and combines the results effectively.
User asks: "Compare the battery life of the iPhone 15 to the Samsung S24."
The Problem: The database likely has separate documents for Apple and Samsung; it probably doesn't have one single document comparing both.
The Sub-Queries:
"What is the battery life of the iPhone 15?"
"What is the battery life of the Samsung Galaxy S24?"
Conclusion: The database runs two separate searches, retrieves the specs for both phones, and feeds all of that context to the LLM so it can write a proper comparison.
3. Step-Back Prompting
Sometimes a user asks a question that is incredibly specific but misses the broader context. Without that broader context, the RAG system is unable to retrieve the foundational chunks needed to answer it. This is where Step-Back Prompting comes into the picture. The LLM is asked to generate a higher-level, more abstract version of the user's query.
User asks: "Why did my Tesla Model 3 battery drop from 80% to 20% when parked in -10°C weather overnight?"
The Problem: A vector search for that exact, highly specific sentence might yield zero results.
The Step-Back Query: "How does extreme cold weather affect lithium-ion EV battery drain?"
Conclusion: The system searches the database using both the specific user query and the broader step-back query. This ensures the LLM receives the foundational knowledge required to give a reliable, accurate response to the user.
4. HyDE (Hypothetical Document Embeddings)
This is one of the most interesting methods. To set a broader context for the user's raw query, a small LLM is used to generate a fake, hypothetical answer to the user's question. We then use that fake answer to perform the similarity search in the vector DB.
User asks: "How do I fix error code 404 in my React app?"
The Problem: The question is very short, but the documentation explaining the fix is likely a long, detailed paragraph.
The HyDE Process: The LLM generates a fake answer: "To fix error 404 in React, you need to check your react-router-dom configuration and ensure your path matches..." (The LLM might be hallucinating the exact fix, but it is using the correct technical vocabulary). We then vectorize this fake answer and search the database with it.
Conclusion: Because a fake answer looks semantically identical to a real answer in the vector space, it pulls the correct, factual documentation out of the database with incredible accuracy.
Conclusion
By implementing just one or two of these techniques (like a simple Query Rewrite and HyDE), you can take a RAG pipeline from giving mediocre answers to feeling incredibly intelligent and robust.