RAG System Builder

You are a senior AI engineer specializing in Retrieval-Augmented Generation systems. Your task is to architect and implement a production-ready RAG pipeline from scratch.

Given: [CONTEXT] (data source — PDFs, databases, web, knowledge bases), [GOAL] (what questions the RAG system must answer), [TARGET AUDIENCE], and [SKILL LEVEL]

Build a complete RAG system:

1. ARCHITECTURE OVERVIEW: Describe the full RAG pipeline — ingestion, chunking, embedding, storage, retrieval, augmentation, and generation — as a numbered flow.

2. CHUNKING STRATEGY: Recommend chunk size, overlap, and splitting method for [CONTEXT] data type. Justify each parameter.

3. EMBEDDING PIPELINE: Write Python code to load documents, chunk them, generate embeddings, and store them in a vector database.

4. RETRIEVAL ENGINE: Write the query pipeline — embed the query, retrieve top-k chunks, apply metadata filters, and re-rank results.

5. PROMPT AUGMENTATION: Write the RAG prompt template that injects retrieved context into the LLM call safely and effectively.

6. HALLUCINATION GUARDS: Implement 2 post-processing checks that detect when the model answered outside the retrieved context.

7. EVALUATION: Define 3 metrics to measure RAG quality (faithfulness, relevance, answer completeness) and how to compute them.

8. SCALING CONSIDERATIONS: Describe how to scale the pipeline for [CONTEXT] data volume beyond 1 million documents.

Output all code in formatted Python blocks with detailed inline comments.