Practical AI for Developers: RAG, Embeddings, and Production Considerations

Retrieval-Augmented Generation (RAG) has matured from an experimental technique into the industry standard for building domain‑aware AI. While large language models (LLMs) are incredibly capable, the knowledge they carry is frozen at the moment their training ends. RAG gives them a “library card” to your private, real‑time data by combining embedding search with prompt engineering.

In this post I’ll walk through a robust RAG architecture, the production caveats most tutorials ignore, and a practical checklist for making retrieval feel fast and reliable.

1. Core Architecture

At its core, a RAG system is just a data pipeline that transforms raw content into something an LLM can query contextually. The high‑level flow looks like this:

1. Ingest – extract text from source systems (PDFs, Notion, Slack, SQL, etc.).
2. Embed – turn each chunk of text into a high‑dimensional vector using an embedding model such as text-embedding-3-small or an open‑source alternative.
3. Store – persist vectors in a specialised vector database (Pinecone, Milvus, pgvector, etc.).
4. Query – when a user asks a question, embed the query and retrieve the K nearest neighbours.
5. Prompt – insert those passages into the LLM prompt as “context”.
6. LLM – the model generates an answer constrained by the injected context, which greatly reduces hallucinations compared to an unconstrained prompt.

// pseudo‑flow for a single query
queryVec = embed(query)
results   = vectorStore.search(queryVec, topK=5)
prompt    = buildPrompt(query, results)
answer    = llm.generate(prompt)

2. Production Concerns: Beyond "Hello World"

Most RAG tutorials stop once you can run an end‑to‑end example locally. In the wild, three issues turn prototypes into brittle systems.

A. Embedding Model Drift

Changing your embedding model without re‑indexing the database is a guaranteed way to break search. Vectors generated by one model are not comparable to another.

Fix: version your embeddings. Add metadata such as embed_version: "v2" to every vector and run a background job to re‑embed when you upgrade. Do not swap models in prod until the new index is built.

B. Cost & Latency Control

Ingress and search add real dollars and milliseconds.

• Batching: use batch APIs for ingestion to reduce network overhead.
• Incremental updates: hash source documents (e.g. MD5) and only re‑embed when the hash changes.

C. Multi‑tenant Security

Security is the biggest hurdle in enterprise RAG. Never trust the LLM to “ignore” unauthorized documents. Apply hard filters at the database level:

SELECT * FROM vectors
WHERE organization_id = 'org_123'  -- enforced before the LLM ever sees the data
ORDER BY vector <-> query_vector
LIMIT 5;

3. The Quality Checklist

If your system feels inaccurate or low‑effort, run through these optimizations:

RAG isn’t magic; it’s engineering. Treat the embedding layer as production data, pay attention to latency and security, and your “AI assistant” will reliably answer questions using up‑to‑date internal knowledge.