RAG — Retrieval Augmented Generation — is the engine behind every knowledge-grounded AI agent. At EPPC26, Rick Van Rousselt from Advantive walked through the technical foundations, knowledge source options in Copilot Studio, and hard-won lessons from building a real-world agent for a Dutch bank and insurance company serving 6,000 employees.

What RAG Is and Why We Need It

RAG solves three problems that make bare LLMs insufficient for enterprise use: knowledge cutoffs (LLMs know nothing after their training date), company data stays internal (procedures and policies are never on public websites), and hallucination (LLMs generate plausible-sounding text when they don’t know the answer). RAG retrieves relevant documents from your own sources and injects them as context so the LLM answers from grounded information.

Chunks and Embeddings: The Technical Foundation

Chunking — documents are cut into smaller pieces. Chunk size matters enormously: too large wastes tokens; too small loses context. Embeddings convert each chunk into a high-dimensional numerical vector capturing semantic meaning — what makes semantic search find relevant chunks rather than keyword matches. Rick demonstrated with a 3D embedding plot where conference-related sentences cluster together, far from food or sports content.

Knowledge Sources in Copilot Studio: What Works

  • File uploads — great for small document sets and demos. Rick uploaded AutoCAD building files and got accurate answers immediately.
  • SharePoint — works for simple structured content like HR policies. Struggles when sites are full of tables — which banks love and the default indexer handles poorly.
  • Azure AI Search — the workhorse for large-scale enterprise knowledge. Rick’s bank has 8,000+ SharePoint pages. Their solution: extract content, convert to Markdown first, then index into Azure AI Search. More work, but it handles scale and complex content reliably.
  • MCP servers — just starting to emerge. Rick was beginning an MCP integration for the same bank.
  • Public websites — Rick was direct: he hasn’t seen this work reliably.

A Cautionary Case Study: The Tax Agent That Failed

A customer wanted an agent to answer tax questions with 100% correct answers, every time. Rick tested six different knowledge source configurations with identical agent instructions. Result: the same question produced slightly different answers on different runs. Not wrong — but never exactly identical. The project did not succeed.

This is a fundamental characteristic of generative AI. The lesson: if your requirement is deterministic exact output that never varies, validate that generative models can meet it before you commit.

Testing and Evaluation in Production

  • Manual checks first — always. Domain experts catch obvious failures fast.
  • Golden answer sets — 50–100 expert-written Q&A pairs as ground truth for automated evaluation.
  • LLM-as-judge nightly scoring — a separate LLM scores all daily sessions 1–5. Consistency matters more than the absolute score: a sudden drop signals that something changed (document deleted, law updated).
  • Previous chat conversations — with compliance clearance, real user conversations (kept 2 weeks) used as test inputs.
  • Session reset after 10 questions — bank users typically need 2–3 answers. If someone has asked 10 questions, the growing context is making things worse. Auto-reset fixes this and controls costs.
  • Token logging — 8,000 log files analysed with AI revealed users leaving chats open all day, inflating context and cost significantly.

Takeaways

  • Use Azure AI Search for enterprise-scale knowledge — file uploads for demos
  • Convert SharePoint tables to Markdown before indexing
  • Validate that generative models can meet accuracy requirements before you promise them
  • Build a full evaluation pipeline: golden answers + LLM-as-judge + real chat history
  • Monitor consistency daily — a stable score means a stable system
  • Control context windows with session resets

Session: “Unleashing the Power of RAG in Copilot Studio” — EPPC26, Copenhagen, July 1, 2026
Presenter: Rick Van Rousselt (Advantive)