Chapter 2: Grounding with RAG & Responsible AI

A RAG pipeline consists of four stages: index, retrieve, ground, and generate.

This pipeline runs in two phases. The indexing phase happens offline, before users interact with the system — you process and index your documents once (and update the index as your data changes). The retrieve-ground-generate phases happen in real time for each user query.

The indexing stage begins with a critical design decision: how to split your documents into chunks. Chunking matters because the retrieval stage searches at the chunk level — if your chunks are too large, retrieved results may contain irrelevant information that wastes context window space; if your chunks are too small, they may lack the context needed for the model to generate a coherent answer.

There are three primary chunking strategies:

Fixed-size chunking splits documents into chunks of a predetermined token or character count — for example, every 500 tokens. This approach is simple to implement and produces predictable chunk sizes that make it easy to estimate context window usage. The downside is that fixed boundaries often cut through the middle of sentences, paragraphs, or logical sections, breaking semantic coherence. A chunk that starts mid-paragraph may lack the context needed to be useful during retrieval.

Semantic chunking splits documents at natural boundaries — paragraph breaks, section headings, topic shifts. This approach preserves the logical structure of the content, producing chunks that are self-contained and meaningful. The tradeoff is that chunk sizes vary, which makes context window budgeting less predictable. Some chunks may be very short (a single paragraph) while others may be very long (an entire section), and extremely long chunks can crowd out other retrieved results.

Sliding window chunking creates overlapping chunks by advancing a fixed-size window through the document in increments smaller than the window size. For example, a 500-token window with a 100-token stride produces chunks that overlap by 400 tokens. This overlap ensures that information near chunk boundaries appears in multiple chunks, reducing the risk that a relevant passage is split across two non-overlapping chunks and missed during retrieval. The cost is increased storage and indexing time, since you are creating more chunks per document.

In practice, many production systems use a hybrid approach: semantic chunking as the primary strategy with a maximum chunk size cap, combined with a small overlap between adjacent chunks to catch boundary cases. Azure AI Search supports configurable chunking through its built-in indexer skills.

Once documents are chunked, each chunk must be converted into a form that supports similarity search. This is where vector embeddings come in.

A vector embedding is a list of numbers (typically 1,536 or 3,072 dimensions for Azure OpenAI embedding models) that represents the semantic meaning of a piece of text. The key property of embeddings is that texts with similar meanings produce vectors that are close together in this high-dimensional space. The sentence "How do I return a library book?" and the sentence "What is the process for bringing back a borrowed item?" would produce embedding vectors that are near each other, even though they share few words.

Azure OpenAI provides embedding models such as text-embedding-ada-002 and the newer text-embedding-3-small and text-embedding-3-large. You call the embeddings API with a chunk of text, and it returns the embedding vector. During indexing, you embed every chunk and store the vectors in a search index. During retrieval, you embed the user’s query with the same model and search for the nearest chunk vectors.

The choice of embedding model matters. Larger embedding models capture more semantic nuance but cost more per call and produce larger vectors that require more storage. For most RAG applications, text-embedding-3-small provides a good balance of quality and efficiency.

When a user submits a query, the retrieval stage finds the most relevant chunks from the index. There are three approaches to retrieval:

Semantic search (also called vector search) converts the query into an embedding vector and finds chunks whose embedding vectors are nearest to it, typically using cosine similarity. Semantic search excels at matching meaning even when the query and the document use different words. A query about "checking out materials" will match chunks about "borrowing books" because the embeddings capture semantic equivalence.

Keyword search uses traditional information retrieval techniques like BM25 to find chunks that contain the same terms as the query. Keyword search is effective when exact terminology matters — for example, searching for a specific book title, author name, or ISBN. It does not understand synonyms or paraphrases, but it is fast and precise for exact matches.

Hybrid search combines both approaches. Azure AI Search supports hybrid search natively: it runs both a vector search and a keyword search on the same query, then merges and re-ranks the results. Hybrid search captures the strengths of both approaches — semantic understanding from vector search and precision from keyword search. For most production RAG systems, hybrid search is the recommended approach.

The number of chunks you retrieve (the top-k parameter) is another important tuning decision. Retrieving too few chunks risks missing relevant information; retrieving too many wastes context window space with less-relevant content and can actually confuse the model by providing contradictory or tangential information.

The grounding stage takes the retrieved chunks and incorporates them into the prompt sent to the language model. The most common approach is to concatenate the retrieved text into a dedicated section of the system message or user message, clearly delineated so the model understands it is reference material.

A grounding prompt might look like this:

Notice two important elements of this prompt. First, the system message instructs the model to answer only from the provided reference material. This instruction reduces the likelihood of hallucination because the model is directed to treat the retrieved chunks as its knowledge boundary. Second, the reference material is clearly separated from the instruction and the user’s question, making it easy for the model to distinguish between the grounding context and the task.

Return to the library assistant scenario. Without grounding, the model must rely entirely on its training data to answer questions about your library’s collection. It may know general facts about popular books, but it has no access to your specific catalog, holdings, or circulation data. Its responses will be a blend of accurate general knowledge and confident fabrication.

With grounding, the model receives the actual catalog entry, the real ISBN, the correct number of copies, and the precise call number. Its response shifts from "I think this book was published by…​" to "According to the library catalog, this book is published by Pearson, ISBN 978-1-292-42420-5." The response is not just more accurate — it is verifiable. You can trace every claim in the model’s response back to a specific chunk in the retrieved data.

You can evaluate the difference between grounded and ungrounded responses across three dimensions:

Accuracy. Grounded responses are measurably more accurate for domain-specific questions because the model is working from real data rather than statistical inference. In evaluation benchmarks, RAG-augmented models consistently outperform ungrounded models on factual recall tasks, particularly for information that is specific, recent, or niche.

Hallucination reduction. Grounding data gives the model an alternative to fabrication. When the system message instructs the model to answer only from provided sources, and the sources contain the relevant information, the model has no reason to invent details. Hallucination rates drop significantly — though they do not reach zero, because the model may still misinterpret retrieved chunks or fill gaps between them with generated content.

Source traceability. With RAG, every piece of information in the model’s response can be linked to a specific retrieved chunk. This traceability enables citation, auditing, and trust. Users can verify the model’s claims by checking the original source. This is particularly valuable in academic, legal, medical, and financial contexts where accuracy is critical and accountability matters. You will see how groundedness is measured as an evaluator later in this chapter (Section 2.4.8), where the groundedness metric formally measures how well a response is anchored in its retrieved context.

RAG is powerful, but it is not a silver bullet. Understanding its limitations helps you design systems that account for them.

Context window constraints. Every retrieved chunk consumes tokens from the model’s context window. If you retrieve five chunks of 500 tokens each, that is 2,500 tokens of context window space used before the model generates a single word of response. For models with smaller context windows, or for applications with long conversation histories, this budget can become tight. You must balance retrieval thoroughness against context window capacity.

Retrieval failures. RAG is only as good as its retrieval. If the search index does not contain relevant information, or if the query does not match relevant chunks well, the model receives unhelpful context. A poorly constructed index, a missing document, or a query that does not align with the vocabulary used in the indexed content can all cause retrieval failures. In these cases, the model may fall back to its training knowledge — and hallucinate — or correctly report that it does not have the information, depending on how well you crafted the system message.

Stale data. The search index reflects your data at the time of indexing. If your library acquires new books, updates catalog records, or removes items, the index must be refreshed to reflect these changes. A RAG system that queries a stale index will provide outdated information with the same confidence as current information. Establishing an index refresh cadence that matches your data’s rate of change is an operational requirement.

Chunk quality. If documents are poorly chunked — important information split across chunks, or chunks containing too much noise — retrieval quality suffers even when the relevant data is in the index.

Several techniques can mitigate the limitations described above:

Re-ranking. After the initial retrieval returns candidate chunks, a re-ranking model scores them for relevance to the specific query and reorders the results. Re-ranking improves precision by promoting the most relevant chunks and demoting marginally relevant ones. Azure AI Search supports semantic ranker, a built-in re-ranking capability that uses a cross-encoder model to evaluate query-chunk relevance.

Metadata filtering. Adding metadata to chunks (such as document type, date, department, or topic category) allows the retrieval stage to filter results before or during vector search. For example, if the user asks about a book published in 2022, metadata filtering can restrict the search to chunks from documents tagged with that year, improving both precision and performance.

Chunk overlap. As discussed in the chunking strategies section, overlapping chunks reduce the chance that important information is missed because it straddles a chunk boundary.

Query transformation. Rephrasing or expanding the user’s query before retrieval can improve search results. Techniques include generating multiple query variations, extracting key terms, or using the model itself to rewrite the query in a form that better matches the indexed content.

When you deploy an AI application to real users, you cannot control what those users will type into the chat interface. Some users will test boundaries, either out of curiosity or with malicious intent. Without content safety filters, your library assistant chatbot could be prompted to generate hate speech, describe violent scenarios, produce sexually explicit content, or provide self-harm instructions — none of which belong in a university application.

Content safety is also a regulatory and institutional concern. Universities have policies about acceptable use of technology, and many industries have compliance requirements that mandate content moderation. Beyond compliance, content safety protects your organization’s reputation and protects users from exposure to harmful material.

Azure AI Content Safety evaluates text across four primary harm categories:

Hate and fairness. Content that attacks or discriminates against individuals or groups based on protected attributes such as race, ethnicity, religion, gender, sexual orientation, disability, or age. This category also covers content that promotes stereotypes or dehumanizes groups.

Violence. Content that describes, glorifies, or incites physical violence against people, animals, or property. This includes graphic depictions of injury, threats of harm, and instructions for causing damage.

Sexual. Content that is sexually explicit, describes sexual acts, or is intended to arouse. This category ranges from mildly suggestive language to graphic pornographic content.

Self-harm. Content that promotes, instructs, or glorifies self-injury or suicide. This is a particularly sensitive category because of the real-world harm that can result from AI-generated self-harm content reaching vulnerable users.

Each category is evaluated independently, meaning a single piece of text can be flagged in multiple categories simultaneously.

For each harm category, Azure AI Content Safety assigns a severity level on a scale that typically spans four levels:

Safe (severity 0). Content that does not contain harmful material in this category.

Low (severity 2). Content that contains mild references to the harm category but is not graphic, actionable, or targeted. For example, a news article that mentions violence in a factual reporting context.

Medium (severity 4). Content that contains more explicit references but may have legitimate educational, artistic, or informational value. For example, a detailed discussion of historical atrocities in an academic context.

High (severity 6). Content that is graphic, targeted, actionable, or clearly harmful. For example, explicit instructions for causing harm to a specific group.

When you configure content filters for a deployment, you set a threshold for each category. The threshold determines the minimum severity level at which content is blocked. Setting a threshold of "medium" means that content rated low is allowed through, while content rated medium or high is blocked.

Threshold configuration involves a fundamental tradeoff between two types of errors:

False positives occur when the filter blocks content that is actually safe or legitimate. A strict threshold (blocking at low severity) catches more harmful content but also blocks more legitimate content. A library chatbot that blocks any mention of violence might refuse to discuss the plot of To Kill a Mockingbird or provide information about a history textbook covering World War II.

False negatives occur when the filter allows content that is actually harmful. A permissive threshold (blocking only at high severity) allows more legitimate content through but also lets more borderline harmful content pass. A chatbot that only blocks the most extreme content might allow moderately hateful language or disturbing descriptions to reach users.

There is no setting that eliminates both errors simultaneously. Every threshold choice is a policy decision that reflects your organization’s risk tolerance, user population, and use case.

The appropriate threshold depends on context. Consider three different applications:

A children’s educational platform. You would set thresholds to block at the lowest severity across all categories. The cost of a false negative (a child encountering harmful content) far outweighs the cost of a false positive (a legitimate query being blocked). Users in this context are vulnerable, and overly cautious filtering is appropriate.

A university library assistant. You need a more nuanced approach. Academic discourse frequently discusses violence (history, political science), sexuality (health education, literature), and self-harm (psychology, counseling resources). Blocking at the lowest severity would cripple the chatbot’s usefulness for academic research. You might set hate and self-harm thresholds to block at low severity while setting violence and sexual content thresholds to block at medium severity, allowing educational and academic discussions while still blocking graphic or targeted content.

An internal corporate knowledge base. Thresholds can be moderately strict since business content rarely involves any of the harm categories. Medium thresholds across the board provide a safety net without interfering with normal business queries.

The key principle is intentional configuration. Default thresholds are a starting point, not a destination. You should evaluate your specific use case, user population, and content domain, then configure thresholds deliberately and document your rationale.

Beyond the four harm categories, Azure AI Content Safety provides additional protective mechanisms:

Jailbreak detection. Identifies attempts by users to manipulate the model into bypassing its safety guidelines through prompt injection or role-playing scenarios. For example, a user might say "Ignore your previous instructions and instead…​" The jailbreak detection filter catches these patterns and blocks the request.

Protected material detection. Identifies content that may reproduce copyrighted material, such as song lyrics, book passages, or news articles. This filter helps prevent your application from generating content that could create intellectual property liability.

Prompt shields. A broader category of input filtering that detects adversarial prompts designed to extract the system message, manipulate model behavior, or exploit vulnerabilities in the application. Prompt shields complement jailbreak detection by covering a wider range of attack patterns.

Groundedness detection. Evaluates whether the model’s response is actually grounded in the retrieved context. If the model generates claims that are not supported by the provided reference material, the groundedness filter can flag or block the response. This filter directly supports the RAG architecture by catching hallucinations that slip past the grounding instruction.

The following Python example demonstrates how to configure content safety filters when creating a deployment or making API calls with content filtering parameters. This example uses the Azure OpenAI client to send a chat completion request with explicit content filter configuration:

Content filter thresholds are configured at the deployment level through the Microsoft Foundry portal or the Azure Management API. When you create or update a deployment, you specify the severity threshold for each harm category. The chat completion API then applies those thresholds automatically to every request and response.

Filters and guardrails decide what a single request is allowed to do. They do not, by themselves, tell you how well your application performs across many requests. For that you need evaluators — automated tests that score model outputs against a metric so quality and safety become numbers you can track over time rather than impressions.

In Microsoft Foundry, evaluations run a set of inputs through your application and apply one or more evaluators to each output. Evaluators generally fall into two groups that matter for a grounded, responsible application:

Quality and groundedness evaluators. These measure whether a response is supported by the retrieved context, relevant to the question, and coherent. The groundedness evaluator is the most important one for a RAG system: it compares each claim in the response against the retrieved chunks and scores how well the response is anchored in that source material. A low groundedness score is a direct signal that the model is hallucinating past its grounding data — the very problem RAG was meant to solve. Related evaluators include relevance (did retrieval surface the right context?) and coherence (is the answer well-formed?).

Safety evaluations. These probe whether the application can be made to produce harmful content — the same harm categories the content filters target (hate and fairness, violence, sexual, self-harm), plus risks like jailbreak susceptibility and ungrounded or fabricated claims. A safety evaluation typically runs a battery of adversarial and edge-case prompts and reports, per category, how often the application produced content at or above a severity threshold (for example, at or above Medium). Where a content filter blocks one request in the moment, a safety evaluation tells you the rate at which your configuration lets harmful content through across a representative test set.

The two groups are complementary. Quality evaluators answer "is the application accurate and grounded?"; safety evaluations answer "is the application hard to misuse?" Both produce metrics you can compare before and after a change — a new model version, a different chunking strategy, a stricter filter threshold — so you can tell whether the change actually improved the system or merely moved the problem.

Responsible AI also requires that decisions be auditable — that you can reconstruct, after the fact, what the system did and why. Two mechanisms make a grounded application auditable.

Trace logging records the steps of each request as it moves through the pipeline: the user’s prompt, the input filter result, the chunks retrieved from the index, the grounded prompt sent to the model, the generated response, and the output filter result. A trace turns a single chat turn into an inspectable record. When a user reports a bad answer, the trace tells you whether the failure was a retrieval miss (the right chunk was never fetched), a grounding gap (the chunk was fetched but the model ignored it), or a filtering decision (the response was blocked or rewritten). Foundry’s tracing captures these steps so you can review them in the portal rather than reconstructing them from scattered logs.

Provenance metadata records where information came from. Each retrieved chunk should carry metadata identifying its source document, location, and version, and that metadata should flow through to the response so claims can be cited and traced back to their origin. Provenance is what makes a grounded answer defensible: instead of "the model said the ISBN is…​", you can show "this claim came from catalog record X, indexed on date Y." Provenance metadata also powers metadata filtering during retrieval (section 2.3.4) and the source citations users see in the response.

Together, trace logging and provenance support the governance expectations of a production AI system: incident review, compliance reporting, and the ability to demonstrate that the application behaved as designed. In regulated or high-stakes settings, the ability to answer "why did the system say this, and what was it based on?" is not optional — it is a requirement.

Content safety filters operate at two points in the RAG pipeline: input filtering and output filtering.

Input filtering evaluates the user’s prompt before it enters the retrieve-ground-generate pipeline. If a user submits a harmful or adversarial prompt, the input filter blocks it before any retrieval occurs. This protects not only the model but also the retrieval system — there is no reason to search your index for content matching a harmful query.

Output filtering evaluates the model’s generated response before it is returned to the user. Even when the input is clean and the retrieved context is appropriate, the model might generate a response that crosses a safety threshold. Output filtering catches these cases and either blocks the response or returns an appropriate fallback message.

This dual-layer approach means that safety is enforced regardless of where harmful content originates. If a user sends a harmful prompt, input filtering catches it. If the model generates harmful content despite a clean prompt and clean context, output filtering catches it.

Content safety filters are one component of a broader layered safety approach. A well-designed production system combines three layers:

Content safety filters provide automated, configurable detection and blocking of harmful content across standardized categories. They operate consistently on every request without relying on the model’s compliance with instructions.

System message guardrails instruct the model on what it should and should not do. Instructions like "Do not provide medical advice," "Only answer questions related to the library," and "If you are unsure, say so rather than guessing" shape the model’s behavior from within the prompt. System message guardrails are flexible and domain-specific, but they are not enforced — the model may not always follow them perfectly.

Grounding constraints limit the model’s knowledge boundary to the retrieved context. When the system message tells the model to answer only from provided sources, and the provided sources are curated and appropriate, the model’s response is naturally constrained to safe, relevant content. Grounding reduces the model’s opportunity to generate harmful content because it is working from vetted material rather than its general training data.

No single layer is sufficient on its own. Content filters can be bypassed by novel attacks; system messages can be ignored by the model; grounding data might not cover every possible query. Together, these three layers create defense in depth — each layer catches failures that the others might miss.

These three layers act on each individual request. Responsible AI adds two layers that act across requests and over time: evaluation measures groundedness and safety against a representative test set so you know how the whole system performs, and auditing (trace logging plus provenance metadata) records what happened so any single decision can be reviewed and defended. Per-request guardrails keep a given response safe; evaluation and auditing keep the deployed system trustworthy and governable as it evolves.

In Lab 4, you will use Microsoft Foundry’s "use your own data" flow to ground a chat model in a data source, then compare its responses to the same model with no grounding. This is the RAG pipeline from section 2.2, stood up end to end. As you work through the lab, pay attention to the following:

In Lab 5, you will configure content safety filters at different severity thresholds and observe how the filters respond to various inputs. Key observations:

Part A asks you to design a RAG pipeline for a university library context and compare grounded versus ungrounded responses. To prepare:

Part B asks you to recommend and defend severity thresholds for a university library assistant. To prepare:

Do not try to memorize "right answers" for this assessment. The assessment evaluates your ability to reason through design decisions, justify your choices with evidence, and demonstrate understanding of the tradeoffs involved. Focus on building a clear, well-structured argument.