Chapter 1: Foundations: Models, Deployment & Evaluation

Microsoft Foundry offers three core capabilities that you will use throughout this course:

Model Catalog. The model catalog is a searchable directory of language models from multiple providers, including OpenAI, Meta, Mistral, and Microsoft. Each model in the catalog has a model card — a summary page that describes the model’s capabilities, supported tasks, pricing, context window size, and regional availability. The model catalog is your starting point for any deployment decision, because it lets you compare models side by side before committing resources.

Deployment Management. Once you select a model, Microsoft Foundry provides the tools to deploy it as an API endpoint. You can choose between deployment options (which we cover in detail in Section 1.3), configure scaling settings, and manage endpoint lifecycle — all from the portal or programmatically through the Azure SDK. Each deployment produces an API endpoint — a URL that your application calls to send prompts and receive completions.

Monitoring and Responsible AI. After deployment, Microsoft Foundry provides dashboards for tracking usage, latency, error rates, and token consumption. It also integrates content safety tools, which you will explore in depth in Chapter 2.

When you first open the Microsoft Foundry portal, you will see a project-based workspace. Each project groups related deployments, datasets, and evaluation runs. Within a project, the left navigation panel gives you access to the model catalog, your active deployments, a playground for interactive testing, and evaluation tools.

The playground is particularly useful during development. It lets you send prompts to a deployed model, adjust parameters in real time, and observe how changes affect output — all without writing code. You will use the playground extensively in Lab 1 to verify that your project is set up correctly before you build an application on top of it in Lab 3.

Before you deploy a model or write a line of application code, you create a Foundry project. A project is the organizing container for your work: it holds your deployments, connects to the underlying Foundry Tools resource, stores datasets and evaluation runs, and applies access control. Creating a project is the first thing you do in Lab 1, and getting the setup right at this stage saves you from rework — and from security and cost surprises — later.

When you create a project, you make a few foundational decisions:

Every call your application makes to a Foundry endpoint must be authenticated. There are two broad approaches, and the choice between them is one of the most consequential security decisions you make at project setup.

The first approach is API keys. When you deploy a model, Foundry generates a secret key string. Your application sends that key with every request to prove it is authorized. API keys are simple to use and convenient for quick experiments, which is why most introductory tutorials — and the first code example in this chapter — use them. But keys are also a liability: a key is a long-lived secret that grants full access to the resource, it is easy to accidentally commit to source control or paste into a chat message, and rotating a leaked key means updating every application that uses it.

The second approach is keyless authentication using managed identity. Instead of a static secret, your application authenticates as a trusted Azure identity, and Azure issues a short-lived access token automatically. With managed identity, there is no key to store, leak, or rotate — the identity is granted permission to the Foundry resource through Azure role-based access control (RBAC), and credentials are managed by the platform. This is the approach Microsoft recommends for production workloads, and it is a recurring theme of the AI-103 "plan and manage" domain.

In Python, the azure-identity library provides DefaultAzureCredential, which automatically discovers the right credential for your environment — your signed-in developer account during local development, and a managed identity once the application runs in Azure. You will see how the key-based and keyless patterns differ in the code examples in Section 1.4. Keyless access is the recommended production approach; you should reach for API keys only for short-lived local experiments.

Part of planning a project is understanding its limits and its bill before you hit them in production:

These planning concerns are foundational here; you will return to quotas, cost management, and monitoring in greater depth as you build production-grade solutions later in the course.

A serverless deployment is the simplest way to get a model running. You deploy the model, receive an API endpoint, and pay only for the tokens you consume — there is no reserved capacity and no infrastructure to manage. Azure handles scaling automatically: if your application sends ten requests per minute or ten thousand, the platform allocates compute resources on demand.

The pricing model for serverless deployments is pay-per-token. You are billed based on the number of input tokens (the prompt you send) and output tokens (the completion the model generates). This makes serverless deployment ideal for workloads with variable or unpredictable traffic, because you never pay for idle capacity.

However, serverless deployments come with tradeoffs. Because resources are allocated on demand, you may experience higher and less predictable latency — the time between sending a request and receiving a response. During periods of high platform demand, serverless endpoints may also be subject to rate limiting or throttling, which can affect throughput. For a prototype or internal tool with modest traffic, these tradeoffs are usually acceptable. For a customer-facing chatbot processing thousands of requests per hour, they may not be.

A provisioned deployment reserves dedicated compute capacity for your model using throughput units (PTUs). Each throughput unit represents a fixed amount of processing capacity, measured in tokens per minute. When you create a provisioned deployment, you specify the number of throughput units you need, and Azure allocates that capacity exclusively for your endpoint.

The key advantage of provisioned deployment is predictable performance. Because capacity is reserved, your endpoint delivers consistent latency regardless of platform-wide demand. There is no throttling and no competition with other tenants for compute resources. This makes provisioned deployment the right choice for production workloads where response time guarantees matter — for example, a customer service chatbot that must respond within 200 milliseconds.

The tradeoff is cost structure. Provisioned deployments are billed hourly for the reserved throughput units, whether or not you use them. If your application has low or sporadic traffic, you are paying for capacity that sits idle. Provisioned deployment also requires you to estimate your throughput needs in advance, which adds a planning step that serverless deployment does not require.

The following table summarizes the key differences:

When choosing between deployment options, ask yourself four questions:

A chat completion request is built around a list of messages, each with a designated role. There are three roles:

System message. The system message sets the model’s persona, behavioral guidelines, and constraints. It is the first message in the conversation and is not shown to the end user. Think of the system message as instructions you give to the model before it begins responding to users. For example: "You are a helpful customer service agent for an e-commerce company. Be concise, polite, and only provide information about orders, returns, and shipping."

User message. The user message contains the end user’s input — the question, instruction, or prompt they type into the chat interface.

Assistant message. The assistant message contains a previous response from the model. Including prior assistant messages in the request enables multi-turn conversations, because the model can see the history of the exchange.

Together, these messages form a conversation history that gives the model context for generating its next response. In a multi-turn chat application, your code accumulates user and assistant messages over the course of the conversation and includes them in each subsequent API call.

Beyond the message list, the chat completion API accepts several parameters that control how the model generates its response. The four most important parameters are:

The system message is your most powerful tool for shaping model behavior without touching code. A well-crafted system message can:

When writing system messages, be specific and explicit. Vague instructions like "Be helpful" give the model little to work with. Concrete instructions like "Answer questions about our return policy. If the customer asks about something outside return policy, politely redirect them to the appropriate department" produce much more predictable behavior.

You will experiment with system messages in Lab 3 and observe how different instructions change the model’s responses to the same user prompts.

The interplay between parameters creates a wide range of possible output behaviors. Consider two extremes:

Configuration A: Factual and deterministic. Temperature 0.0, max_tokens 200, frequency_penalty 0.0. This configuration produces short, consistent, highly predictable responses. Run the same prompt ten times and you will get nearly identical output. This is ideal for FAQ bots, data extraction, and any task where consistency matters more than creativity.

Configuration B: Creative and exploratory. Temperature 0.9, max_tokens 800, frequency_penalty 0.5. This configuration produces longer, more varied, less repetitive responses. Run the same prompt ten times and you will get ten different outputs. This is better for brainstorming assistants, creative writing tools, and applications where novelty is a feature.

The following Python code demonstrates a complete chat completion API call with annotated parameters:

The example above uses an API key, which is fine for a quick local experiment. For production, the recommended approach is keyless authentication with managed identity, as introduced in Section 1.2. The change is small: you replace the api_key argument with a token provider built from DefaultAzureCredential, and you no longer store or transmit a secret.

The Microsoft Foundry model catalog organizes models into families and capability tiers. At a high level, you will encounter:

The right tier depends on your task complexity. If your chatbot answers simple shipping questions, a smaller model handles it well at low cost. If your application needs to analyze complex contracts and produce detailed summaries, you need a larger, more capable model.

Every model has a context window — the maximum number of tokens it can process in a single request, including both the input (system message, conversation history, user prompt) and the output (the model’s response). Context window sizes vary dramatically across models:

The context window constrains what your application can do. If you are building a multi-turn chatbot, the conversation history grows with each exchange. Once the accumulated messages exceed the context window, you must either truncate older messages (losing context) or summarize them (adding complexity and potential information loss). Applications that need to process long documents — legal contracts, research papers, codebases — require models with larger context windows.

Context window size also affects cost. Larger context windows allow more input tokens per request, and you pay for every token sent. A request that fills a 128,000-token context window costs significantly more than one that uses 4,000 tokens of a smaller window.

Every model selection decision creates a three-way tension between performance, cost, and user experience:

There is no universally correct answer. The right model depends on your specific requirements — and those requirements may differ across features within the same application. A common pattern is to use a smaller, cheaper model for routine tasks (intent classification, simple FAQ) and route complex queries to a larger model. This tiered approach optimizes cost without sacrificing quality where it matters most.

Model selection also introduces operational risks that responsible engineering practices can mitigate:

Model deprecation. Cloud providers periodically retire older model versions. If your application is tightly coupled to a specific model version, a deprecation notice forces an urgent migration. To mitigate this risk, abstract your model calls behind a configuration layer so you can switch models by changing a configuration value rather than rewriting code.

Versioning and reproducibility. Model behavior can change between versions. A prompt that produces excellent results with one model version may produce different results after an update. Pin your deployments to specific model versions and test thoroughly when upgrading.

Fallback strategies. If your primary model endpoint becomes unavailable — due to regional outages, quota exhaustion, or throttling — your application needs a plan. Common fallback strategies include deploying a secondary model in a different region, maintaining a serverless endpoint as a backup for a provisioned deployment, or implementing graceful degradation that informs users of temporary limitations rather than failing silently.

The Microsoft Foundry portal lets you compare candidate models directly. Starting from the model catalog, you can select two or more models and view their attributes side by side: provider, parameter count (where published), context window size, supported deployment types, regional availability, and published benchmark scores. This comparison view is the right first filter — it quickly eliminates models that are unavailable in your region, too small for your context needs, or priced outside your budget.

Benchmark scores published on model cards are useful but limited. A model that tops a general reasoning leaderboard may still underperform on your domain — summarizing insurance claims, answering questions about your product catalog, or extracting fields from your invoices. That is why catalog comparison is a starting point, not a final answer.

To measure how models perform on your task, you run an evaluation in Foundry. The pattern is consistent regardless of the metric:

Evaluation output in Foundry generally falls into three buckets, and the best model is rarely the winner in all three:

The goal is not to find the single "best" model in the abstract — it is to find the model that best balances quality, latency, and cost for your requirements. A high-volume FAQ bot might rationally choose a fast, cheap model that scores slightly lower on quality, while a contract-analysis tool might accept higher cost and latency in exchange for top-tier accuracy. When you run your evaluation in Lab 2, capture the numbers for all three dimensions so you can defend your recommendation with evidence rather than intuition.

In Lab 1, you will create a Microsoft Foundry project and set it up for secure development. As you work through the lab, pay attention to the following:

In Lab 2, you will compare candidate models and run an evaluation. This is where the ideas in Section 1.6 become concrete:

In Lab 3, you will build a chat application that calls the chat completion API. This is where parameter tuning becomes tangible:

Document your observations carefully. Your screenshots and notes from these labs provide the evidence you will need for the discussion post and weekly reflection.