LAB 334: Evaluate and improve the quality and safety of your AI applications

Important

This lab is designed for use in both instructor-led (75-mins, in-venue) and self-guided (self-paced, at home) sessions.

1. Instructor-led session attendees. Listen to the in-venue instructions and follow directions in the Skillable VM panel.
2. Self-guided session attendees. You will need to take a few extra steps for setup. Start with the Self-Guided: Quickstart below.

Workshop Description

You’ve built a custom AI application grounded in your enterprise data. How do you ensure response quality and safety? Join us as we explore AI-assisted evaluation workflows with built-in and custom evaluators on Azure AI. Learn what each metric represents, then understand how to analyze the scores for your specific application. Learn why observability is key, and how the generated telemetry can be used both locally, and in the cloud, to help you assess and debug your application performance.

• Level: Intermediate/Advanced
• Duration: 75 minutes

Learning Objectives

By the end of this workshop you should be able to:

1. Explain key evaluation metrics and workflows for GenAIOps
2. Create and run code-first evaluations with your Azure AI project
3. Understand and use built-in evaluators for quality and safety
4. Understand and use custom evaluators for your application needs
5. Understand and use simulators to create datasets for evaluations
6. Run manual evaluations and view results in the Azure AI Foundry portal

Pre-Requisites

To complete this lab you need:

1. A personal GitHub account → create one for free if needed
2. An Azure subscription → with quota for the required models
3. Familiarity with Python → and usage of Jupyter notebooks
4. Familiarity with Generative AI → basic tools and concepts

An Azure subscription pre-provisioned with the required models and application infrastructure will be made available to Microsoft Build Lab 334 attendees in-venue, for the duration of the session.

Self-Guided: Quickstart

To get started on this lab at home, follow these steps:

1. Fork this repo to your personal profile
2. Launch GitHub Codespaces - wait till ready
3. Open the VS Code terminal in codespaces
4. Run mkdocs serve > /dev/null 2>&1 &
5. Select the browser option in the pop-up dialog

You will see a preview of the instruction guide in a new tab. Click the "Workshop" menu item - and get started.

Questions & Feedback

We welcome feedback to help us improve the learning experience.

1. File an issue. We welcome feedback on ways to improve the workshop for future learners.
2. Join the Azure AI Foundry Discord. Meet Azure AI community members and share insights.
3. Visit the Azure AI Foundry Developer Forum. Get the latest updates on Azure AI Foundry.

Session Resources

The material in this repo was presented at Microsoft Build 2025 under the session title "Evaluate Reasoning Models For Your Generative AI Solutions". Please access these links for the session materials, recording, and Learn resources.

Resources	Links	Description
Build session page	https://build.microsoft.com/sessions/LAB334	Event session page with downloadable recording, slides, resources, and speaker bio
Microsoft Learn	Develop, replatform, and improve AI apps via advanced Azure AI services	Official Collection or Plan with skilling resources to learn at your own pace