AI-300 Operationalizing Machine Learning and Generative AI Solutions Question and Answers

Managed Identity is Azure ' s solution for eliminating stored secrets and API keys from application configurations. When a Microsoft Foundry resource or its dependent compute has a system-assigned or user-assigned managed identity, it can authenticate to Azure services such as Storage, Key Vault, and Cognitive Services using Azure Entra ID tokens that are automatically issued and rotated by Azure. There are no keys to store, rotate, or accidentally expose in code or configuration files, directly satisfying the requirement to authenticate without stored credentials. Role-Based Access Control (RBAC) is the Azure governance mechanism for scoping permissions. By assigning specific built-in roles to the managed identity or to user and group principals at the correct scope — subscription, resource group, resource, or project level — the organization can enforce least-privilege access across different projects and shared resources.

Microsoft Learn Reference Topic: Configure managed identities and RBAC for Microsoft Azure AI Foundry resources

Correct sequence:

Collect retrieval logs.

Run RAG evaluators.

Adjust chunking strategy.

Re-index documents.

The process should begin by collecting retrieval logs . Retrieval diagnostics provide the evidence needed to understand which queries were issued, which document chunks were returned, and whether relevant content is being surfaced. Without retrieval telemetry, optimization becomes speculative rather than measurable.

Next, run RAG evaluators . Microsoft Foundry provides retrieval-oriented evaluators that assess how relevant retrieved context is to a query. The Retrieval evaluator measures contextual relevance without requiring ground truth, while the Document Retrieval evaluator can calculate metrics such as Fidelity and NDCG when labeled retrieval ground truth exists. These measurements help establish the retrieval-quality baseline before modifications are made.

After identifying retrieval deficiencies, adjust the chunking strategy . Microsoft specifically recommends reviewing chunk size and chunking methodology when retrieval returns irrelevant or incomplete passages. Chunks that are too small can lose context, while oversized chunks can introduce irrelevant material and reduce retrieval precision.

Finally, re-index the documents so the revised chunking configuration is reflected in the searchable corpus. Changing temperature or regenerating the prompt template primarily affects generation behavior rather than correcting the underlying retrieval pipeline.

Study Guide Reference: Implement generative AI quality assurance and observability — RAG evaluation, retrieval telemetry, retrieval-quality metrics, chunk optimization, indexing, and grounded-response assessment.

Different query types require fundamentally different retrieval algorithms. For queries that include exact policy identifiers, keyword or BM25 search is the correct choice because BM25 scores documents based on term frequency and inverse document frequency — an exact match on a specific policy identifier such as POL-2024-HR-042 scores very highly regardless of semantic context. This is the right approach when semantic similarity is low but exact term matching is critical. For natural-language questions where keywords may not be an exact match, semantic or vector search is the correct choice because vector embeddings capture meaning rather than exact tokens, finding relevant documents even when the user ' s vocabulary differs from the document ' s terminology. Azure AI Search supports both modes through its hybrid search capability, and the correct configuration maps each query type to its optimal retrieval algorithm.

Microsoft Learn Reference Topic: Configure hybrid search in Azure AI Search – BM25 keyword search vs. semantic vector search

Two distinct RAG problems require two distinct solutions. When retrieved results frequently include duplicated content from the same document, the cause is overlapping chunks receiving similarly high similarity scores. The solution is Maximum Marginal Relevance reranking, which diversifies the result set by penalizing results that are semantically similar to already-selected results, reducing redundancy. When retrieved chunks sometimes span unrelated policy sections, the cause is fixed-size character chunking that splits content without regard for semantic boundaries. The solution is semantic chunking — splitting on natural sentence or section boundaries — ensuring each chunk is semantically coherent and does not straddle unrelated content. Both problems must be addressed together: reranking alone solves duplication but not relevance; semantic chunking alone solves relevance but not duplication.

Microsoft Learn Reference Topic: Optimize RAG retrieval in Azure AI Search – Chunking strategies and MMR reranking

Microsoft ' s Azure AI Foundry documentation defines built-in roles scoped to the resource hierarchy. The Azure AI Administrator role grants permissions to manage the Azure AI Hub resource itself — including network settings, shared connections, quota, and governance configuration — which is appropriate for platform engineers who configure resource-level settings. The Azure AI Developer role grants permissions to create, update, and manage project-level assets such as deployments, prompt flows, fine-tuning jobs, and evaluations, without access to resource-level configuration. This precisely matches the developer requirement and enforces least privilege. Disabling Entra ID authentication (option B) violates security policy and removes the identity-based access control that makes RBAC possible. Sharing a single API key (option D) violates the least-privilege principle — all users would have identical, undifferentiated access with no audit trail.

Microsoft Learn Reference Topic: Manage access to Azure AI Foundry – Built-in roles: Azure AI Administrator and Azure AI Developer

Correct sequence:

Create a data asset.

Create a Foundry Search resource.

Create a vector index.

Create a flow.

First, create a data asset so that the locally stored text files are available within the project as a managed data source. The content must then be made searchable for Retrieval-Augmented Generation.

Next, create a Foundry Search resource to provide the search infrastructure that will store and serve the indexed document content. Microsoft documents Azure AI Search as a supported vector store for Foundry and Azure Machine Learning RAG workloads.

Third, create a vector index from the source documents. During vector-index creation, the content is processed into chunks and embeddings, which enables semantic similarity retrieval. Microsoft documents that vector indexes can be created from local files, folders, or registered data assets and then consumed by an Index Lookup operation.

Finally, create a flow and configure it to query the vector index and supply the retrieved context to the connected Azure OpenAI model. Microsoft specifically documents adding an existing vector index to a prompt flow through the Index Lookup tool.

Create a connection is the unused action in this question because the scenario already specifies a connected Azure OpenAI Service model.

Study Guide Reference: Design and implement a GenAIOps infrastructure — RAG architecture, project data, vector indexing, Azure AI Search, prompt flow, and grounding with enterprise content.

All three requirements point to Git as the underlying mechanism, but each uses a different Git capability. For reviewing changes before production: a branch-based workflow where prompt changes are made on feature branches and merged to the main branch only after pull request approval enforces the review gate. For restoring previous versions: Git ' s commit history and tag system provide a precise, immutable record of every prompt state, and a git revert or checkout to a specific commit SHA restores any prior version instantly. For governance parity with application code: by storing prompts in the same Git repository as application code, all the same CI/CD, branch protection, code review, and audit trail policies apply automatically. The alternatives such as Blob Storage or embedded configuration files lack native review workflows, branch protection, and full audit history.

Microsoft Learn Reference Topic: Prompt management and versioning with Git integration in Microsoft Foundry

The two requirements target fundamentally different stages of the AI application lifecycle. Continuous runtime monitoring addresses production behavior — an application that intermittently times out needs real-time visibility into latency, error rates, and request volumes for every call in production. Tracing backed by Application Insights or OpenTelemetry exporters is correct here because it captures granular, per-request telemetry continuously without requiring manual triggering — it is always-on by design. Release-time validation addresses pre-release quality gates — before new prompts or model updates go live, you need to verify they have not introduced regressions. An evaluation pipeline in Microsoft Foundry ' s prompt flow is correct here: it runs the updated application against a predefined test dataset, scores outputs on quality and safety metrics, and passes or fails the release based on thresholds. Evaluation pipelines run on demand triggered by CI/CD, not continuously.

Microsoft Learn Reference Topic: Microsoft Foundry observability – Tracing for runtime monitoring vs. Evaluation pipelines for release gates

Azure Machine Learning infrastructure can be described as Infrastructure as Code using ARM templates, Bicep, or Terraform. To make provisioning fully automated, two separate concerns must be addressed. First, the trigger: a CI/CD system like GitHub Actions or Azure DevOps Pipelines must detect changes to IaC template files in the repository, configured through a branch-push or pull-request trigger. Second, the execution: the CI/CD pipeline runs the provisioning commands such as az ml workspace create or bicep deploy non-interactively using a service principal or managed identity, eliminating the need for a human to run scripts manually. Together, these two mechanisms deliver the GitOps-style automation that modern MLOps practices require, ensuring every template change automatically results in a consistent, auditable infrastructure update.

Microsoft Learn Reference Topic: Automate Azure Machine Learning with GitHub Actions – CI/CD for ML infrastructure

Define how features are retrieved at inference: Feature retrieval specification

Ensure feature consistency between training and inference: Feature retrieval specification

Enable automated feature lookup in production: Feature retrieval specification

The feature retrieval specification is the key model-level contract for all three requirements. Microsoft defines it as a portable specification containing the exact list of features associated with a model, including the relevant feature store, feature set, and feature-set version . The same specification participates in both training and inference, making it the connective artifact across the model lifecycle.

During training, the feature retrieval specification is used to obtain the required feature values and generate training data. Microsoft requires the specification to be packaged with the model artifact when the model depends on feature-store features. At online inference time, the scoring script loads this packaged specification, resolves the feature list, and initializes online feature lookup before calling the feature store to retrieve current values.

This also prevents training-serving inconsistencies because the deployed model carries the feature dependencies and specific feature-set versions established during training, rather than relying on manually reconstructed production configuration. Microsoft explicitly states that packaging the retrieval specification with the model minimizes changes between training and inference workflows.

A feature set specification defines feature sources and transformations, a feature set asset provides managed versioning, and materialization precomputes feature values. None of those alone defines the model ' s complete retrieval contract at inference.

Study Guide Reference: Implement machine learning model lifecycle and operations — package a feature retrieval specification with the model artifact and operationalize feature-store-backed models.

Median stopping and Truncation selection both evaluate a run relative to the broader population of training runs rather than comparing it only against the single best run.

The Median Stopping Policy calculates running averages of the primary metric across all runs . At an evaluation interval, a run can be terminated when its best primary-metric performance is worse than the median of the running averages across the population. Microsoft explicitly defines this policy as being based on running averages of the primary metric of all runs.

The Truncation Selection Policy periodically ranks runs by their primary metric and terminates a configured percentage of the lowest-performing runs . For example, a 20-percent truncation policy terminates runs falling within the lowest 20 percent at the applicable evaluation interval. It therefore compares each candidate with the population of comparable runs rather than only the current leader.

Bandit is specifically unsuitable because it uses an allowable slack relative to the best-performing run , which directly violates the requirement. The Default behavior applies no early termination policy and allows runs to execute to completion.

Study Guide Reference: Implement machine learning model lifecycle and operations — automated hyperparameter tuning, early termination policies, Median Stopping Policy, Truncation Selection Policy, and efficient experiment execution.

Imagine a hospital analytics firm with data scientists who kick off dozens of training jobs throughout the week. During peak hours, five jobs compete for the same GPU cluster and fail or queue for hours. On quiet nights, that cluster sits completely idle, burning money. Managed compute targets with autoscaling solve both problems: the cluster scales out automatically when multiple jobs arrive simultaneously and scales back to zero when idle. Option A (single shared cluster) is exactly the resource-contention problem Fabrikam already has. Option B (fixed-size cluster) wastes money during off-peak hours. Option C (dedicated per-experiment clusters) eliminates contention but is prohibitively expensive for a cost-conscious healthcare firm. Autoscaling managed compute is the cloud-native answer to variable workload demand.

Microsoft Learn Reference Topic: Azure Machine Learning compute targets – Configure autoscale for compute clusters

MLflow is the industry-standard open-source platform for experiment tracking, and Azure Machine Learning has first-class native integration with it. When you use MLflow within an Azure ML job, parameters, metrics, and artifacts are automatically logged to the run history of the AML workspace, making every run reproducible and comparable. Option A (AML job output logs) only captures console output and lacks structured parameter and metric logging. Option C (Application Insights logs) is designed for application-level telemetry, not ML experiment metadata. Option D (Azure Monitor alerts) is a reactive notification tool, not a tracking system. MLflow ' s autologging capability means that for common frameworks such as scikit-learn, XGBoost, and PyTorch, parameters and metrics are captured without a single line of custom code, directly answering Fabrikam ' s requirement for consistent experiment tracking.

Microsoft Learn Reference Topic: Track ML experiments with MLflow – Azure Machine Learning MLflow integration

In Azure Machine Learning, a workspace is the top-level resource that acts as a centralized hub for every ML activity. It stores experiments, pipelines, datasets, models, environments, and compute targets all in one place. Before you can register assets or deploy endpoints, the workspace itself must exist and be shared across the team. Fabrikam ' s core challenge is inconsistent experiment tracking, ad-hoc versioning, and no standardized deployment path — all symptoms of the absence of a single, governed workspace. Option A (registering assets) is a subsequent action only possible after the workspace exists. Option C (online endpoint) is a deployment concern, not an asset-management foundation. Option D (Foundry project) is appropriate for generative AI workloads, not for the traditional ML pipeline standardization described here. The workspace is the prerequisite that makes every other governance action possible.

Microsoft Learn Reference Topic: Azure Machine Learning workspaces – Microsoft Learn: Manage Azure Machine Learning workspaces

The following custom role can do everything in the workspace except for the following actions:

It can ' t create or update a compute resource.

It can ' t delete a compute resource.

It can ' t add, delete, or alter role assignments.

It can ' t delete the workspace.

To create a custom role, first construct a role definition JSON file that specifies the permission and scope for the role. The following example defines a custom role named " Data Scientist Custom " scoped at a specific workspace level:

data_scientist_custom_role.json :

{

" Name " : " Data Scientist Custom " ,

" IsCustom " : true,

" Description " : " Can run experiment but can ' t create or delete compute. " ,

" Actions " : [ " * " ],

" NotActions " : [

" Microsoft.MachineLearningServices/workspaces/*/delete " ,

" Microsoft.MachineLearningServices/workspaces/write " ,

" Microsoft.MachineLearningServices/workspaces/computes/*/write " ,

" Microsoft.MachineLearningServices/workspaces/computes/*/delete " ,

" Microsoft.Authorization/*/write "

],

" AssignableScopes " : [

" /subscriptions/ < subscription_id > /resourceGroups/ < resource_group_name > /providers/Microsoft.MachineLearningServices/workspaces/ < workspace_name > "

]

}

Domain specialization: Supervised fine-tuning

Poor response accuracy of a RAG-based solution: Apply prompt engineering

For domain specialization , Supervised Fine-Tuning (SFT) is the correct first strategy. Microsoft describes SFT as the foundational fine-tuning technique for training a model from labeled input-output pairs , and specifically identifies domain specialization as one of its principal use cases. Microsoft also recommends starting with SFT for most customization projects because it supports task specialization, instruction following, style, and domain-specific behavior. Fabrikam already possesses evaluation data containing input-output pairs, which aligns directly with the SFT data model.

For poor RAG response accuracy , the first action is prompt engineering . The case explicitly requires advanced fine-tuning only when prompt engineering is insufficient. In a RAG system, prompt engineering determines how the model interprets retrieved context, constrains answers to grounding information, handles missing evidence, and formats responses. Microsoft notes that inadequate RAG prompting can produce false or incomplete answers even when retrieval returns appropriate content.

DPO is primarily appropriate for alignment using preferred versus non-preferred responses, while RFT targets complex reward-based reasoning optimization. Neither is the initial technique for domain specialization in this requirement.

Study Guide Reference: Optimize generative AI systems and model performance — prompt engineering, RAG optimization, supervised fine-tuning, preference optimization, and model customization strategy.

Managed online endpoints with traffic splitting are the best fit because Azure Machine Learning supports multiple deployments behind a single online endpoint and allows production traffic to be distributed between them. This enables a blue-green or progressive rollout strategy : deploy the new model version alongside the current version, validate it independently, send a small percentage of production traffic to it, monitor behavior, and then gradually increase traffic. If problems occur, traffic can be redirected immediately to the previous deployment, providing a practical rollback mechanism.

This approach also aligns with Fabrikam’s requirement to favor managed services and automation because the team has limited Azure DevOps experience. Managed online endpoints handle much of the serving, scaling, security, and monitoring infrastructure automatically and support both traffic splitting and mirrored traffic for safer validation.

Option B can also implement blue-green deployment, but AKS introduces substantially more infrastructure and operational management than Fabrikam requires. Option A would require custom deployment, routing, scaling, and rollback logic. Option D is intended for asynchronous batch inference rather than staged real-time production serving.

Study Guide Reference: Implement machine learning model lifecycle and operations — managed online endpoints, multiple deployments, traffic splitting, blue-green deployment, staged rollout, monitoring, and rollback.

The first action should be to evaluate the model output . Fabrikam already has evaluation data containing manually defined expected responses, and the technical requirements explicitly state that the organization must use its existing evaluation datasets based on real data with input-output pairs . Microsoft Foundry supports evaluating a model against an existing curated dataset to establish measurable baseline performance before optimization. Evaluation results can identify weaknesses in accuracy, relevance, groundedness, safety, or other quality dimensions and provide objective evidence for subsequent changes.

This sequencing is critical because Fabrikam also requires that advanced fine-tuning be applied only when prompt engineering is insufficient . Therefore, immediately fine-tuning the model would bypass the required baseline evaluation and optimization process. Fine-tuning should be driven by demonstrated performance gaps, not applied automatically.

Deploying directly to production is also inappropriate because leadership requires low operational risk and gradual rollout. Production traffic should not be used as the first evaluation mechanism when curated evaluation data already exists. Synthetic interaction data is useful when evaluation coverage is insufficient, but Microsoft specifically positions synthetic generation as an option when an adequate test dataset is unavailable. Fabrikam already has suitable evaluation data.

Study Guide Reference: Optimize generative AI systems and model performance — evaluate model performance, benchmark against curated datasets, iterate on prompts, and apply advanced fine-tuning when justified.

MLflow training runs produce model artifacts — the serialized model files, conda environment, and MLmodel specification — stored in the run ' s outputs folder. These artifacts are transient run outputs but are not yet a versioned, named model that can be deployed. To make the model a first-class, versioned, deployable artifact, you must explicitly register it. Locating artifacts from the run (action A) is necessary because you need the run ' s artifact URI, typically in the form runs:/run_id/model, to register from. Registering in the AML workspace (action B) creates an entry in the model registry with a name and auto-incremented version, making the model discoverable, governable, and deployable across environments. Tagging the experiment (option C) does not version the model. Exporting to local storage (option D) removes the model from Azure ML ' s managed infrastructure, losing lineage and governance.

Microsoft Learn Reference Topic: Register MLflow models in the Azure Machine Learning model registry

Git commits on the main branch provide the immutable, ordered history that auditors need: every change is timestamped, attributed to a specific developer, and reversible via git revert — this is the audit trail and rollback mechanism required by the first requirement. Git branches allow developers to create and test multiple prompt variants in complete isolation from the production prompt on the main branch, addressing the second requirement. A developer on an experiment branch can run full evaluations without touching the production prompt. When a variant is approved, it is merged via pull request. You cannot use branches alone for audit history because branches can be deleted, and you cannot use main-branch commits alone for parallel variant comparison without disrupting the history. Git ' s branch-and-merge model provides both capabilities simultaneously.

Microsoft Learn Reference Topic: Version control for AI prompts – Git branch strategies for prompt management in Microsoft Foundry