NCP-AIN NVIDIA-Certified Professional AI Networking Question and Answers

The "What Just Happened" (WJH) service on Cumulus switches provides real-time visibility into network problems by monitoring various layers and components, including layer 1, layer 2, layer 3, tunnel, buffer, and Access Control List (ACL) related issues. WJH streams detailed and contextual telemetry data, enabling administrators to diagnose and troubleshoot network problems effectively.​

Reference Extracts from NVIDIA Documentation:

"WJH can monitor layer 1, layer 2, layer 3, tunnel, buffer and ACL related issues."​

"The WJH service enables you to diagnose network problems by looking at dropped packets."​

In InfiniBand networks, P_Keys (Partition Keys) control communication boundaries. Each port can belong to one or more partitions with either full or limited membership.

From NVIDIA InfiniBand Documentation (Partitioning and P_Keys):

"A limited (or partial) membership permits a port to communicate only with other ports in the same partition that have full membership. It cannot communicate with other limited members, even if they are in the same P_Key partition."

This makes limited/partial membership ideal for multi-tenant security , where tenant ports can reach infrastructure ports (full members) but not other tenant ports (limited members).

Incorrect Options:

A & B are not valid InfiniBand P_Key types.

C (Full membership) allows unrestricted communication within the same partition.

The ib_write_bw tool is part of the Perftest package and is specifically designed to measure the bandwidth of RDMA write operations between two InfiniBand endpoints. It provides accurate assessments of RDMA throughput, which is crucial for verifying the performance of InfiniBand networks in high-performance computing and AI training environments.​

The NVIDIA Network Operator is designed to simplify and automate the deployment and management of networking components in Kubernetes clusters, particularly those utilizing NVIDIA Spectrum-X switches. It manages the installation and configuration of necessary drivers, plugins, and other networking resources to enable features like RDMA and GPUDirect RDMA, which are essential for high-performance AI workloads.​

By leveraging Kubernetes Custom Resource Definitions (CRDs) and the Operator Framework, the Network Operator ensures that networking components are consistently and correctly configured across the cluster, reducing manual intervention and potential configuration errors.​

To check the status and link layer of InfiniBand interfaces, the ibstat command is used. For example:​

ibstat -d mlx5_0

This command provides detailed information about the InfiniBand device, including its state (e.g., Active), physical state (e.g., LinkUp), and link layer (e.g., InfiniBand).​

The correct utility is ibnetdiscover .

From the official NVIDIA InfiniBand Utilities Guide:

"ibnetdiscover scans the fabric and returns a topology of all switches and end nodes, including their GUIDs, LIDs, port numbers, and link speeds."

It generates a fabric map with node-to-port relationships and shows:

GUIDs

LIDs (Local IDs)

Link speeds and widths

Switch-to-host connections

This is essential for network topology validation and mapping physical port additions.

Incorrect Options:

ib_check_routes – for routing table diagnostics.

ibhosts – shows host information but not switch-level port mapping.

ibswitches – shows switch info, but lacks port-level LID/link speed mapping.

The CloudAI Benchmark is designed to evaluate and establish performance baselines in AI-optimized networking environments like NVIDIA Spectrum-X. It assesses various performance metrics, including throughput and latency, ensuring that the network meets the demands of AI workloads. This benchmarking is essential for validating the benefits of Spectrum-X and for ongoing performance monitoring.​

The ibstat command is utilized to display the operational status of InfiniBand Host Channel Adapters (HCAs). It provides detailed information, including the state (e.g., Active, Down), physical state (e.g., LinkUp, Polling), and link layer (e.g., InfiniBand, Ethernet) of each port on the HCA. This information is crucial for diagnosing connectivity issues and ensuring that the InfiniBand interfaces are functioning correctly.​

Reference Extracts from NVIDIA Documentation:

"The ibstat command displays the status of the host channel adapters (HCAs) in your InfiniBand fabric. The status includes the HCAs' state, physical state, and link layer."​

"For proper operation, you are looking for ‘State: Active‘ and ‘Physical State: LinkUp’."​

NVIDIA Air is a cloud-based network simulation tool designed to create digital twins of data center infrastructure, including Spectrum-X networks. It allows users to model switches, SuperNICs, and storage components, enabling the simulation, validation, and automation of network configurations before physical deployment. This facilitates Day 0, 1, and 2 operations, ensuring that network designs are tested and optimized for AI workloads.​

Reference Extracts from NVIDIA Documentation:

"NVIDIA Air enables cloud-scale efficiency by creating identical replicas of real-world data center infrastructure deployments."​

"NVIDIA Air allows users to model data center deployments with full software functionality, creating a digital twin. Transform and accelerate time to AI by simulating, validating, and automating changes and updates."​

"NVIDIA Air supports simulation of NVIDIA Spectrum Ethernet (Cumulus Linux and SONiC) switches and NVIDIA BlueField DPUs and SuperNICs as well as the NetQ network operations toolset."​

The ibdiagnet utility is a fundamental tool for InfiniBand fabric discovery, error detection, and diagnostics. It provides comprehensive reports on the fabric's health, including error reporting, switch and Host Channel Adapter (HCA) configuration dumps, various counters reported by the switches and HCAs, and parameters of devices such as switch fans, power supply units, cables, and PCI lanes. Additionally, ibdiagnet performs validation for Unicast Routing, Adaptive Routing, and Multicast Routing to ensure correctness and a credit-loop-free routing environment.​

Reference Extracts from NVIDIA Documentation:

"The ibdiagnet utility is one of the basic tools for InfiniBand fabric discovery, error detection and diagnostic. The output files of the ibdiagnet include error reporting, switch and HCA configuration dumps, various counters reported by the switches and the HCAs."​

"ibdiagnet also performs Unicast Routing, Adaptive Routing and Multicast Routing validation for correctness and credit-loop free routing."​

In Cumulus Linux, microbursts are short-lived, high-volume traffic bursts that often go undetected by coarse-grained monitoring like SNMP.

The two tools specifically used for this purpose are:

What Just Happened (WJH)

"WJH provides real-time packet drop visibility and classifies drops by reason (e.g., congestion, ACLs, etc.), enabling microburst detection."

ASIC monitoring at millisecond granularity

"Deep telemetry is enabled via the switch ASIC, which provides sub-second counters that capture microburst patterns otherwise missed by SNMP."

Incorrect Options:

A and C provide low-frequency sampling, insufficient for microbursts which last milliseconds.

Comprehensive and Detailed 150 to 250 words of Explanation From NVIDIA AI Networking Topics:

The correct answer is A. Cyber-AI platform . NVIDIA’s official UFM Cyber-AI documentation states that the UFM Cyber-AI platform determines a data center’s unique “vital signs” and uses them to identify performance degradation, component failures, and abnormal usage patterns . This directly matches the requirement for proactive detection of issues in a large InfiniBand fabric. NVIDIA also documents that Cyber-AI provides Link Anomaly detection and Link Failure Prediction , where machine learning models analyze monitoring information and predict future link failures 1 to 24 hours in advance . In addition, NVIDIA’s Cyber-AI materials describe integration with AI cybersecurity capabilities to help detect and mitigate security threats , which makes Cyber-AI the only option in this list that combines predictive analytics with security-oriented intelligence. By contrast, UFM Telemetry focuses on collecting and presenting real-time network data such as bandwidth, congestion, latency, and errors, while UFM Enterprise is the broader management platform and Host Agent is not the AI analytics engine for predictive failure and threat detection. Therefore, for AI-driven analytics, predictive maintenance, anomaly detection, and security-focused analysis in InfiniBand environments, Cyber-AI platform is the verified NVIDIA answer.

Configuring NVUE to ignore certain underlying Linux files allows administrators to manage specific settings manually or through automation tools like Ansible without NVUE overwriting these configurations. This approach enables the persistent manipulation of settings using both NVUE and flat-file methods, providing flexibility in network management.

Before upgrading the DOCA SDK on a BlueField DPU , it is mandatory to uninstall the existing OFED drivers to prevent compatibility conflicts.

From the NVIDIA DOCA Installation Guide :

"Before upgrading DOCA or BlueField-related software, you must remove existing OFED packages using: /usr/sbin/ofed_uninstall.sh -force."

This ensures:

Clean driver state

No residual kernel modules or userspace libraries

Proper registration of new DOCA/OFED versions

Incorrect Options:

A and C may not resolve conflicts.

D installs but doesn’t remove conflicting packages.

NVIDIA AIR (AI-Ready Infrastructure) is a cloud-based simulation platform designed to model and validate data center network deployments, including Spectrum-X Ethernet networks, using realistic topologies and configurations. When creating a custom topology in NVIDIA AIR, users can define network links between devices (e.g., spine and leaf switches) using a DOT file format, which is based on the Graphviz graph visualization software. The question asks for the correct syntax to define a link between port 1 on a spine switch (spine-01) and port 41 on a leaf switch (gpu-leaf-01) in a NVIDIA AIR simulation.

According to NVIDIA’s official NVIDIA AIR documentation, the DOT file format is used to specify network topologies, including nodes (devices) and links (connections between ports). The syntax for defining a link in a DOT file uses a double dash (--) to indicate a connection between two ports, with each port specified in the format " < node > ":" < port > ". For Spectrum-X networks, which typically use Cumulus Linux or SONiC on NVIDIA Spectrum switches, ports are commonly labeled as swpX (switch port X) rather than ethX (Ethernet interface), especially for switch-to-switch connections in a leaf-spine topology. The correct syntax for the link between port 1 on spine-01 and port 41 on gpu-leaf-01 is:

"spine-01":"swp01" -- "gpu-leaf-01":"swp41"

This syntax uses swp01 and swp41 to denote switch ports, consistent with Cumulus Linux conventions, and the double dash (--) to indicate the link, as required by the DOT file format.

Exact Extract from NVIDIA Documentation:

“You can create custom topologies in Air using a DOT file, which is the file type used with the open-source graph visualization software, Graphviz. DOT files define nodes, attributes, and connections for generating a topology for a network. The following is an example of a link definition in a DOT file:

"leaf01":"swp31" -- "spine01":"swp1"

This specifies a connection between port swp31 on leaf01 and port swp1 on spine01. Port names typically follow the switch port naming convention (e.g., swpX) for Cumulus Linux-based switches.”

— NVIDIA Air Custom Topology Guide

This extract confirms that option A is the correct answer, as it uses the proper DOT file syntax with swp01 and swp41 for port names and the double dash (--) for the link, aligning with NVIDIA AIR’s topology definition process for Spectrum-X simulations.

Analysis of Other Options:

B. "spine-01":"swp1" to "gpu-leaf-01":"swp41" : This option uses the correct port naming convention (swp1 and swp41) but incorrectly uses the word to as the connector instead of the double dash (--). The DOT file format requires -- to define links, making this syntax invalid for NVIDIA AIR.

C. "spine-01":"eth1" to "gpu-leaf-01":"eth41" : This option uses ethX port names, which are typically used for host interfaces (e.g., servers) rather than switch ports in Cumulus Linux or SONiC environments. Switch ports in Spectrum-X topologies are labeled swpX. Additionally, the use of to instead of -- is incorrect for DOT file syntax, making this option invalid.

D. "spine-01":"eth1" - "gpu-leaf-01":"eth41" : This option uses a single dash (-) instead of the required double dash (--) and incorrectly uses ethX port names instead of swpX. The ethX naming is not standard for switch ports in Spectrum-X, and the single dash is not valid DOT file syntax, making this option incorrect.

Why "spine-01":"swp01" -- "gpu-leaf-01":"swp41" is the Correct Answer:

Option A correctly adheres to the DOT file syntax used in NVIDIA AIR for defining network links:

Node and Port Naming : The nodes spine-01 and gpu-leaf-01 are specified with their respective ports swp01 and swp41, following the swpX convention for switch ports in Cumulus Linux-based Spectrum-X switches.

Link Syntax : The double dash (--) is the standard connector in DOT files to indicate a link between two ports, as required by Graphviz and NVIDIA AIR.

Spectrum-X Context : In a Spectrum-X leaf-spine topology, connections between spine and leaf switches (e.g., Spectrum-4 switches) use switch ports labeled swpX, making swp01 and swp41 appropriate for this simulation.

This syntax ensures that the NVIDIA AIR simulation accurately models the physical connection between spine-01 port 1 and gpu-leaf-01 port 41, enabling validation of the Spectrum-X network topology. The DOT file can be uploaded to NVIDIA AIR to generate the topology, as described in the documentation.

The correct answer is PXN (Peer eXchange Network) .

From the NVIDIA NCCL Documentation :

"PXN enables communication between GPUs connected via NVLink and NICs by treating the GPUs as a distributed switch. This architecture improves bandwidth utilization by enabling any GPU to communicate with the NIC via the shortest path available, even if it's not directly connected to the NIC."

This enhances GPU-to-NIC and NIC-to-GPU transfers, leveraging the NVLink topology. It significantly boosts performance in multi-GPU setups where not every GPU is directly connected to the NIC.

Other options:

Adaptive Routing is a fabric-level feature for dynamic path rerouting.

Graph Search Optimization is used internally for topology modeling in NCCL.

SHARP v2 is a switch-based collective acceleration method, unrelated to PXN.

RoCE Adaptive Routing is a key feature of NVIDIA Spectrum-X that ensures high performance and resiliency in the network, even in the event of a link failure. This technology dynamically reroutes traffic to the least congested and operational paths, effectively mitigating the impact of link failures. By continuously evaluating the network's egress queue loads and receiving status notifications from neighboring switches, Spectrum-X can adaptively select optimal paths for data transmission. This ensures that the network maintains high throughput and low latency, crucial for AI workloads, even when certain links are down.​

Reference Extracts from NVIDIA Documentation:

"Spectrum-X employs global adaptive routing to quickly reroute traffic during link failures, minimizing disruptions and preserving optimal storage fabric utilization."​

"RoCE Adaptive Routing avoids congestion by dynamically routing large AI flows away from congestion points. This approach improves network resource utilization, leaf/spine efficiency, and performance."​

To enable RDMA over Converged Ethernet (RoCE) on a Cumulus switch, the correct command is:​

nv set qos roce enable on

This command configures the Quality of Service (QoS) settings to support RoCE, ensuring that the necessary parameters for lossless Ethernet are applied.​