NVIDIA AI infrastructure relies on a tightly coupled stack involving the GPU driver and the DOCA (Data Center Infrastructure-on-a-Chip Architecture) drivers, which include the OFED stack for InfiniBand. When upgrading from an older version, " residue " from previous installations or custom kernel modules can cause version mismatches or symbol errors, preventing the new drivers from loading. The " 100% verified " best practice is to perform a clean installation. This involves completely uninstalling the existing drivers (using apt- get purge or the .run file ' s --uninstall flag) and ensuring that kernel modules are removed from memory. This prevents conflict between the legacy nvidia.ko and the new version. Once the system is " clean, " the new DOCA and GPU drivers can be installed as a matched set, ensuring that features like GPUDirect RDMA function correctly without being hindered by legacy configuration files.

The correct command sequence is sudo apt-get purge nvidia-* & & sudo apt-get autoremove. Purging removes NVIDIA driver packages and their configuration files, while autoremove removes dependency packages that were installed automatically and are no longer required. NVIDIA's current data center driver removal guidance for Ubuntu 22.04 and 24.04 uses apt remove --autoremove --purge with NVIDIA-related package patterns, including driver, Fabric Manager, firmware, headless, DKMS, compute, decode, encode, and related libraries. A simple apt-get remove nvidia-driver-550 removes only one package and may leave DKMS modules, libraries, configuration files, Fabric Manager packages, or version-specific dependencies behind. Manually deleting /usr/lib/nvidia is unsafe because it bypasses package management and can leave the system in an inconsistent state. ubuntu-drivers uninstall is not the most complete answer for fully purging all NVIDIA driver components during conflict troubleshooting. After removal, administrators should verify loaded modules, package state, and reboot before installing the intended validated driver stack. Clean removal is important in AI infrastructure because driver conflicts can break CUDA, NCCL, DCGM, Fabric Manager, and container runtime behavior.

The first step is to review the firmware update logs and run nvsm show health. After a DGX H100 firmware upgrade, a missing GPU can result from incomplete firmware activation, failed component update, PCIe enumeration failure, GPU tray communication issues, BMC inventory mismatch, or an actual hardware fault.
NVSM is the correct DGX platform-level health tool because it checks hardware state across GPUs, NVSwitch components, PCIe devices, storage, power, cooling, and system sensors. Firmware logs are equally important because they show whether each update completed successfully and whether a reboot, cold power cycle, BMC reset, or AC power cycle is still required. Replacing the GPU immediately is premature and may cause unnecessary downtime. Ignoring the issue is unsafe because production AI workloads expect all GPUs to be visible and healthy. Re-running firmware across all components without diagnosis can hide the original failure or introduce more risk. Proper bring-up practice is to collect evidence, verify hardware health, confirm firmware activation state, and then decide whether reseating, power cycling, reapplying firmware, or service escalation is required.

The nvidia-smi (NVIDIA System Management Interface) utility is the primary tool for monitoring and managing the state of NVIDIA GPUs. When a node exhibits " jitter " or lower performance compared to its peers in a cluster, nvidia-smi provides the necessary granular data to identify the bottleneck. It reports critical metrics such as GPU utilization percentages, memory usage, and, most importantly, " Clocks Throttle Reasons. " If a GPU is running slower due to power capping, thermal issues, or a hardware error (like an uncorrectable ECC error), nvidia-smi will display this state immediately. While lspci (Option A) can confirm if the GPU is physically visible on the PCIe bus, it cannot provide any telemetry regarding its operational performance or health. iblinkinfo (Option D) is a network-level tool that only monitors InfiniBand link states.
In an AI infrastructure context, nvidia-smi is the first-line diagnostic to determine if a GPU is " healthy " and operating at its intended clock speeds.

Updating an NVIDIA DGX system (like the H100) is a multi-layered process because the system contains numerous programmable logic devices, including CPLDs, FPGAs, and the EROT (Electrically Resilient Root of Trust) modules. Many of these low-level hardware components cannot be updated via a simple operating system reboot. NVIDIA's official firmware update procedure requires a specific sequence to "commit" the new images to the hardware. First, the update utility (like nvfwupd) writes the images to the flash memory. To activate them, a "Cold Power Cycle" (removing and restoring power) is necessary to force the hardware to reload from the newly written flash blocks. Furthermore, because the BMC (Baseboard Management Controller) orchestrates the power-on sequence and monitors the EROT, it must be reset (Option D) to synchronize its state with the new component versions. Finally, an "AC Power Cycle" ensures that even the standby-power components, such as the power delivery controllers and CPLDs, undergo a full hardware reset.
Skipping these steps can result in "Incomplete" or "Mismatched" firmware versions, where the OS reports one version while the hardware continues to run old, potentially buggy code in the background.