View source for Proxmox Server vGPU

 This assumes NVidia vGPU compatible cards and is meant for the VXRail type system. See the official Proxmox documentation for consumer card passthroughs.

== PCI Passthrough ==

=== Verifying IOMMU parameters ===
Verify IOMMU is enabled

==== iDRAC ====

# Log into the iDRAC
# Select the BIOS options under Configuration from IDRAC
# Select Processor settings and ensure that Virtualization Technology is enabled.[[File:Virtualization Technology in VXRail.png]]
# Click apply if changes were made at the bottom of the list and reboot [[File:IDRAC settings apply button.png]]

==== LifeCycle Controller ====
TBC

=== Verify IOMMU Isolation ===
Add CPU passthrough for the modules at boot time<syntaxhighlight lang="bash" line="1" start="1">
cat << EOF >> /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
EOF
</syntaxhighlight>

===== Update GRUB =====

# Edit the file <code>/etc/default/grub</code> and to the ''GRUB_CMDLINE_LINUX_DEFAULT="quiet"'' add <code>intel_iommu=on iommu=pt</code>
# Run <code>proxmox-boot-tool refresh</code>

Reboot the system

===== Check PCIe devices show =====
For working PCI passthrough, you need a dedicated IOMMU group for all PCI devices you want to assign to a VM.

When executing (replacing {nodename} with the name of your node):<syntaxhighlight lang="bash" line="1" start="1">
# pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""
</syntaxhighlight>you should get:<syntaxhighlight lang="bash">
┌──────────┬────────┬──────────────┬────────────┬────────┬────────────────────────────────────────┬...
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                            |                         
╞══════════╪════════╪══════════════╪════════════╪════════╪════════════════=═══════════════════════╪
|0x030200  |0x1eb8  | 0000:3b:00.0 |   3        | 0x10de |  TU104GL [Tesla T4]                    |
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────┼
</syntaxhighlight>

== Activate GPU Passthrough ==

=== Blacklisting drivers ===
The standard NVidia and nouveau drivers from Linux needs to be blacklisted. <syntaxhighlight lang="bash">
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf 
echo "blacklist nvidia*" >> /etc/modprobe.d/blacklist.conf 
</syntaxhighlight>After blacklisting, you will need to reboot.

=== Setup Proxmox VE Repositories ===

==== Ensure that the correct repositories are enabled: ====
Ensure that the relevant repositories are enabled. Most systems we use are using the <code>no-subscription</code> model. You can use the <code>Repositories</code> management panel in the Proxmox VE web UI for managing package repositories, see the [https://pve.proxmox.com/wiki/Package_Repositories documentation] for details.

==== Setup DKMS: ====
Because the NVIDIA module is separate from the kernel, it must be rebuilt with Dynamic Kernel Module Support (DKMS) for each new kernel update.

To set up DKMS, you must install the headers package for the kernel and the DKMS helper package. In a root shell, run<syntaxhighlight lang="bash" line="1" start="1">
apt update
apt install dkms libc6-dev proxmox-default-headers --no-install-recommends

</syntaxhighlight>

== Installing Host Drivers ==
'''NOTE:''' The drivers can be downloaded using the btechts account on the nVidia website or found under the folder <code>\VMware\7 Ent - VXRail\nVidia\</code> of the software share.

# Copy the '''KVM''' based run file to the host/node
# Make the file executable
# Install the drivers with the dkms module enabled
<syntaxhighlight lang="bash" line="1" start="1">
chmod +x NVIDIA-Linux-x86_64-xxx.xxx.xx-vgpu-kvm.run
./NVIDIA-Linux-x86_64-xxx.xxx.xx-vgpu-kvm.run
</syntaxhighlight>After the installer has finished successfully, you will need to reboot your system, either using the web interface or by executing <code>reboot</code>.

=== Enable SR-IOV ===

# Create the folder structure <code>/usr/local/lib/systemd/system/</code>
# Create the file <code>/usr/local/lib/systemd/system/nvidia-sriov.service</code>
# Add the service information below to the service
# Reload the systemctl daemon
# Enable the SR-IOV module
<syntaxhighlight lang="bash" line="1" start="1">
mkdir -p /usr/local/lib/systemd/system/
cat <<EOF > /usr/local/lib/systemd/system/nvidia-sriov.service
[Unit]
Description=Enable NVIDIA SR-IOV
After=network.target nvidia-vgpud.service nvidia-vgpu-mgr.service
Before=pve-guests.service

[Service]
Type=oneshot
ExecStart=/usr/lib/nvidia/sriov-manage -e ALL

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now nvidia-sriov.service
</syntaxhighlight>

== Configure GRUB ==

=== ZFS File systems ===

# Edit the file <code>/etc/kernel/cmdline</code> and add to the end of it <code>intel_iommu=on</code> 
# Run the command <code>proxmox-boot-tool refresh</code> 
# Reboot the machine
# Confirm the IOMMU parameter with <code>cat /proc/cmdline</code>