Proxmox Server vGPU - Revision history

Boerj2 at 18:19, 20 March 2024

2024-03-20T18:19:15Z

← Older revision		Revision as of 14:19, 20 March 2024
Line 37:		Line 37:
	# Confirm the IOMMU parameter with <code>cat /proc/cmdline</code>		# Confirm the IOMMU parameter with <code>cat /proc/cmdline</code>
	Reboot the system		Reboot the system

			Confirm that IOMMU is enabled after reboot by running: <code>dmesg \| grep -e DMAR -e IOMMU</code>

	===== Check PCIe devices show =====		===== Check PCIe devices show =====
Line 49:		Line 51:
	\|0x030200 \|0x1eb8 \| 0000:3b:00.0 \| 3 \| 0x10de \| TU104GL [Tesla T4] \|		\|0x030200 \|0x1eb8 \| 0000:3b:00.0 \| 3 \| 0x10de \| TU104GL [Tesla T4] \|
	├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────┼		├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────┼
	</syntaxhighlight>		</syntaxhighlight>'''CONFIRM''' that each of the devices has a unique IOMMU group.

	== Activate GPU Passthrough ==		== Activate GPU Passthrough ==

Boerj2: /* Verify IOMMU Isolation */

2024-03-20T17:55:30Z

Verify IOMMU Isolation

← Older revision		Revision as of 13:55, 20 March 2024
Line 24:		Line 24:
	vfio_virqfd		vfio_virqfd
	EOF		EOF
	</syntaxhighlight>		</syntaxhighlight>Update the kernel by running <code>update-initramfs -u -k all</code>

	===== Update GRUB =====		===== Update GRUB =====

Boerj2 at 17:45, 20 March 2024

2024-03-20T17:45:51Z

@@ Line 31: / Line 31: @@
 # Run <code>proxmox-boot-tool refresh</code>
 Reboot the system
@@ Line 104: / Line 109: @@
 systemctl enable --now nvidia-sriov.service
 </syntaxhighlight>

Boerj2 at 17:44, 20 March 2024

2024-03-20T17:44:31Z

← Older revision		Revision as of 13:44, 20 March 2024
Line 31:		Line 31:
	# Run <code>proxmox-boot-tool refresh</code>		# Run <code>proxmox-boot-tool refresh</code>

	Reboot the ~~systemFor~~ working PCI passthrough, you need a dedicated IOMMU group for all PCI devices you want to assign to a VM.		Reboot the system

			===== Check PCIe devices show =====
			For working PCI passthrough, you need a dedicated IOMMU group for all PCI devices you want to assign to a VM.

	When executing (replacing {nodename} with the name of your node):<syntaxhighlight lang="bash" line="1" start="1">		When executing (replacing {nodename} with the name of your node):<syntaxhighlight lang="bash" line="1" start="1">

Boerj2: /* Verify IOMMU Isolation */

2024-03-20T17:40:27Z

Verify IOMMU Isolation

@@ Line 17: / Line 17: @@
 === Verify IOMMU Isolation ===
-For working PCI passthrough, you need a dedicated IOMMU group for all PCI devices you want to assign to a VM.
+Add CPU passthrough for the modules at boot time<syntaxhighlight lang="bash" line="1" start="1">
 When executing (replacing {nodename} with the name of your node):<syntaxhighlight lang="bash" line="1" start="1">

Boerj2: Created page with " This assumes NVidia vGPU compatible cards and is meant for the VXRail type system. See the official Proxmox documentation for consumer card passthroughs. == PCI Passthrough == === Verifying IOMMU parameters === Verify IOMMU is enabled ==== iDRAC ==== # Log into the iDRAC # Select the BIOS options under Configuration from IDRAC # Select Processor settings and ensure that Virtualization Technology is enabled.File:Virtualization Technology in VXRail.png # Click app..."

2024-03-20T16:42:41Z

Created page with " This assumes NVidia vGPU compatible cards and is meant for the VXRail type system. See the official Proxmox documentation for consumer card passthroughs. == PCI Passthrough == === Verifying IOMMU parameters === Verify IOMMU is enabled ==== iDRAC ==== # Log into the iDRAC # Select the BIOS options under Configuration from IDRAC # Select Processor settings and ensure that Virtualization Technology is enabled.File:Virtualization Technology in VXRail.png # Click app..."

New page

This assumes NVidia vGPU compatible cards and is meant for the VXRail type system. See the official Proxmox documentation for consumer card passthroughs.

== PCI Passthrough ==

=== Verifying IOMMU parameters ===
Verify IOMMU is enabled

==== iDRAC ====

# Log into the iDRAC
# Select the BIOS options under Configuration from IDRAC
# Select Processor settings and ensure that Virtualization Technology is enabled.[[File:Virtualization Technology in VXRail.png]]
# Click apply if changes were made at the bottom of the list and reboot [[File:IDRAC settings apply button.png]]

==== LifeCycle Controller ====
TBC

=== Verify IOMMU Isolation ===
For working PCI passthrough, you need a dedicated IOMMU group for all PCI devices you want to assign to a VM.

When executing (replacing {nodename} with the name of your node):<syntaxhighlight lang="bash" line="1" start="1">
# pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""
</syntaxhighlight>you should get:<syntaxhighlight lang="bash">
┌──────────┬────────┬──────────────┬────────────┬────────┬────────────────────────────────────────┬...
│ class │ device │ id │ iommugroup │ vendor │ device_name |
╞══════════╪════════╪══════════════╪════════════╪════════╪════════════════=═══════════════════════╪
|0x030200 |0x1eb8 | 0000:3b:00.0 | 3 | 0x10de | TU104GL [Tesla T4] |
├──────────┼────────┼──────────────┼────────────┼────────┼────────────────────────────────────────┼
</syntaxhighlight>

== Activate GPU Passthrough ==

=== Blacklisting drivers ===
The standard NVidia and nouveau drivers from Linux needs to be blacklisted. <syntaxhighlight lang="bash">
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia*" >> /etc/modprobe.d/blacklist.conf
</syntaxhighlight>After blacklisting, you will need to reboot.

=== Setup Proxmox VE Repositories ===

==== Ensure that the correct repositories are enabled: ====
Ensure that the relevant repositories are enabled. Most systems we use are using the <code>no-subscription</code> model. You can use the <code>Repositories</code> management panel in the Proxmox VE web UI for managing package repositories, see the [https://pve.proxmox.com/wiki/Package_Repositories documentation] for details.

==== Setup DKMS: ====
Because the NVIDIA module is separate from the kernel, it must be rebuilt with Dynamic Kernel Module Support (DKMS) for each new kernel update.

To set up DKMS, you must install the headers package for the kernel and the DKMS helper package. In a root shell, run<syntaxhighlight lang="bash" line="1" start="1">
apt update
apt install dkms libc6-dev proxmox-default-headers --no-install-recommends

</syntaxhighlight>

== Installing Host Drivers ==
'''NOTE:''' The drivers can be downloaded using the btechts account on the nVidia website or found under the folder <code>\VMware\7 Ent - VXRail\nVidia\</code> of the software share.

# Copy the '''KVM''' based run file to the host/node
# Make the file executable
# Install the drivers with the dkms module enabled
<syntaxhighlight lang="bash" line="1" start="1">
chmod +x NVIDIA-Linux-x86_64-xxx.xxx.xx-vgpu-kvm.run
./NVIDIA-Linux-x86_64-xxx.xxx.xx-vgpu-kvm.run
</syntaxhighlight>After the installer has finished successfully, you will need to reboot your system, either using the web interface or by executing <code>reboot</code>.

=== Enable SR-IOV ===

# Create the folder structure <code>/usr/local/lib/systemd/system/</code>
# Create the file <code>/usr/local/lib/systemd/system/nvidia-sriov.service</code>
# Add the service information below to the service
# Reload the systemctl daemon
# Enable the SR-IOV module
<syntaxhighlight lang="bash" line="1" start="1">
mkdir -p /usr/local/lib/systemd/system/
cat <<EOF > /usr/local/lib/systemd/system/nvidia-sriov.service
[Unit]
Description=Enable NVIDIA SR-IOV
After=network.target nvidia-vgpud.service nvidia-vgpu-mgr.service
Before=pve-guests.service

[Service]
Type=oneshot
ExecStart=/usr/lib/nvidia/sriov-manage -e ALL

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now nvidia-sriov.service
</syntaxhighlight>

== Configure GRUB ==

=== ZFS File systems ===

# Edit the file <code>/etc/kernel/cmdline</code> and add to the end of it <code>intel_iommu=on</code>
# Run the command <code>proxmox-boot-tool refresh</code>
# Reboot the machine
# Confirm the IOMMU parameter with <code>cat /proc/cmdline</code>