So either way you have pro gpu (non geforce) or p2p enabled driver, but no nvlink bridge and you try vllm and it hangs....
In fact vllm relies on NCCL under the hood will try to p2p assuming it has nvlink. But if your gpu can p2p over pcie but still nvlink fails.
Thats why everywhere you see NCCL_P2P_DISABLE=0
So how can you use p2p over pcie ? By telling nccl which level of p2p is ok. https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-p2p-level
By adding VLLM_SKIP_P2P_CHECK=1 NCCL_P2P_LEVEL=SYS (of course if your iommu is properly setup) you tell nccl that whatever stuff he needs to cross on your motherboard is fine
Note: on saphire rappid pcie p2p is limited to gen 4 due to NTB limitations
Here the accepted values for NCCL_P2P_LEVEL
LOC : Never use P2P (always disabled) NVL : Use P2P when GPUs are connected through NVLink PIX : Use P2P when GPUs are on the same PCI switch. PXB : Use P2P when GPUs are connected through PCI switches (potentially multiple hops). PHB : Use P2P when GPUs are on the same NUMA node. Traffic will go through the CPU. SYS : Use P2P between NUMA nodes, potentially crossing the SMP interconnect (e.g. QPI/UPI). [link] [comments]
