I want to locally run decent llms, the best cost effective setup i thought of is 8 v100 (16gb) on a 4028GR-TXRT for the x8 nvlink if i find a barebones one or a SYS-4028GR-TRT for 900 usd and run a custom watercooling setup with watercooling blocks from aliexpress (theyre around 35 usd each) and run the v100 setup at 75% power or lower for higher efficiency
the v100 cost 99usd including their heatsink, this setup has 128gb of vram and im planning on not putting any of the model's weights on the ram so it wont have abyssmally shit performance
it comes out cheaper than an rtx 5090 while having better performance (on paper)
has anyone tried this setup and can tell if its a waste of money and time ? its cheaper than a 128gb vram/lpddr ryzen halo max+ 395 or whatever its named
[link] [comments]
