Ok so hear me out, I have a rather unique situation here and wants some good recommendations.
I currently have a server (ESC8000A-E12) that's designed to host 8xH100, it's already set up and working with 2x2080TI with 22GB of mod. I got this very long ago during the stable diffusion era and the idea of running LLMs (ChatGPT was just a thing back then) on this never crossed my mind.
Jump to the present and everyone is deploying LLMs on their local hardware, and I'm currently thinking about "finishing" the machine by filling out the last 6 GPU slots. I have access to reliable supplies of 2080TI 22GB for ~$290 each. Giving me 176GB of VRAM for just under $2K.
However, I do understand that Turing is a very old architecture that doesn't even support BF16 (only FP16) or FA2. I've browsed on this reddit for some time looking for alternative solutions to compare. The best one I have is the 5060ti 16GB, which because of the FP4 support and better architecture, you could get a better per-GPU performance. But a 5060ti 16GB costs twice as much as the 2080TI 22GB, plus I would need to discard and replace the two I currently have. Yet I'm also concerned about the longevity of this, if support for Turing continue to degrade.
A 4090 with 48GB sounds good but a single one alone would cost me more than 8x2080ti 22GB.
Open to any suggestions, thanks in advance!
[link] [comments]
