I've come to a point where I find the 27b and 31b models quite impressive.
I have a 16 GB AMD Radeon 7800xt. It performs quite well. It was $700. Here is my question:
Is the dual GPU approach performance hit worth it if I save around $400 over a single larger card? Is 32gb even a meaningful step up and is running 9700xt pro with a second 7800xt for total of 48gb a more realistic requirement for these size models?
I would like to have more vram for running these models and I could go with dual 16 GB cards or a single larger card, but here's the cost difference:
A)
Sell 7800xt for $550.
Buy, single 9700xt pro , 32gb, $1900+ tax. Final cost $1600.
B)
Add second 7800xt, $550 on second hand market. Final cost $700 + $550.
C)
Add 9700xt pro, total price $1900+tax plus $700.
Price isn't a factor, only to outline the difference so that it can be compared with performance, to decide if it's even worth it.
The bandwidth of these cards is the same, except for the fact there's a second PCIe device.
I've been using llama.cpp, and like it, but vllm is an option if dual GPU setup on vllm runs better.
[link] [comments]
