Do you think we will see optimizations in the future that will make something like 5060ti as fast as 3090?
I am a super noob but as I understand it, right now:
1) GGUF model quants are great, small and accurate (and they keep getting better).
2) GGUF uses mixed data types but both 5060ti and 3090 (while using FlashAttention) just translate them to fp16/bf16. So it's not like 5060ti is using it's fp4 acceleration when dealing with q4 quant.
3) At some point, we will get something like Flash Attention 5 (or 6) which will make 5060ti much faster because it will start utilizing its FP4 acceleration when using GGUF models.
4) So, 5060ti 16GB is fast now, it's also low power and therefore more reliable (low power components break less often, because there is less stress). It's also much newer than 3090 and it has never been used in mining (unlike most 3090s). And it doesn't have VRAM chips on the backplate side that get fried overtime time (unlike 3090).
Now you might say it comes to 16GB vs 24GB but I think 16GB VRAM is not a problem because:
1) good models are getting smaller 2) quants are getting more efficient 3) MoE models will get more popular and with them you can get away with small VRAM by only keeping active weights in the VRAM.
Do I understand this topic correctly? What do you think the modern tendencies are? Will Blackwell get so optimized that it will become extremely desirable?
[link] [comments]
