Feedback on my 256gb VRAM local setup and cluster plans. Lawyer keeping it local.

I’m a lawyer who got Claude code pilled about 90 days ago, then thought about what I wanted to do with AI tools, and concluded that the totally safest way for me to experiment was to build my own local cluster. I did an earlier post about what I was working on, and the feedback was helpful.

Wondering if anyone has feedback or suggestions for me in terms of what I should do next.

Anyway, node 1 is basically done at this point. Gigabyte threadripper board, 256gbs of ddr4, and 8 32gb nvidia v100s. I have two PSUs on two different regular circuits in my office, 2800 watts total (haven’t asked the landlord for permission to install a 240 volt yet). I am running … windows … because I still use the computer for my regular old office work. But I guess my next steps for just this node are probably to get a 240 plug installed, and maybe add 2 or 4 more v100s, and then call it a day for node 1.

Took one photo of one of th 4-card pass through boards. Each of these NVlinks 128gbs of sxm v100s, and they get fed back into the board at x16 using two pex switches and 4 slim sass cables.

The only part that’s remotely presentable is the 4 card board I have finished. There’s a 2 card board on footers and 2pcie v100s. I have 2 more 2 card sxm boards and a 4 card sxm board in waiting. And 3 sxm v100s and heatsinks (slowly buying more).

Goal is to do local rag databases on the last 10 years of my saved work, to automate everything I can so that all the routine stuff is automatic and the semi routine stuff is 85% there. Trying to get the best biggest reasoning models to run, then to test them with rag, then to qlora train.

Wondering if anyone has suggestions on how to manage all the insane power cables this requires. I put this 4 card board in an atx tower case, and have one more for the second board, but I have the rest of the stuff (motherboard board, 2 pcie cards, 2 card sxm board) open bench/open air like a mining rig. Would love some kind of good looking glass and metal 3 level air flow box or something.

Also wondering if anyone has really used big models like GLM or full deepseek or minimax 2.5 locally for anything like this. And if anyone has done Qlora training for legal stuff.

In terms of what’s next, I will start on Node 2 after I get some of the stray heatsinks and riser cables out of my office and thermal paste off of my suit. I have a romed2 board and processor, and a variety of loose sticks of ddr4 server ram that will probably only add up to like 192gb. I have 3 rtx3090s. Plan is I guess to add a fourth and nvlink them.

My remaining inventory is a supermicro x10drg board and processor, 6 p40s, 6p100s, 4 16gb v100 sxms, another even older x10 board and processor, more loose sticks of server ram, and then a couple more board and processor combos (x299a 64gb ddr4, and my 2019 gaming pc).

Original plan (and maybe still plan) was to just have so much vram I could slowly run the biggest model ever over a distributed cluster, and use that to tell me the secret motives and strategy of parties on the other side of cases. And then maybe use it to tell me why I can never be satisfied and always want more. Worried Opus 4.6 will be better at all that.

I wrote this actual post without any AI help, because I still have soul inside.

Will re post it in a week with Claude rewriting it to see how brainwashed you all are.

Anyway, ask me questions, give me advice, explain to me in detail why I’m stupid. But be real about it you anime freaks.