The final monster: 32 GPUs and 1 TB of VRAM
- 32× AMD MI50 32GB - 1 terabyte of VRAM
- Kimi K2.6 int4: 9.7 tokens/s generation · ~264 tokens/s prompt processing
- Architecture: 2 nodes of 16 GPUs linked over 10Gb Ethernet
- ~640 W idle / ~4,800 W peak inference
- My honest verdict: for those with free energy only
After the 16 GPUs, only one step was left: the monster. Two nodes of 16 GPUs, simply linked by a 10Gb Ethernet cable. One terabyte of VRAM on a shelf, at home. Enough to load Kimi K2.6, a model with over a trillion parameters, entirely locally: 9.7 tokens/s generation, ~264 tokens/s prompt processing.
Am I proud of the result? Yes. Would I do it again? No. And I wrote exactly that in the post: unless you have solar panels or free energy, this setup does not make sense. At idle it draws ~640 W, and at full inference it climbs to ~4,800 W - water-heater territory, continuously.
And to be completely transparent: before taking the photo, the monster was off. I had turned it on only for that. My point was to show one thing: one day, we will be able to load an AGI locally by adding up consumer GPUs. That day has not come yet - but we are much closer than most people imagine.
The end of an era (and the start of another)
With more time on this build, I am convinced much better is possible: my risers were unstable (3.5 GB/s instead of the expected 7 GB/s), and a better-optimized software stack should reach 600–1,000 tokens/s prompt processing and 10–12 tokens/s generation at least. But I decided to stop there. Small models on small rigs are simply faster, more energy-efficient and more pleasant for everyday use - and that is where the real life of local AI happens.
What the community asked me
83 comments, plenty of humor ("I felt a great disturbance in the Grid"), and some real questions:
- "640 W at idle?! That's huge."
- It is, and it is the real hidden cost of big rigs: fans, PCIe switch cards, the machine itself. The answer is simple: shut it down when you are not using it. Plenty of people on the subreddit do the same - the machine only runs when it serves.
- "So, was it worth it?"
- For learning and pushing the limits: clearly yes. For daily use: no, unless free energy. Hence my return to smaller, more efficient setups.
- "Why doesn't generation speed scale with more GPUs?"
- Because generation is bound by memory bandwidth, not compute - adding cards does not make it explode. Prompt processing, on the other hand, benefits a lot from the added compute.