I do not run any LLMs locally. But a guy can dream, can't he? Curious if this matches up with what anyone else who is running things locally has found.
ContentsContents
| Section | TL;DR |
| How much are you willing to spend? | $2k gets you Qwen and good STT (pretty far!); $40k gets you almost-Opus |
| Base system | Last-gen EPYC + eBay DDR4 for $5.6k |
| GPUs | 4× RTX PRO 6000, 384GB VRAM, where the money went |
| c-payne switch sub-BOM | Indie PCIe switching so GPUs talk peer-to-peer |
| GPU enclosure | A day of carpentry |
| Making the switch behave | BIOS bifurcation, link speed, ASPM |
| Kernel / GRUB params | iommu=off or NCCL hangs |
| ACS disable | Keep P2P traffic inside the switch fabric |
| GPU power limiting | Running $46k of silicon on a 110V circuit |
| Result | Gen4 line rate: 27.5/50.4 GB/s, sub-µs latency |
runners/ | Ready-to-run serving configs: https://github.com/jamesob/local-llm/blob/master/runners/GLM-5.2-594B: vLLM docker-compose, DCP4+MTP5, ~80 t/s @ 240k ctx |
tools/ | https://github.com/jamesob/local-llm/blob/master/tools/measure-gpu-speed.sh: P2P bandwidth/latency benchmark |
| Resources | rtx6kpro repo, c-payne |
I like the enclosure he built: