lukeed

Four Sparks and a switch

Field notes from wiring four DGX Sparks into one inference cluster. The fabric works now. Here is everything that cost me a day.

494 wordshardware · networking · inference · rdma

Four NVIDIA DGX Sparks sit on a shelf in my office. Each one is a GB10 with 128 GB of unified memory. They connect to a MikroTik CRS812 through 400G ports split into 2×200G, and together they serve one OpenAI-compatible endpoint.

It works. Getting there took a month, and most of that month was spent learning rules that nobody wrote down.

CRS812 4 × 400G → 2 × 200G breakout spark1 spark2 spark3 spark4 GB10 · 128 GB GB10 · 128 GB GB10 · 128 GB GB10 · 128 GB rail A rail B
Two 200G rails per node, eight lanes into one switch.

#Where it landed

Layer Result
RDMA 111.9 Gb/s per rail, both rails, all four nodes
Aggregate about 223 Gb/s per node, against NVIDIA's 185
NCCL all_reduce, 4 nodes 20.7 to 22.0 GB/s bus bandwidth, zero errors
Serving single node, tensor parallel across 2 or 3, or N replicas behind one endpoint

#Rule 1: never hot-plug the QSFP cables

If you re-seat a cable on a powered Spark, the CX-7 drops egress to about 12.7 Gb/s. Every error counter reads zero. A reboot does not fix it. A link down and up does not fix it.

healthy rail after hot-plug 111.9 12.7 Gb/s Every error counter reads zero in both cases.
Same cable, same port, same counters. Nine times slower.

Only a full AC drain clears it:

sudo poweroff
# unplug AC, hold the power button, wait 60–120 s
# move cables, then restore power

If throughput is ever between 12 and 16 Gb/s, check this first. Not the switch, not the MTU, not QoS, not TCP. I lost two days to the switch.

With 2×200G breakout, the lanes stay dark until the switch speed is set by hand. Dead lanes with cables attached are the normal state before configuration, not a fault. I filed that one as a bad cable, twice.

#Rule 3: fabric addresses are not in hostname order

NVIDIA Sync hands out fabric addresses by ascending MAC address. So spark2 got .1 and spark1 got .2. Adding or reimaging a node renumbers the others. Never write a fabric address into a config by hand.

#Rule 4: pin .local names in /etc/hosts

Avahi advertises every address on the fabric interface, and the fabric has no route from my laptop. Overriding HostName in ~/.ssh/config means putting your block above Sync's Include, which also puts your IdentityFile first, and Sync cannot use a key with a passphrase. /etc/hosts has none of these problems.

#Rule 5: container tags are dates

nvcr.io/nvidia/vllm:26.05-py3 ships vLLM 0.20.1. The tag is the YY.MM of the container release, not the vLLM version. What actually gates a model is the architecture registry check, not a version number.

#Daily use

Every model is a recipe. One script starts, measures, and stops them:

./bin/srun list
./bin/srun start qwen3.6-35b-a3b-nvfp4
./bin/srun bench qwen3.6-35b-a3b-nvfp4 -c 4
./bin/srun status

There is one GB10 per Spark, so tensor parallelism must divide the node count. The quotient is the replica count. A small body router, sprouter, reads the model field of each request and sends it to the replica with the fewest requests in flight. Clients see one base URL.

./bin/srun start deepseek-v4-flash-0731   # TP=2 on spark1+spark2 → 1 replica
./bin/srun start glm-5.2                  # TP=3 on spark2+3+4    → 1 replica

Next is a Firecracker layer, so agents get a disposable machine next to the model they talk to.