The reflex when you want local AI is to buy one giant GPU and cram everything onto it. I did the opposite. I split my local stack across two machines I already owned — a Mac and a Windows box with a modest graphics card — and the two-machine version is faster, cheaper, and harder to knock over than the single-box build would have been. Here's the pattern, end to end.
The split: brain vs muscle
The two halves do different jobs, and I keep them physically separate so they stop fighting over the same resources.
The Mac is the brain. It orchestrates everything — it decides what runs when, holds the
application logic, and serves a small vision model on Apple Silicon through MLX on
localhost:9445. Vision tagging is bursty and latency-sensitive, and Apple's unified memory
handles a 4-bit vision model without breaking a sweat. Keeping that on the Mac means the machine
I actually interact with stays responsive.
The Windows box is the muscle. It's an old CyberPower tower with a 4GB AMD GPU, and its only
job is heavy reasoning inference on an OpenAI-compatible endpoint at :1234. Long chains of
thought, multi-step problems — the grind work. When a reasoning request takes twenty seconds,
it takes twenty seconds on a machine sitting in the corner, not on the laptop I'm typing into.
That's the whole idea: put the interactive, memory-friendly work on the Mac, and exile the slow, GPU-bound grind to a box whose only purpose is to grind.
Fitting a 7B reasoning model on a 4GB GPU
The GPU box has a 4GB card. A 7-billion-parameter model at a normal quant does not fit in 4GB, so the naive setup spills half the layers onto the CPU — and those CPU layers become the whole bottleneck. My first pass ran about 12.6 tokens/sec that way. Painful.
The fix is counterintuitive: use a smaller quant so the entire model fits on the GPU. I moved to an IQ3_XS imatrix quant of the reasoning model — roughly 3.2GB of weights and KV cache combined — and full-offloaded every layer to the GPU over Vulkan. All 29 layers on-card, zero on CPU. That took throughput to about 31.9 tokens/sec — a 150%+ jump over the split-layer baseline, on the exact same hardware.
The lesson generalizes: on a memory-starved GPU, the win isn't a higher-fidelity quant that half-fits. It's the largest quant that fits entirely on the card. Eliminating the CPU-layer handoff matters more than a marginal quality bump, and a good imatrix quant at IQ3 holds up fine for reasoning work. I kept a higher-fidelity quant on disk as a one-line revert if I ever decide the quality tradeoff isn't worth it — but I haven't needed it.
Two supervisors, one discipline
Two operating systems, two init systems — and I deliberately use each one's native supervisor instead of bolting a third-party process manager on top.
On the Mac, that's launchd. Each service — the vision server, the app processes — is a
launchd job with KeepAlive and RunAtLoad. Kill the vision server and it respawns in about
fifteen seconds. Reboot the Mac and the whole stack comes straight back with no manual step.
On the Windows box, the equivalent is NSSM — the Non-Sucking Service Manager — which wraps the llama.cpp server as a proper Windows service. Same guarantees: it auto-restarts on crash, it survives a reboot, and it's supervised by the OS rather than by something I have to remember to start.
The discipline is the same on both sides even though the tooling differs: the platform is the supervisor. I don't run pm2, I don't run a Docker daemon eating RAM in the background just to keep a process alive. Each OS already ships an init system built for exactly this. Use it.
Wiring the two together — and the fallback
Both inference servers speak the same OpenAI-compatible HTTP dialect, so from the application's point of view they're interchangeable endpoints on the LAN. The Mac talks to the GPU box over the local network; nothing here touches the public internet or a cloud API.
The one piece of glue that matters is a thin client wrapper. It health-checks the GPU box first and routes reasoning calls there. If that box is down — powered off, mid-reboot, model reloading — it falls back to a model on the Mac so the pipeline never hard-stops. Vision is the exception: it stays Mac-only, because that's where the MLX vision model lives and there's no second copy to fall back to.
That health-check-and-fallback is maybe forty lines of code, and it's what turns "two separate machines" into "one stack that happens to run on two machines."
Why heterogeneous beats one big box
Three reasons I'd build it this way again:
Cost. The marginal cost is zero. I reused hardware I already had instead of buying a single expensive GPU. The Windows tower was sitting idle; now it earns its keep.
Blast radius. When the GPU box goes down, the Mac keeps orchestrating and vision keeps working, and reasoning falls back. One machine failing degrades the stack; it doesn't kill it. A single-box build has no such seam — when it's down, everything is down.
Independent upgrades. I can swap the quant on the GPU box, or move to a bigger card, without touching the Mac. I can change the vision model on the Mac without the GPU box noticing. The two halves evolve on their own schedule because the only contract between them is an HTTP endpoint.
You don't need a datacenter to run real local AI. You need two machines you already own, a LAN between them, each OS's own supervisor keeping the processes alive, and a quant small enough to fit the whole model on whatever GPU you've got. That's the entire pattern.
FAQ
Why split local AI across two machines instead of using one powerful box?
Separation of concerns and resilience. The Mac stays interactive and handles vision on Apple Silicon; the Windows box absorbs the slow, GPU-bound reasoning work in the corner. If one machine goes down the other keeps running with a fallback, and you can upgrade either half without disturbing the other. A single box has no seam — when it's down, everything is.
How do you run a 7B model on a 4GB GPU?
Use a small enough quant that the entire model fits on the card, then offload every layer to the GPU. An IQ3_XS imatrix quant lands around 3.2GB including KV cache, which fits fully in 4GB over Vulkan. Full offload took throughput from ~12.6 tok/s (with layers spilling to CPU) to ~31.9 tok/s. The bottleneck was never the quant quality — it was the CPU-layer handoff.
Why NSSM on Windows and launchd on the Mac instead of Docker or pm2?
Both are the native init system of their OS, built to keep services alive and restart them on boot. Leaning on the platform's own supervisor means one fewer moving part to babysit — no process manager to resurrect, no Docker daemon holding RAM just to keep a server running. Same discipline on both machines, different tool per OS.
What happens when the GPU box is offline?
A thin client wrapper health-checks the GPU box before routing reasoning calls to it. If it's unreachable — powered off, rebooting, reloading a model — the wrapper falls back to a model on the Mac so the pipeline never hard-stops. Vision is the exception: it runs only on the Mac's MLX server, so there's no second copy to fall back to.