DeepSeek-R1 on a Windows GPU Box
DeepSeek-R1-Distill-Qwen-7B runs as the local reasoning brain on the CyberPower Windows GPU box at port 1234, served by llama.cpp under a reboot-survivable NSSM LlamaServer service. The IQ3_XS imatrix quant (3.19 GB) fully offloads all 29 layers onto the 4 GB AMD GPU via Vulkan, hitting 31.9 tok/s with model weights at 2962 MB plus KV cache at 238 MB for roughly 3.2 GB total residency.
Service and Quant Configuration
The endpoint loads C:\Users\dajai\deepseek-7b-iq3xs.gguf with the flags --port 1234 --host 0.0.0.0 -c 8192 -t 8 -np 1 --cache-ram 2048 -fa on -ctk q8_0 -ctv q8_0 -ngl 99 so every layer stays on-device. NSSM registers the service on the LAN address 192.168.0.140 so the box returns as DeepSeek after restarts without manual intervention. Quality checks still score 5/5 on multi-step reasoning even at this aggressive quant, and callers must request at least 600 max tokens or the content field arrives empty. A separate LM Studio instance on 1235 is ignored; only the NSSM service on 1234 is the production brain.
Why Full On-GPU Fit Beats Larger Quants
A smaller imatrix quant that fits entirely inside the 4 GB VRAM removes the CPU-layer bottleneck that previously left nine layers on the host under Q4, where every forward pass paid PCIe transfer costs. Hybrid execution therefore stalled at 12.6 or 18.3 tok/s baselines, while pure GPU residency delivers the measured 153 percent and 75 percent speedups to 31.9 tok/s. On limited AMD hardware the decisive lever is choosing a quant small enough for complete ngl 99 coverage rather than chasing higher bit-width at the expense of partial offload. The result is a responsive local reasoning engine free of external API latency and free of the silent model-swap failures that once broke the stack.