The five minutes my blog served a half-built site
A few weeks ago I took my own blog down. Not with a bad commit, not with a bad server config — with a race. Two builds kicked off against the same .next directory at nearly the same time. One was mid-write when the running server tried to serve a route, and Next.js threw the invariant everyone who has done this recognizes:
Error: ENOENT: client reference manifest does not exist
The site was up. The process was healthy. Cloudflare was routing traffic. And every request 500'd for about five minutes because the directory the live server was reading out of was, at that instant, a half-built output.
I run three production Next.js sites on one Mac Studio — dajai.io, hellcatblondie.io, sovereignagiasi.com — each a separate launchd service on its own port, all fronted by a single Cloudflare tunnel. When you self-host like this, there's no platform quietly doing blue-green for you. If your deploy script has a broken window, your visitors live in it. This is the discipline I built so a rebuild can never serve a broken site again.
Why "rm -rf .next && build && restart" is a trap
The naive self-host deploy looks reasonable:
cd /srv/site
git pull
rm -rf .next
npm run build
launchctl kickstart -k gui/501/io.dajai.nextjs
There are two broken windows hiding in there.
The first opens the instant rm -rf .next runs. Between that line and the moment the new build finishes, the live server is pointed at a directory that is empty or partial. Every request in that window fails. A Next.js build on my hardware is 40–90 seconds. That's a minute-plus of guaranteed 500s on every deploy, even when nothing goes wrong.
The second window is the one that actually bit me: nothing stops two of these from running at once. A cron-triggered rebuild and a manual deploy overlapping means two npm run build processes writing into the same .next. The output is corrupt the moment they interleave, and the manifest error is exactly what that corruption looks like.
The fix is not "be more careful." The fix is to make the unsafe state impossible.
Rule 1: Never build in place over what the server is reading
Build to a fresh directory. Keep the live one untouched until the new one is proven good. Then flip a symlink — an atomic operation — so the server starts reading the new output on the very next request with no in-between state.
RELEASE="/srv/site/releases/$(date +%Y%m%d-%H%M%S)"
git clone -q /srv/site/repo "$RELEASE"
cd "$RELEASE"
npm ci --silent
npm run build # writes into $RELEASE/.next — the live site never sees this
The running server is still happily serving the current release this entire time. Nothing it reads has changed.
Rule 2: A lock, so two deploys can't stomp each other
This is the part that would have saved me. Before anything touches disk, take an exclusive lock. If another deploy holds it, exit — don't wait, don't queue, just refuse.
exec 9>/srv/site/deploy.lock
if ! flock -n 9; then
echo "another deploy is running; aborting"
exit 1
fi
flock -n is non-blocking. The second overlapping deploy dies immediately instead of racing the first. One line, and the exact failure that took my blog down becomes structurally impossible.
Rule 3: Validate the build BEFORE you cut over
A build can exit 0 and still be missing the artifacts that matter. So before I flip anything, I check that the output actually contains what a working site needs — the build manifest and the critical route files.
test -f "$RELEASE/.next/BUILD_ID" || { echo "no BUILD_ID"; exit 1; }
grep -q "app-build-manifest" "$RELEASE/.next/app-build-manifest.json" \
|| { echo "manifest missing"; exit 1; }
Then I boot the new release on a scratch port and probe the routes I can't afford to have broken:
PORT=3999 node "$RELEASE/server.js" &
PROBE=$!
sleep 4
for path in / /blog /music; do
code=$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1:3999$path")
[ "$code" = "200" ] || { echo "$path -> $code"; kill $PROBE; exit 1; }
done
kill $PROBE
If any probe fails, the script exits and the live site is completely untouched — still serving the last known-good release. A bad build never reaches a visitor. This is the single most valuable habit on the list: cutover is gated on evidence, not on npm run build merely returning.
Rule 4: Flip atomically, then kickstart
Only now, with a validated build and the lock held, do I move the pointer:
ln -sfn "$RELEASE" /srv/site/current # atomic swap
launchctl kickstart -k gui/501/io.dajai.nextjs
ln -sfn replaces the symlink in one syscall — there is no moment where current points at nothing. The launchd service has its working directory set to /srv/site/current, so the kickstart restarts it against the new release. Cloudflare keeps routing to the same port the whole time; the tunnel never knew a deploy happened.
Because every release is its own dated directory, rollback is the same atomic flip in reverse:
ln -sfn /srv/site/releases/20260709-142200 /srv/site/current
launchctl kickstart -k gui/501/io.dajai.nextjs
I keep the last five releases and prune the rest. Disk is cheap; a five-minute outage is not.
What actually changed
The shape of a safe deploy is: lock, build fresh, validate, flip, kickstart. The naive script is: rm, build, restart. The difference between them is the difference between a deploy that has a broken window on every run and one that has no broken window ever — because the live directory is never the build target, two deploys can't overlap, and a bad build can't reach the symlink.
Self-hosting behind your own tunnel means you own the uptime. That's the whole appeal. It also means the discipline that platforms hide from you is now yours to write down and enforce. I learned that during five minutes of 500s I'd rather not repeat.
FAQ
Why did Next.js throw "client reference manifest does not exist"?
Because the server read the .next directory while a build was mid-write. The manifest file that Next.js expects for a route hadn't been written yet — or was being overwritten by a second concurrent build. It's a symptom of building in place over a live directory, not a bug in Next.js. Build to a fresh directory and the file is always complete before the server ever sees it.
Does a symlink flip really restart with zero dropped requests?
The flip itself is atomic — ln -sfn is a single syscall with no empty intermediate state. The brief gap is the service restart after kickstart, which on my setup is a second or two. For true zero-drop you'd run two instances and drain one, but for a personal/creator stack the symlink-plus-fast-restart pattern keeps the outage to the boot time of a warm Next.js server, not the 40–90 seconds a full rebuild takes.
Is flock enough, or do I need a real deploy queue?
For one machine, flock -n is enough. It's a kernel-level advisory lock; the second deploy sees the lock is held and exits immediately. You only need a queue if you want overlapping deploys to run sequentially instead of being refused — and for most self-hosted stacks, "refuse the second one, let the human retry" is the safer default than silently queuing a stale build behind a fresh one.
How many old releases should I keep?
I keep five. That covers "roll back one bad deploy" plus a little headroom to bisect if a regression slipped through a few deploys ago. Each release is a full copy of the built site, so watch disk if your output is large — but pruning to the last five has never cost me a rollback I needed.