Everyone argues about which giant LLM to self-host. Almost nobody talks about the boring workload quietly billing them every month: embeddings. Every semantic search, every dedup pass, every "find similar items" feature fires an embedding call, and most stacks send those calls to a paid API out of pure habit.
Embeddings are the single easiest AI workload to bring home. Here's the pattern I run in production on a Mac Studio: a small Python sidecar on localhost that serves embeddings for free, fast enough that the API version has no argument left.
The workload profile
Think about what an embedding call actually is. Tiny input — a caption, a message, a filename. Tiny output — a fixed vector of floats. No generation, no sampling, no context window drama. It's a single forward pass through a small encoder, the kind of compute a laptop from five years ago handles without noticing.
Now look at what you pay to ship that workload across the internet: per-token pricing on volume that adds up precisely because embeddings get called constantly, plus network latency that usually exceeds the inference time itself, plus a hard dependency — your search breaks when the API hiccups, rate-limits you, or deprecates the model and re-prices the replacement.
For frontier-scale generation, those trade-offs can be worth it. For a 22-million-parameter encoder? You're renting a forklift to carry a grocery bag.
The sidecar
My version is one Python file — embed-server.py — wrapping sentence-transformers with the all-MiniLM-L6-v2 model behind a minimal HTTP endpoint on a local port (mine sits on 9447). MiniLM produces 384-dimension vectors and the sidecar pushes roughly 80 items per second on Apple Silicon — throughput to embed an entire media vault of several thousand items in about a minute, on hardware I already own, at a marginal cost of zero.
The design choices that matter:
It's a sidecar, not a daemon. It doesn't run at boot and doesn't sit resident eating RAM. When a job needs embeddings, the job (or I) starts it, the work runs, and it goes away. Batch-shaped workloads deserve batch-shaped processes — a permanently-resident server for something you call in bursts is how a Mac quietly loses its memory headroom.
Consumers point at an env var. Every script that embeds anything reads the endpoint from LOCAL_EMBED_URL instead of hard-coding a provider. That one convention is the whole migration story: when I swapped my vault-tagging pipeline from a remote embedding call to the sidecar, the change was an environment variable, not a refactor. If I ever swap MiniLM for a bigger encoder, nothing downstream changes either.
One model, pinned. The sidecar serves exactly one model. Vectors are only comparable to vectors from the same model, so a "flexible" embedding server is a foot-gun — mix models and your similarity search silently degrades into noise. Pinning the model in the sidecar makes the invariant structural.
What it powers
This isn't a demo — the sidecar backs real search over a production content vault: thousands of media items, each captioned by a local vision model, each caption embedded into the 384-dim space, all of it queryable with cosine similarity in SQLite. "Find items like this one" runs entirely on my desk. No tokens metered, no content leaving the machine — which matters more than the money when the data is your business's actual inventory.
That's the under-appreciated part of local embeddings: privacy comes free. An embedding API sees every string you embed. For a business whose content is the asset, shipping the entire catalog to a third party as a side effect of making it searchable is a strange default. The sidecar makes the private path the lazy path.
Do the math on your own stack
The pattern generalizes to a simple rule: match the model to the workload, and match the process shape to the call pattern.
- Tiny model, high call volume, small payloads → local sidecar, always. Embeddings are the canonical case; rerankers and classifiers are the same shape.
- Bursty batch jobs → boot on demand, don't daemonize.
- Decouple with an env var → consumers should never know or care what's behind the endpoint.
- Pin the model → comparable vectors or bust.
MiniLM is 22M parameters and open source. sentence-transformers is a pip install. The server is under a hundred lines. Against that, a metered API charging you monthly to run a grocery-bag model in someone else's building — with your data in the envelope — isn't a serious option anymore. It's just the default you haven't questioned yet.
Bring the small models home first. The big ones can wait.
FAQ
Is a 384-dimension local model actually good enough for real search?
For most practical semantic search — captions, messages, titles, tags — yes, comfortably. MiniLM sits in the sweet spot where quality is solid and speed is effectively free. If your domain is highly specialized, test a larger encoder locally before assuming you need an API: the jump from "small local" to "big local" solves most gaps before "paid remote" enters the picture.
Why not just run the embeddings inside each script instead of a server?
Model load time. Importing the model costs seconds per process; a sidecar loads it once and serves every consumer over localhost. You keep one warm copy of the weights instead of paying the load tax in every script, and every language in your stack can hit it over plain HTTP.
What hardware does this need?
Almost any. MiniLM at ~80 items/sec is Apple Silicon performance, but the model is small enough that a modest x86 box or an old laptop serves fine at lower throughput. This is the whole point — embeddings are the AI workload that does not need the GPU you don't have.
How do I migrate an existing pipeline off a paid embedding API?
Route the endpoint through an environment variable, stand up the sidecar, then re-embed your corpus with the local model — don't mix old and new vectors, they live in different spaces and similarity across them is meaningless. A few thousand items re-embed in minutes at local throughput, and after that the meter simply stops.