23 — Solomon: Inference Serving, Kubernetes & Autoscaling

Solomon 5 of 6 · Solomon/solomon/serving, Solomon/genengine/serving, Solomon/k8s Stack: FastAPI, SSE + WebSocket streaming, continuous batching, KV cache, Prometheus, Docker, k3s, Helm, HPA Status: Live in-cluster. HPA observed scaling to 3 replicas under load.

Where this sits. This is where the ML project becomes an infrastructure project — and the part most directly aligned with a Google Cloud / platform engineering loop.

The serving layer

FastAPI exposing:

SSE streaming and WebSocket streaming — two transports, one generation engine
Continuous batching — requests join an in-flight batch rather than waiting for a batch window
KV cache — proven exactly equal to full recompute (see the model)
Prometheus metrics and a generated OpenAPI surface
RS256 platform JWT verification — the same asymmetric-signing model as the Auth platform, with the 401 path verified, not assumed

Why continuous batching is the interesting choice

Static batching makes the first request in a window wait for the window to close. Continuous batching lets a request join a batch already decoding and lets finished sequences leave without stalling the rest. It is the difference between throughput that looks good on a benchmark and latency that is acceptable to a user — and it is why p95 454 ms and 116 tok/s could both be promotion gates rather than a trade-off.

Kubernetes

k3s + Helm. Chart templates cover a multi-phase topology:

k8s/helm/templates/
  inference.yaml              the serving deployment + HPA
  mlflow.yaml                 in-cluster tracking
  phase2-orchestrator.yaml    request orchestration
  phase2-members.yaml         ensemble members
  phase2-retrieval.yaml       the Go retrieval tier
  phase2-search.yaml          search
  phase2-networkpolicy.yaml   east-west traffic restriction
  phase3-gend.yaml            generation daemon
  phase3-host-members.yaml    host-resident members
  secret.yaml

Manifests pass kubectl apply --dry-run=server — validated against the live API server, not just parsed.
The image was rebuilt with the promoted checkpoint and rolled out, so the running pods are provably the artifact that passed the gates.
A NetworkPolicy restricts east-west traffic rather than leaving the namespace flat.

Load behavior

Metric	Result
Requests	198
Failures	0
p95 latency	2.7 s
SLO	6 s
HPA	scaled to 3 replicas under load

The blockers, written down

k8s/INGRESS.md records what could not be done from this host: the Cloudflare Tunnel hostnames are dashboard-managed, so the public hostname → the in-cluster service and the API hostname → the in-cluster service are manual steps, documented as such. BLOCKERS.md records that Phase 2 must restart its general-member to load the promoted weights.

Separating "done" from "blocked on something outside this host" — and naming the exact hostname and port — is the operational writing habit that shows up again in the gap registers.

Known defects, recorded not hidden

From CODE_REVIEW.md:

SSE per-token decode drops inter-word spaces (server and WebSocket path).
The engine swallows generation exceptions → an empty 200 followed by [DONE].
Disconnect-cancellation is dead code → leaked generations when a client hangs up.
make deploy under sudo strips KUBECONFIG.
A prompt-trim edge case at max_tokens ≥ max_len − 1.

Defect 2 is the one to discuss: a streaming endpoint that returns 200 with no content is worse than one that errors, because every client treats it as success. Naming that yourself is the signal.

Interview surface this opens

Continuous batching vs static batching vs request queuing
Streaming transports: SSE vs WebSocket, and backpressure in each
What metric you autoscale an inference service on (not CPU)
NetworkPolicy and east-west isolation in a small cluster
Why an empty 200 is a worse failure than a 500