# 23 — Solomon: Inference Serving, Kubernetes & Autoscaling

**Solomon 5 of 6** · `Solomon/solomon/serving`, `Solomon/genengine/serving`, `Solomon/k8s`
**Stack:** FastAPI, SSE + WebSocket streaming, continuous batching, KV cache, Prometheus, Docker, k3s, Helm, HPA
**Status:** Live in-cluster. HPA observed scaling to 3 replicas under load.

> **Where this sits.** This is where the ML project becomes an infrastructure
> project — and the part most directly aligned with a Google Cloud / platform
> engineering loop.

---

## The serving layer

**FastAPI** exposing:

- **SSE streaming** and **WebSocket streaming** — two transports, one generation engine
- **Continuous batching** — requests join an in-flight batch rather than waiting for a batch window
- **KV cache** — proven exactly equal to full recompute (see [the model](19-solomon-transformer-model.md))
- **Prometheus metrics** and a generated **OpenAPI** surface
- **RS256 platform JWT** verification — the same asymmetric-signing model as
  [the Auth platform](09-auth-identity-platform.md), with the **401 path
  verified**, not assumed

### Why continuous batching is the interesting choice

Static batching makes the first request in a window wait for the window to
close. Continuous batching lets a request join a batch already decoding and
lets finished sequences leave without stalling the rest. It is the difference
between throughput that looks good on a benchmark and latency that is
acceptable to a user — and it is why **p95 454 ms and 116 tok/s could both be
promotion gates** rather than a trade-off.

## Kubernetes

**k3s + Helm.** Chart templates cover a multi-phase topology:

```
k8s/helm/templates/
  inference.yaml              the serving deployment + HPA
  mlflow.yaml                 in-cluster tracking
  phase2-orchestrator.yaml    request orchestration
  phase2-members.yaml         ensemble members
  phase2-retrieval.yaml       the Go retrieval tier
  phase2-search.yaml          search
  phase2-networkpolicy.yaml   east-west traffic restriction
  phase3-gend.yaml            generation daemon
  phase3-host-members.yaml    host-resident members
  secret.yaml
```

- Manifests pass **`kubectl apply --dry-run=server`** — validated against the
  live API server, not just parsed.
- The image was **rebuilt with the promoted checkpoint and rolled out**, so the
  running pods are provably the artifact that passed the gates.
- A **NetworkPolicy** restricts east-west traffic rather than leaving the
  namespace flat.

## Load behavior

| Metric | Result |
|---|---|
| Requests | 198 |
| Failures | **0** |
| p95 latency | 2.7 s |
| SLO | 6 s |
| HPA | scaled to 3 replicas under load |

## The blockers, written down

`k8s/INGRESS.md` records what could not be done from this host: the Cloudflare
Tunnel hostnames are dashboard-managed, so
`the public hostname → the in-cluster service` and
`the API hostname → the in-cluster service` are **manual steps**, documented
as such. `BLOCKERS.md` records that Phase 2 must restart its general-member to
load the promoted weights.

> Separating "done" from "blocked on something outside this host" — and naming
> the exact hostname and port — is the operational writing habit that shows up
> again in the [gap registers](12-technical-documentation.md).

## Known defects, recorded not hidden

From `CODE_REVIEW.md`:
1. **SSE per-token decode drops inter-word spaces** (server *and* WebSocket path).
2. **The engine swallows generation exceptions** → an empty 200 followed by `[DONE]`.
3. **Disconnect-cancellation is dead code** → leaked generations when a client hangs up.
4. `make deploy` under sudo **strips `KUBECONFIG`**.
5. A prompt-trim edge case at `max_tokens ≥ max_len − 1`.

Defect 2 is the one to discuss: a streaming endpoint that returns 200 with no
content is *worse* than one that errors, because every client treats it as
success. Naming that yourself is the signal.

## Interview surface this opens

- Continuous batching vs static batching vs request queuing
- Streaming transports: SSE vs WebSocket, and backpressure in each
- What metric you autoscale an inference service on (not CPU)
- NetworkPolicy and east-west isolation in a small cluster
- Why an empty 200 is a worse failure than a 500