4 — Solomon: LLM Training & Serving Platform

Rank 5 of 28 · Tier A · Solomon/ Stack: PyTorch (no HuggingFace), SentencePiece, FastAPI, MLflow, Docker, k3s + Helm + HPA, Prometheus, Go (retrieval tier), Kafka, GitLab CI Status: Trained, eval-gated, promoted, containerized, deployed to k3s, scaled to 3 replicas under load.

Why this ranks here. It is the ML-platform / infrastructure story, and it is complete end to end: corpus → tokenizer → training → gated evaluation → promotion → container → Kubernetes → autoscaling → streaming inference → clients. For Google (Cloud/ML infra) and any bank building internal model serving, this is directly on-target. It ranks below the Tier S three only because it is single-node training, not a distributed training story.

What was built

A decoder-only causal language model implemented in pure PyTorch from the Annotated Transformer reference, with every component either verbatim or with its deviation justified in writing, then carried all the way to a deployed, autoscaled inference service.

Explicitly no HuggingFace anywhere — model, tokenizer, data loading, and the LAMBADA/HellaSwag eval sets were all sourced and implemented directly.

Model

6.84M params · d_model 256 · 6 layers · 8 heads · d_ff 1024 · vocab 8192 · weight-tied · pre-norm · KV-cached decode proven exactly equal to full recompute · greedy + beam + temperature/top-k/top-p.

Data

WikiText-103 raw (CC BY-SA), cleaned, deduped: 130M train / 275K val / 314K test tokens. Data card with contamination checks against the eval suites. SentencePiece BPE 8192 with byte fallback: 4.02 chars/token, 0 UNK.

Training

Noam schedule + label smoothing, grad clipping, early stopping, checkpoint/resume, MLflow tracking, torch.compile. 8,000 steps, 3.0 h wall, val PPL 40.0. Report with loss/LR curves, hparams, and a SHA256 manifest.

Evaluation — a promotion gate, not a report

All 9 gates green, and only then is the checkpoint promoted to artifacts/checkpoints/serving.pt:

This gate is the part that matters for hiring. It is the difference between "I trained a model" and "I built the pipeline that decides whether a model is allowed into production."

Serving

FastAPI with SSE and WebSocket streaming, continuous batching, KV cache, Prometheus metrics, OpenAPI, and RS256 platform JWT verification (401 path verified).

Kubernetes

k3s + Helm: inference deployment with HPA, observed scaling to 3 replicas under load, MLflow in-cluster, manifests passing kubectl apply --dry-run=server, image rebuilt against the promoted checkpoint and rolled out. Chart templates cover a multi-phase topology (orchestrator, members, retrieval, search, network policy).

Retrieval tier — Go, 4,757 lines

Solomon/retrieval/ is a separate Go service: chunking, embedding, search, store, an agent loop with tool calling and a grounding guard (with its own test), RSS/fetcher ingestion, migrations, auth, and a kafkax package. This is where the ML platform becomes a distributed system.

Testing

Python 62/62 at 86 % coverage — including a causal-mask leak test, a positional-encoding formula test, an overfit-one-batch test, greedy determinism, and contract tests. Web vitest 11/11, Playwright 2/2 including a live e2e with no mocks. Load: 198 requests, 0 failures, p95 2.7 s against a 6 s SLO. GitLab CI wired.

Intellectual honesty on display

CODE_REVIEW.md and BLOCKERS.md list what is still wrong, unprompted: SSE per-token decode dropping inter-word spaces; the engine swallowing generation exceptions into an empty 200; disconnect-cancellation dead code leaking generations; make deploy sudo stripping KUBECONFIG; a --resume shuffle restart; a localStorage seed-clobber race.

Bring this file to the interview. A candidate who ships a written list of their own unfixed bugs, ranked, is signalling exactly the thing senior loops are trying to detect.

Scope

Interview surface this opens

• Transformer internals at the tensor level, defended component by component
• KV caching and how you prove it is equivalent to full recompute
• Continuous batching and the latency/throughput trade-off
• Model promotion gates, contamination checking, and reproducibility manifests
• HPA on a stateful-ish inference workload: what metric do you scale on?
• Why no HuggingFace — and what you learned by not having it