Designs by Duhart ← All work
Rank
16 of 28
Tier
Tier B  65 of 120

22 — Solomon: Evaluation Gates & Model Promotion

Solomon 4 of 6 · Solomon/solomon/evaluation, Solomon/reports/evaluation Stack: PyTorch, custom LAMBADA + HellaSwag loaders (from original sources, no HF) Result: all 9 gates green → checkpoint promoted to artifacts/checkpoints/serving.pt

Where this sits. The single most transferable piece of Solomon. It is not "I evaluated my model" — it is a promotion pipeline where evaluation has the authority to block a release, which is exactly what a bank or a platform team needs and rarely gets.

The nine gates

A checkpoint is promoted only if all nine pass. Not most. All.

# Gate Result What it protects against
1 Test perplexity 90.0 The model is worse than the last one
2 Bits per character 1.62 Tokenizer-independent sanity check
3 Token accuracy 24.9 % Next-token prediction actually works
4 Distinct-2 0.95 Output has lexical variety
5 Repetition rate 0.5 % The classic degenerate-loop failure
6 LAMBADA 1.3 % Long-range context is being used
7 HellaSwag 24.7 % Commonsense continuation
8 Serving p95 454 ms It is fast enough to ship
9 Throughput 116 tok/s It is cheap enough to ship

Two of those gates are not quality gates — and that is the design

Gates 8 and 9 are latency and throughput. A model that is accurate and too slow is not shippable, and discovering that after rollout is the expensive way to learn it. Putting serving performance inside the promotion gate means "good enough" is defined once, covering both axes, before anything reaches a cluster.

Diversity and degeneration gates

Distinct-2 (0.95) and repetition rate (0.5 %) catch the failure mode perplexity is blind to: a model can score well on likelihood while producing loops. Two independent metrics, one failure mode, both required.

Benchmarks loaded from original sources

LAMBADA and HellaSwag are loaded from their original distributions, not from a dataset hub. That is more work and it is the correct choice: it means the eval set is the one the paper describes, and the contamination check against the training corpus is meaningful.

What promotion actually does

  1. All 9 gates evaluate → green.
  2. A model card and an HTML report are written to reports/evaluation/20260610-024614/.
  3. The checkpoint is copied to artifacts/checkpoints/serving.pt.
  4. The container image is rebuilt against the promoted checkpoint.
  5. The rollout is performed.

The image on the cluster is therefore provably the checkpoint that passed, traceable back through the SHA256 manifest to the training run and the corpus.

Rollback is an image tag, not a retrain — the previous serving.pt and its manifest are retained.

Interview surface this opens