21 — Solomon: Training Loop & Experiment Tracking

Solomon 3 of 6 · Solomon/solomon/training, Solomon/reports/training Stack: PyTorch, MLflow (self-hosted), torch.compile, GitLab CI Result: 8,000 steps · 3.0 h wall · validation perplexity 40.0 · CPU only

Where this sits. The reproducibility artifact. Anyone can run a training script; this is the pipeline that makes a run reproducible, resumable, and auditable months later.

The loop

Noam schedule + label smoothing as the baseline, plus gradient accumulation, gradient clipping at 1.0, early stopping, checkpoint/resume, sample generations at every eval, and torch.compile.

Seeds pinned. Requirements locked. Dockerfile environment fixed.

CPU-only, and said so

No GPU on the host. Mixed precision is therefore N/A — and that is written down as a documented constraint rather than quietly omitted. The model was sized so a real 3-hour run fits the machine's actual spare RAM after the production services on the same box take theirs.

Naming your constraints is more credible than implying you had a cluster.

MLflow — self-hosted, both ways

File/SQLite backend for local runs; a server in-cluster for the deployed case (Solomon/k8s/helm/templates/mlflow.yaml). Same tracking API, two deployment shapes.

The run report — the artifact that matters

Every run emits a report to reports/training/<run-id>/ containing:

• Loss and learning-rate curves (HTML + Markdown)
• Full hyperparameter record
• The data card for the corpus used
• A SHA256 manifest

The manifest is the point. It is what lets you say, six months later, "the model serving traffic right now is exactly the checkpoint produced by this run, from this corpus, under these hyperparameters" — and prove it. That chain is what the promotion gate consumes and what the deployed image is built from.

Results

Known defects, recorded not hidden

From CODE_REVIEW.md: --resume restarts the shuffle. A resumed run does not continue the same data order it would have seen, which makes a resumed run subtly not identical to an uninterrupted one. It is listed as an open defect rather than left to be discovered.

Interview surface this opens

• The Noam warmup schedule: why warmup, and what happens without it
• Label smoothing and its effect on calibration vs raw accuracy
• Gradient accumulation as a substitute for batch size you cannot afford
• Checkpoint/resume correctness — including the data-order problem named above
• Experiment tracking and artifact lineage: run → manifest → image → traffic