# 20 — Solomon: Corpus, Tokenizer & Data Governance

**Solomon 2 of 6** · `Solomon/solomon/data`, `Solomon/solomon/tokenizer`, `Solomon/artifacts/corpus`
**Stack:** SentencePiece (BPE), custom extractors, no HuggingFace datasets

> **Where this sits.** The unglamorous half of ML that separates people who have
> shipped a model from people who have run a notebook. Licensing, cleaning,
> deduplication, **contamination checking**, and a written data card.

---

## The corpus

**WikiText-103 raw**, CC BY-SA — license recorded, not assumed. Cleaned,
exact-line deduplicated, and split:

| Split | Tokens |
|---|---|
| train | 130M |
| validation | 275K |
| test | 314K |

A documented fallback existed if the mirror was unreachable: Simple English
Wikipedia (CC BY-SA, own extractor) plus Project Gutenberg (public domain).
**Planning the fallback before needing it** is the operational habit worth
noting.

## The data card — `artifacts/corpus/DATA_CARD.md`

Provenance, license, cleaning steps, split sizes, and **contamination notes**.

### Why contamination checking is the headline

The corpus was checked against the evaluation prompt suites *and* against
LAMBADA and HellaSwag. Without that check, the
[9-gate promotion pipeline](22-solomon-evaluation-gates.md) is measuring
memorization and cheerfully promoting a bad model.

> Most candidates can describe perplexity. Far fewer will volunteer, unprompted,
> that their eval number is worthless if the eval set leaked into training — and
> fewer still have the check written down as an artifact.

## The tokenizer

**SentencePiece BPE, vocab 8192, byte fallback.** Trained here, not downloaded.

| Metric | Value |
|---|---|
| Compression | 4.02 chars/token |
| UNK tokens | **0** |

Recorded in `artifacts/tokenizer/tokenizer_stats.json`.

Byte fallback is the reason UNK is zero: any codepoint the BPE merges do not
cover decomposes into bytes rather than becoming an unknown. That is a design
choice with a measurable consequence, and the measurement is in the repo.

## Vocabulary sizing

8192 was chosen against a 6.84M-parameter model with **tied embedding and
output weights**. The embedding matrix is `vocab × d_model`; at vocab 32k it
would dominate the parameter budget and starve the layers. This is the
vocab/depth trade-off, made explicitly rather than by default.

## Interview surface this opens

- BPE vs WordPiece vs Unigram vs byte-level, and what byte fallback buys
- Chars-per-token as a compression metric, and why it drives your context budget
- Train/eval contamination: how you detect it and what you do when you find it
- Data licensing in ML pipelines, and why CC BY-SA is recorded in the artifact
- Vocabulary size as a parameter-budget decision, not a default