# 5 — Billboard: Prototype → Production Hardening

**Rank 6 of 28** · Tier A · `app/workers/billboard_users`, `app/BillboardGateway`, `docs/billboard/`
**Stack:** TypeScript (Hono), Cloudflare Workers, D1, R2, KV, Durable Objects, Hyperdrive → Postgres, WASM (kissfft), GitHub CI
**Status:** ~60 commits over the hardening campaign. Upload live in production. CI gates enforced.

> **Why this ranks here.** This is the *production engineering* story, and it
> is the one a bank cares about most. It starts from a written audit that names
> everything broken, then closes the gaps one numbered commit at a time, with
> the CI gates that stop them reopening. Google calls this "reduce toil and
> make the fix stick." Banks call it "evidence."

---

## It starts with an audit, not code

`docs/billboard/` is a **17-chapter engineering manual** written before the
remediation began: executive summary, system architecture, iOS client, audio
engine, feature surfaces, recognition, edge, workers, data layer, an
**API contract diff**, upload current-state, security & auth, a **gap
register**, a **Spotify parity matrix (14 of 52 features)**, an upload v2
design, a remediation roadmap, and a task list.

The audit's findings were blunt:
- **Five distinct API hosts serve one feature mode.** No single gateway; the iOS client fans out directly to each.
- `billboard-feed-api` was a **mock data generator** — writes were discarded.
- `BillboardGateway` Express origin was a **scaffold with one health route**.
- Auth was **forgeable**.
- **No audio upload path existed at all.**

## What was then built

### Auth made real — `[F-2]`
RS256 JWT verification replacing forgeable auth, split into **four privilege
tiers**, with identity bound to the token rather than trusted from the body.
Applied to `global-media-worker` as well.

### The data layer made real — `[F-3]`, `[F-8]`, `[D-4]`
Nine missing D1 tables defined; column drift between code and schema bridged;
`billboard-feed-db` wired and its schema applied; ProfileService moved to
Postgres after Profile was 404ing against an empty D1 table.

### Resumable multipart upload — `[F-1]`, `[W-3]`, `[A-9]`
R2 multipart ingest with 10 MiB equal parts, an audio probe, and an
`IngestDO` Durable Object coordinating the session. TUS uploads **made
actually resumable**. `publishDraft` **made atomic**.

### Contract discipline in CI — `[M-7]`, `[F-10]`, `[CI]`, `[M-3]`
`openapi.yaml` promoted to the single source of truth with generation and
contract tests; an OpenAPI contract for `billboard-users` **enforced in CI**;
contract-diff and shadow-type gates; and the **first CI Billboard ever had** —
typecheck, tests, and a real end-to-end pass on every push.

### Moderation, auto-moderation and DMCA — `[S-3]`, `[auto-mod]`, `[dmca]`
Review-before-playback, report intake, queue, decision, takedown. Then a
**three-way automated classifier in front of the human queue**, with the
thresholds, the cost, and *what the classifier cannot catch* recorded. Then
DMCA: expeditious removal, notice, counter-notice, repeat-infringer policy.

### Audio fingerprinting — `[F-6]`
Built `kissfft.wasm` — the module `JamzFingerprintDO` had always loaded and
never actually had. A Shazam-style FFT constellation matcher running in a
Durable Object at the edge.

### Search and discovery — `[F-9]`, `[F-11]`, `[F-4]`
Typo-tolerant search; a home surface that leads with what you were doing; the
following feed (empty for **every** user) fixed along with two unbound
bindings; `/v1` contract aliases plus four endpoints that had never existed.

## The habit worth showing

Read the commit titles. They are written as **claims with evidence**:

> `[F-11] the following feed was empty for every user; fix it and the two unbound bindings`
> `[F-5] Remove the fake R2 types, and fix the three bugs they were hiding`
> `[F-6] Build kissfft.wasm — the module JamzFingerprintDO loads and never had`
> `[seed] fix truncated Stream uids — the seeded catalog would not have played`
> `docs: record the scope leak and adopt pathspec-scoped commits`

That last one is a process failure, caught and corrected in public. Interviewers
notice.

## Scope

| Metric | Value |
|---|---|
| `billboard-users` Worker | 10,581 lines TypeScript, 65 routes |
| Hardening commits | ~60, numbered against a written gap register |
| Audit manual | 17 chapters + session records |
| D1 tables added | 9 |
| Datastores | D1, R2, KV, Durable Objects, Postgres via Hyperdrive |
| CI gates | typecheck, tests, e2e, contract diff, shadow types |

## Interview surface this opens

- Edge compute constraints: no TCP sockets, and how that reshapes architecture
- Durable Objects as single-writer coordinators, and where they bottleneck
- Multipart/resumable upload protocol design and failure recovery
- Contract-first development and how you enforce it mechanically
- Building a gap register and burning it down without breaking live users