# 28 — Platform Operations, Deployment & CI/CD

**Additional project 4 of 4** · `deploy/`, `app/deploy/systemd`, `.github/`, `.gitlab-ci.yml`, `app/gateways/docker-compose.yml`, `*/RUNBOOK.md`
**Stack:** systemd (units + timers), Docker Compose, k3s/Helm, GitLab CI, GitHub Actions, wrangler, cloudflared

> **Where this sits.** The connective tissue. On its own it is not a project you
> lead with; in aggregate it is the evidence that everything else in this
> portfolio **actually runs somewhere** rather than existing as a repository.

---

## Infrastructure in the tree

```
deploy/
  kafka/      docker-compose + k8s manifests
  postgres/   redis/   coturn/   k8s/ingress.yaml   J8/
app/deploy/systemd/
  theone-backup.service     theone-backup.timer
  theone-health.service     theone-health.timer
  theone-restore-verify.service  theone-restore-verify.timer
app/gateways/docker-compose.yml   Makefile
app/GlobalChatService/deploy/     systemd unit, local bring-up, CQL schema fix
Solomon/k8s/helm/                 8 chart templates
Solomon/Dockerfile  Makefile  mk/  ci/
```

## The three timers are the tell

`theone-backup`, `theone-health`, and — the one that matters —
**`theone-restore-verify`.**

A backup you have never restored is a hypothesis. A **scheduled timer that
verifies the restore** is the difference between claiming you have backups and
knowing it. Most candidates have the first two units. The third is the one to
point at.

## Local reproducibility

- `app/gateways/docker-compose.yml` + `Makefile` — the whole three-gateway stack, Kafka and Cassandra included, on a laptop
- `app/GlobalChatService/deploy/local/` — `bringup.sh`, `docker-compose.yml`, `chat.local.env`, and a README
- `deploy/kafka`, `deploy/postgres`, `deploy/redis`, `deploy/coturn` — each dependency stood up independently

Being able to run a multi-datastore distributed system locally is an
underrated answer to "how do you develop against this?"

## CI, and the fact that it was new

Two systems: `.gitlab-ci.yml` (with **secret detection**) and `.github/`
workflows. The Billboard campaign's `[M-3]` commit is titled *"CI that runs
typecheck, tests and a real end-to-end pass on every push"* and `[CI]` is
*"Contract diff and shadow-type gates, and **the first CI Billboard has had**."*

Saying "this service had no CI and I gave it one" is a stronger claim than "we
had CI."

Gates enforced:

| Gate | What it catches |
|---|---|
| Typecheck | Drift between code and types |
| Unit tests | Regression |
| **Real end-to-end pass** | The thing that only breaks when wired together |
| **Contract diff** | Server drifting from `openapi.yaml` |
| **Shadow types** | Hand-written types shadowing generated ones |
| Secret detection | Credentials in a commit |

The shadow-type gate exists because of a real defect: `[F-5] Remove the fake R2
types, and fix the three bugs they were hiding.` Hand-written types that shadow
the real SDK's types make the compiler agree with code that cannot work. The
gate stops it recurring — **fix the bug, then close the door behind it.**

## Runbooks and edge deployment

`RUNBOOK.md` in GlobalChatService, `docs/deployment/DEPLOYMENT-GUIDE.md`,
`MASTER-CONTROL.md`, `k8s/INGRESS.md`, and per-session `DEPLOY_QUEUE.md` /
`BLOCKED.md` records. Edge deploys go through `wrangler`; origin traffic
arrives via `cloudflared` tunnels.

## Interview surface this opens

- Backup verification as a scheduled job, not a quarterly fire drill
- systemd timers vs cron for service-adjacent scheduled work
- Designing CI gates that encode a specific past defect
- Reproducing a multi-datastore distributed system locally
- Tunnels and edge-to-origin routing without exposing an origin