# 3 — Match Service: Geo-Partitioned Distributed Ranking

**Rank 3 of 28** · Tier S · `app/MatchService`
**Stack:** Go, Cassandra / ScyllaDB (gocql + gocqlx), Redis + Redis Streams, Kafka (segmentio/kafka-go), WebSocket, Zap
**Status:** Running. Two production distributed-systems defects found, fixed, and documented in-source.

> **Why this ranks here.** This project contains the two best *stories* in the
> entire tree. Both are textbook distributed-systems failures that only appear
> under real load, in real clusters, with real consumer groups — the kind an
> interviewer cannot tell you were rehearsed, because the fix is committed with
> the reasoning above it.

---

## What it is

The candidate-generation and ranking pipeline: a geo-partitioned Cassandra
table of match cards, ranking pipelines that write scores back, a "sorting hat"
that indexes ranked cards into discovery decks, a fan-out worker, a Redis
Streams feed index, and a WebSocket hub for realtime match delivery.

```
                 ┌──────────── Kafka ───────────┐
                 ▼                              │
  match_engine → match_ranker → SortingHat → discovery decks
       │              │              │
       ▼              ▼              ▼
  Cassandra dating_global.match_cards        Redis Streams
  PARTITION (service_region, geohash4)       feed_index:writes
       │                                            │
       ▼                                            ▼
  match_fanout ──────► match_wss_hub ──────► clients
```

## War story #1 — the phantom partition

**`user_partition.go`**

`dating_global.match_cards` is partitioned by `(service_region, geohash4)`.
The ranking pipelines write scores back to it — but they iterate a **configured
list of regions**, and the region in a pipeline's loop is the *pool being
ranked*, not necessarily the pool the user's card actually lives in.

Cassandra `UPDATE` is an upsert. So writing with the loop region did not update
a row — **it silently created a partition**: one phantom `match_cards` row per
user per configured region, carrying only score columns and a null
`display_name`.

The sorting hat then indexed those phantoms into the discovery tables, putting
**nameless cards into real user decks.**

The fix is a partition-lookup layer: resolve the user's true partition before
writing, never infer it from the iteration context.

*Why this is a great story:* it is the single most common Cassandra footgun
(upsert-on-write with a wrong partition key), it produced a visible product
symptom, and the root cause is three inference steps away from that symptom.

## War story #2 — the leaking pending-entries list

**`stream_reclaim.go`**

A Redis consumer group moves a message into the consumer's **pending-entries
list (PEL)** on delivery and removes it only on `XACK`. Both feed consumers
`continue` past a processing failure without acknowledging — which is *correct*;
acknowledging a message you failed to process throws the data away.

But nothing ever came back for those entries. They accumulated. At the time of
the fix, `feed_index:writes` held **952 pending messages**, the oldest long
past any useful age.

The fix is an explicit reclaim path: `XAUTOCLAIM`-style recovery of aged
pending entries, so failed work is retried rather than orphaned.

*Why this is a great story:* it demonstrates you understand that
at-least-once delivery is not free — somebody has to own the retry, and if
nobody does, the queue silently becomes a graveyard.

## Other substance

- **`match_pair_gate.go`** + test — the mutual-match invariant, gated so a pair
  cannot be created twice from both sides concurrently.
- **`conversation_id.go`** + test — deterministic conversation identity so two
  services independently derive the same id.
- **`exclusion.go`** + test — seen/blocked exclusion sets at candidate-generation time.
- **`rank_encoding.go`**, **`ranking_vocab_test.go`** — score encoding stability.
- **`match_path_integration_test.go`** — end-to-end path coverage.
- **`match_redis_adapter.go`**, **`match_kafka_adapter.go`** — adapters isolating the transport from the engine.

## Scope

| Metric | Value |
|---|---|
| Go source | 33,638 lines, 51 files |
| Test files | 7, all targeting invariants rather than surface |
| Datastores | Cassandra/Scylla, Redis + Streams, Kafka |
| Partition key | `(service_region, geohash4)` |
| Documented production defects fixed | 2, with root-cause write-ups in-source |

## Interview surface this opens

- Cassandra partition key selection, and why upserts make a wrong key invisible
- Geohash-based geo-sharding: precision vs partition size vs cross-cell queries
- Redis Streams consumer groups vs Kafka consumer groups — PEL, XACK, XAUTOCLAIM
- Retry ownership in at-least-once systems, and dead-letter design
- Deterministic id derivation across services without coordination