3 — Match Service: Geo-Partitioned Distributed Ranking

Rank 3 of 28 · Tier S · app/MatchService Stack: Go, Cassandra / ScyllaDB (gocql + gocqlx), Redis + Redis Streams, Kafka (segmentio/kafka-go), WebSocket, Zap Status: Running. Two production distributed-systems defects found, fixed, and documented in-source.

Why this ranks here. This project contains the two best stories in the entire tree. Both are textbook distributed-systems failures that only appear under real load, in real clusters, with real consumer groups — the kind an interviewer cannot tell you were rehearsed, because the fix is committed with the reasoning above it.

What it is

The candidate-generation and ranking pipeline: a geo-partitioned Cassandra table of match cards, ranking pipelines that write scores back, a "sorting hat" that indexes ranked cards into discovery decks, a fan-out worker, a Redis Streams feed index, and a WebSocket hub for realtime match delivery.

                 ┌──────────── Kafka ───────────┐
                 ▼                              │
  match_engine → match_ranker → SortingHat → discovery decks
       │              │              │
       ▼              ▼              ▼
  Cassandra dating_global.match_cards        Redis Streams
  PARTITION (service_region, geohash4)       feed_index:writes
       │                                            │
       ▼                                            ▼
  match_fanout ──────► match_wss_hub ──────► clients

War story #1 — the phantom partition

user_partition.go

dating_global.match_cards is partitioned by (service_region, geohash4). The ranking pipelines write scores back to it — but they iterate a configured list of regions, and the region in a pipeline's loop is the pool being ranked, not necessarily the pool the user's card actually lives in.

Cassandra UPDATE is an upsert. So writing with the loop region did not update a row — it silently created a partition: one phantom match_cards row per user per configured region, carrying only score columns and a null display_name.

The sorting hat then indexed those phantoms into the discovery tables, putting nameless cards into real user decks.

The fix is a partition-lookup layer: resolve the user's true partition before writing, never infer it from the iteration context.

Why this is a great story: it is the single most common Cassandra footgun (upsert-on-write with a wrong partition key), it produced a visible product symptom, and the root cause is three inference steps away from that symptom.

War story #2 — the leaking pending-entries list

stream_reclaim.go

A Redis consumer group moves a message into the consumer's pending-entries list (PEL) on delivery and removes it only on XACK. Both feed consumers continue past a processing failure without acknowledging — which is correct; acknowledging a message you failed to process throws the data away.

But nothing ever came back for those entries. They accumulated. At the time of the fix, feed_index:writes held 952 pending messages, the oldest long past any useful age.

The fix is an explicit reclaim path: XAUTOCLAIM-style recovery of aged pending entries, so failed work is retried rather than orphaned.

Why this is a great story: it demonstrates you understand that at-least-once delivery is not free — somebody has to own the retry, and if nobody does, the queue silently becomes a graveyard.

Other substance

match_pair_gate.go + test — the mutual-match invariant, gated so a pair cannot be created twice from both sides concurrently.
conversation_id.go + test — deterministic conversation identity so two services independently derive the same id.
exclusion.go + test — seen/blocked exclusion sets at candidate-generation time.
rank_encoding.go, ranking_vocab_test.go — score encoding stability.
match_path_integration_test.go — end-to-end path coverage.
match_redis_adapter.go, match_kafka_adapter.go — adapters isolating the transport from the engine.

Scope

Metric	Value
Go source	33,638 lines, 51 files
Test files	7, all targeting invariants rather than surface
Datastores	Cassandra/Scylla, Redis + Streams, Kafka
Partition key	`(service_region, geohash4)`
Documented production defects fixed	2, with root-cause write-ups in-source

Interview surface this opens

Cassandra partition key selection, and why upserts make a wrong key invisible
Geohash-based geo-sharding: precision vs partition size vs cross-cell queries
Redis Streams consumer groups vs Kafka consumer groups — PEL, XACK, XAUTOCLAIM
Retry ownership in at-least-once systems, and dead-letter design
Deterministic id derivation across services without coordination