1 — Billboard Activity Graph

Rank 1 of 28 · Tier S · app/BillboardActivityService, app/workers/billboard_users Stack: Go 1.25, Kafka (segmentio/kafka-go), Cassandra (gocql), PostgreSQL, Cloudflare Workers Status: Built end-to-end and running. Commit 9d31b7d28.

Why this is ranked first. This is the system design interview. Not like it — it is the exact question ("design a social feed / activity stream") with the exact primitives (outbox, log, consumer groups, idempotency, hybrid fan-out) implemented in production Go against real Kafka and real Cassandra. Every follow-up an interviewer can ask has an answer already written down in the commit message.

The problem

Playlists, follows, feed, and posts are one social graph. The naive build gives each of them its own event pipeline, its own fan-out, and its own consistency story — four systems that drift apart. The directive was one activity log and one materializer.

Complication: the service that writes activity, billboard-users, is a Cloudflare Worker. Workers cannot open raw TCP sockets — not with a library, not with a shim. The Kafka wire protocol and CQL are both permanently unreachable from a Worker.

The architecture

billboard-users (Cloudflare Worker)
      │  writes billboard_feed_events in the SAME transaction as the post
      ▼
Postgres billboard_feed_events                    ← the activity log / outbox
      │  relay: claim → produce → stamp published_at
      ▼
Kafka  billboard.activity.v1   (6 partitions, keyed by actor)
      │
      ├─ group feed-materializer   → Cassandra billboard_feed.feed_timeline
      ├─ group notification-worker → Postgres billboard_notifications
      └─ group fanout              → realtime delivery

Four processes in one binary because they share a database pool and a Cassandra session — not because they are coupled. Each consumer owns its group, so splitting them across hosts later is a deployment change, not a rewrite.

The four decisions worth defending

1. Outbox, not direct produce

A Worker cannot enlist Kafka in a Postgres transaction. Producing directly leaves a window where either a post exists with no event, or an event survives a rolled-back post. Writing the event inside the post's transaction and relaying it afterwards closes that window.

2. At-least-once, with the cost paid explicitly at every sink

The relay produces before stamping published_at. A crash between the two re-delivers. That is deliberate: a lost event cannot be recovered, a duplicate can be absorbed. Each consumer absorbs it differently:

The idempotency key is the outbox row id — stable across re-delivery by construction.

3. Three groups, not three stages

A Cassandra outage must not stop notifications. A notification failure must not keep posts out of feeds. A single shared pipeline does exactly that. Three independent groups off one topic means each can fail, lag, and retry alone.

4. Thin events, not denormalized ones

billboard_feed_events carries ~40 denormalized display columns for the activity-feed UI. None of them are in the Kafka event. A consumer that needs the actor's avatar joins for it. Baking display fields into the event freezes every one of them at write time — and they are wrong the moment a user renames themselves.

Related design directives implemented

• Fractional / LexoRank ordering, not integer positions — reordering one track must not rewrite every row after it.
• System playlists are fixed-ID playlists. "Liked Songs" is a row with a reserved id, not a parallel table with duplicate code paths.
• One follows table discriminated by target_type, not one per followable kind.
• Hybrid fan-out — push below ~10k–50k followers, pull above. The threshold is a tuning constant, not a fork in the architecture.

What was verified before building

Directive 6 ("feed materialization is a Kafka consumer writing to Cassandra") required a Kafka and a Cassandra reachable from this host. Both were confirmed present before a line was written. Directive 2 was identified as a migration, not a new column — /v1/playlists/{id}/shuffle already rewrites an integer position holding live data.

Where a directive would have required infrastructure that was not present, it was written up as blocked rather than quietly substituted.

Scope

Interview surface this opens

• Exactly-once vs at-least-once, and why you never got to claim the former
• Outbox pattern vs CDC vs dual-write, and what each costs
• Partition key choice (actor id) and the hot-partition risk it carries
• Consumer group rebalancing and lag monitoring
• Push/pull fan-out crossover and where the celebrity problem bites
• LexoRank and why integer positions do not survive reordering at scale