Realtime — cross-replica / cross-region broker evolution

Status: cross-replica fan-out design + ship doc, also a cross-region readiness check. This doc covers two phases of the problem — distinguishing what we must fix now from what the roadmap will face but we must leave room for today.

Mortar realtime goes through two independent but related scaling problems. Both are solved through the same realtime.Broker interface, but with different backend implementations.

What we have to solve now: cross-replica fan-out

Even within a single region, Mortar runs multiple replicas — N=3 Mortar processes behind one load balancer.

                   ┌─ Mortar replica A  ←── SSE client X
       LB ─────────┼─ Mortar replica B  ←── (publish happens here)
                   └─ Mortar replica C  ←── SSE client Y

Client X subscribes to replica A; a write request lands on replica B and triggers pg_notify; if replicas don’t share a broker, client X never receives that event.

Early implementation: InMemoryBroker (feature/realtime/inmemory.go). In-process channel fan-out. Correct only with a single process. In the multi-replica case, events get lost — this is the bug we must fix now.

Current fix: switch to RedisBroker (feature/realtime/redis.go), with cross-replica fan-out via Redis pub/sub. One replica publishes → Redis pub/sub → every replica subscribes → each replica fans out to its own SSE clients.

                                      ┌─ Replica A ─→ SSE client X
Postgres LISTEN ──→ Replica B ─→ Redis ─┼─ Replica B ─→ (local SSE clients)
                                      └─ Replica C ─→ SSE client Y

That’s the entirety of what ships now. Multi-replica correct within one region.

The problem on the roadmap: cross-region message bus

On the roadmap, when Mortar scales to multiple regions (cn-hangzhou + cn-shenzhen), the broker needs to fan out cross-region, not just cross-replica.

Why RedisBroker can’t be used cross-region as-is:

• Redis isn’t designed for cross-region use (pub/sub has no persistence, no cross-region replication guarantees)
• Cross-region Redis connection jitter → events lost
• Redis pub/sub has no retention — events during subscriber downtime are all lost

Planned fix: switch to a cross-region message bus (Aliyun NS or Kafka MirrorMaker); the cross-region hop goes through the bus, while intra-region replica fan-out still uses Redis.

Postgres (cn-hz)  ──┐
                    ├──→ Cross-region bus (Aliyun NS / Kafka)
Postgres (cn-sz)  ──┤              │
                    │     ┌────────┴────────┐
                    │     ▼                 ▼
                    │  cn-hz region      cn-sz region
                    │  Redis pub/sub     Redis pub/sub
                    │     │                 │
                    └─────┴──── Mortar replicas per region
                                   ↓
                            SSE clients (in that region)

Why this doc mentions cross-region today

Key design point: today’s broker interface must be swappable, so a later switch to a cross-region message bus lands with zero changes for callers — the realtime feature code stays the same, only the underlying Broker impl switches from RedisBroker to NSBroker.

That’s what “readiness” in the title means — today we actively avoid future pitfalls. The specific readiness checks live in § “readiness checks”.

Current architecture (before cross-replica fan-out)

Postgres LISTEN ──→ Mortar replica A (InMemory broker)
                          ↓
                     SSE clients (same replica)

Cross-replica clients don’t receive events from other replicas (fixed now). Cross-region clients don’t receive events from other regions (fixed later, on the roadmap).

After multi-region (roadmap)

Postgres (region 1) ──┐
                      ├──→ Cross-region message bus
                      │   (Aliyun MQ / Aliyun NS / Kafka)
Postgres (region 2) ──┤
                      │
                      ▼
              Per-region Mortar replicas (Redis pub/sub)
                          ↓
                     SSE clients (in that region)

Cross-region message-bus options:

Recommendation: Aliyun NS for mainland, Kafka for international (AWS regions). The final selection is on the roadmap.

Data sharding model

Every project has a home region (chosen at creation, immutable). The home region is the source of truth for that project’s data. Other regions:

• Read-only mirror: rows replicate asynchronously from the home region (planned)
• Realtime fan-out: home-region Postgres trigger → cross-region bus → every region’s broker → every region’s SSE clients

project X home: cn-hangzhou
  - INSERT into table T (home region)
    ↓ NOTIFY
  - cn-hangzhou Mortar publishes "realtime:proj-X:T"
    ↓ NS topic "mortar.cross.realtime"
  - cn-shenzhen Mortar subscribes + republishes locally
  - cn-shenzhen SSE clients receive event

Cross-region latency budgets:

Cross-continent + overseas isn’t suitable for realtime — switch to polling or webhooks.

readiness checks {#m2-readiness-checks}

We don’t ship multi-region now, but while writing code we avoid creating future traps:

Row 5 (sticky session) — today, when a client disconnects and reconnects to a different replica, it loses events (in-flight messages are locked in the previous replica’s channel). Current fix:

• Client SSE carries Last-Event-ID header; the replica replays history
• Broker adds a short retention window (30s) using Redis Streams instead of pub/sub

Row 6 — the realtime.Broker interface already exists; add an NSBroker placeholder now:

// Current scaffold; full impl later
type NSBroker struct {
    nsClient *aliyunns.Client
    topic    string
}

func (b *NSBroker) Subscribe(...) { /* TODO later */ }
func (b *NSBroker) Publish(...) { /* TODO later */ }

Cross-region cutover plan (roadmap)

The migration path for the cross-region rollout:

1. Add the NSBroker impl + env-var switch (MORTAR_REALTIME_BACKEND=ns)
2. Gradual rollout: enable NSBroker first in the cn-shenzhen staging cluster
3. Spin up the cross-region cluster (full Mortar deploy in cn-shenzhen)
4. Add the project home region field; new projects ask the user to choose
5. Old projects default home region to cn-hangzhou; migration is optional

Zero customer-perceived change — the SSE protocol shape stays the same from today; only the underlying bus changes.

Testing + simulation

We add a cross-region simulation test now:

# Single-machine: bring up 2 mortar + 1 redis + 1 fake NS (simulated by a single redis)
docker compose -f docker-compose.multi-region-sim.yml up
# Then run the test suite: client A connects to mortar-1, client B to mortar-2
# Publish on A, verify B receives

This simulation lands now and provides the regression baseline for the later real implementation.

Risks

• NS doesn’t support 1-byte messages: Aliyun NS minimum payload is 64 bytes — Mortar events either need padding or switch to Kafka
• No cross-region latency SLA: Aliyun has no official cross-region RTT guarantee; clients must accept “real-time ≠ real-time”

References

• monitoring-ai-employee — cross-region failure AI monitoring
• multi-region — overall multi-region design (roadmap)
• architecture § realtime feature