Mortar — multi-region architecture

Status: planned design doc; first multi-region deploy on the roadmap (co-locating Singapore alongside cn-north / cn-south).

Mortar’s current architecture is single-region: one Postgres primary

• one OSS bucket + one FC pool, all in cn-hangzhou. The multi-region work introduces the operational pattern for running independent regional deploys that share a global DNS surface but isolate their data planes.

TL;DR

• 3 target regions — cn-north (Beijing), cn-south (Shenzhen), sg (Singapore for international).
• Independent Mortar deploy per region — each region runs its own full stack: Postgres + Redis + OSS + FC + Mortar binary. Nothing shared.
• Single global DNS routes users to nearest region. Route 53 geo-DNS for sg; 阿里云 DNS GTM for cn-*.
• Per-project region pin — MORTAR_PROJECT_DEFAULT_REGION or, per project, project.region. Data stays in-region for compliance.
• No cross-region replication for hot paths — Postgres logical replication across the 防火长城 is too slow; cache + database are per-region. Only OSS cross-region replication is enabled, for hot static assets.
• Cost: ~2× baseline per region (no shared infra). Justified by data residency requirements (PIPL for cn-*, customer requirements for sg).

Target regions

Future candidates (waiting on customer demand):

• us-west (AWS us-west-2)
• eu-central (AWS eu-central-1, for GDPR-pinned tenants)

Architecture

                       ┌──────────────────┐
                       │  Global DNS      │
                       │  (geo-routing)   │
                       └────────┬─────────┘
                ┌───────────────┼───────────────┐
                ▼               ▼               ▼
       ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
       │  cn-north   │  │  cn-south   │  │     sg      │
       │             │  │             │  │             │
       │  Mortar     │  │  Mortar     │  │  Mortar     │
       │  Postgres   │  │  Postgres   │  │  Postgres   │
       │  Redis      │  │  Redis      │  │  ElastiCache│
       │  OSS        │  │  OSS        │  │  S3         │
       │  FC         │  │  FC         │  │  Lambda     │
       └─────────────┘  └─────────────┘  └─────────────┘
              ↕                ↕                ↕
            (control-plane sync via /v1/_accounts/* API)

Each region is a complete, isolated deploy. The Mortar binary is the same; only the env-var-injected backends differ. Adding a region is a single env-config + helm install.

The shared control plane — billing, account management, audit-log aggregation — runs in one home region and exposes the /v1/_accounts/* API consumed by the mortar CLI. Single global Postgres, replicated to each region as read-only for the local Mortar to lookup tenant metadata (region pin, plan tier, credit balance) without a cross-region round-trip per request.

Data residency

Every Mortar project carries a region field (set at creation, immutable without a paid migration). All data for the project lives ONLY in that region’s Postgres / OSS / FC.

Configuration:

MORTAR_PROJECT_DEFAULT_REGION=cn-north  # what new projects default to
MORTAR_REGION=cn-north                  # what THIS Mortar instance serves

Requests for a project pinned to a different region than the Mortar instance receives them get a 307 redirect to the correct regional endpoint. The SDK transparently follows; the mortar CLI does too.

Why redirect, not proxy: proxying a cn-south read through a cn-north Mortar would cross the 防火长城 twice (once in, once out) and add ~100ms

• a compliance surface area we don’t want. Redirect lets clients hit the right region directly.

Replication strategy per primitive

Database

Independent per-region Postgres — no cross-region replication.

Postgres streaming replication across China-international boundaries incurs 200-400ms lag minimum (long fat network), which would break our RLS isolation model on writes (replica thinks the project doesn’t exist yet) and our credit-gate accuracy. Logical replication would also need network paths through 阿里云专线 to AWS Direct Connect — expensive + brittle.

Cross-region migrations are a paid one-shot operation (admin-only; tooling planned): mortar projects migrate --project=... --target-region=sg dumps the project’s data, ships through the home-region control plane, restores on the target region. Hours of downtime for the project; pre-flighted.

Storage

OSS cross-region replication enabled for hot objects — the user- facing mortar_files bucket replicates async to the other 阿里云 region for read-near-user. AWS S3 cross-region replication mirrors the same pattern for sg.

This is read-only fanout: writes always go to the project’s home region. Reads against a non-home region serve from the replicated bucket (~10ms faster than fetching cross-region).

Cache

Independent per-region Redis — cache is ephemeral; replicating it adds cost without value. Rate-limit counters are per-region anyway (the limit is “60 req/min from this user TO THIS region’s Mortar”, which is what we want).

Realtime pub/sub does NOT cross region. A subscriber in cn-north only sees publishes to cn-north’s Redis; a subscriber in sg only sees sg publishes. Acceptable for the initial multi-region rollout — real cross-region realtime needs a global pub/sub plane (planned).

Compute

Each region runs its own FC pool / Lambda pool. Function code is deployed independently per region (the mortar.compute.deploy() SDK call hits the project’s home region only).

Cold starts are per-region; warm pool is per-region. Cross-region compute would mean shipping the function code + invocation payload across the WAN — too slow + too expensive.

Network egress

Per-region egress accounting. Each region’s egress meter tracks its own bytes-out; the credit ledger rolls them up against the project’s single credit pool. Project tier caps apply globally (not per-region) — a Tiny tier project shares its 25 GB egress cap across whatever regions it spans.

Failure modes + RPO / RTO

Note that “full region outage” recovery is project-dependent: a paid migration (~hours) restores the project to a different region. Most tenants either accept the outage (cheaper) or pay the migration fee upfront for active-passive geo-redundancy (planned enterprise tier).

Cost

Single-region baseline today: ~¥30k/月 for production capacity at our current tenant count.

Three-region baseline (planned): ~¥80k/月 (≈ 2.7× single-region, not 3×, because the sg region is smaller — international demand is currently a fifth of cn-*).

The premium is justified by:

• PIPL compliance: 国内 tenants’ data legally must stay in 国内 Postgres — region pin enforces this.
• Latency: a Shenzhen user hitting cn-south instead of cn-north saves ~30ms round-trip.
• Customer requirements: 国企 + 政企 customers want to see the region tag in audit logs.

Acceptance criteria

• Helm chart parameterises every MORTAR_*_REGION knob for the target region.
• project.region column added; migration backfills existing projects to cn-north.
• DNS routing: Route 53 geo-DNS lives + cuts over 50% of sg traffic.
• Per-region health check + status page (status.mortar.cloud).
• mortar projects migrate command reviewed + tested against a fixture project.
• Cross-region request 307 redirect works end-to-end from the JS SDK.
• Audit-event aggregation surfaced through mortar audit list shows the region tag.