Machine-translated draft — terminology + flow still being reviewed.

Mortar — multi-region architecture

Status: planned design doc; first multi-region deploy on the roadmap (co-locating Singapore alongside cn-north / cn-south).

Mortar’s current architecture is single-region: one Postgres primary

  • one OSS bucket + one FC pool, all in cn-hangzhou. The multi-region work introduces the operational pattern for running independent regional deploys that share a global DNS surface but isolate their data planes.

TL;DR

  • 3 target regions — cn-north (Beijing), cn-south (Shenzhen), sg (Singapore for international).
  • Independent Mortar deploy per region — each region runs its own full stack: Postgres + Redis + OSS + FC + Mortar binary. Nothing shared.
  • Single global DNS routes users to the nearest region. Route 53 geo-DNS for sg; Aliyun DNS GTM for cn-*.
  • Per-project region pin — MORTAR_PROJECT_DEFAULT_REGION or, per project, project.region. Data stays in-region for compliance.
  • No cross-region replication for hot paths — Postgres logical replication across the Great Firewall is too slow; cache + database are per-region. Only OSS cross-region replication is enabled, for hot static assets.
  • Cost: ~2× baseline per region (no shared infra). Justified by data residency requirements (PIPL for cn-*, customer requirements for sg).

Target regions

Region tagProvider regionPrimary use case
cn-northAliyun cn-beijing-hNorthern China users / state-owned enterprise customers
cn-southAliyun cn-shenzhen-aSouthern China + HK users
sgAWS ap-southeast-1Singapore + ASEAN; international fallback

Future candidates (waiting on customer demand):

  • us-west (AWS us-west-2)
  • eu-central (AWS eu-central-1, for GDPR-pinned tenants)

Architecture

                       ┌──────────────────┐
                       │  Global DNS      │
                       │  (geo-routing)   │
                       └────────┬─────────┘
                ┌───────────────┼───────────────┐
                ▼               ▼               ▼
       ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
       │  cn-north   │  │  cn-south   │  │     sg      │
       │             │  │             │  │             │
       │  Mortar     │  │  Mortar     │  │  Mortar     │
       │  Postgres   │  │  Postgres   │  │  Postgres   │
       │  Redis      │  │  Redis      │  │  ElastiCache│
       │  OSS        │  │  OSS        │  │  S3         │
       │  FC         │  │  FC         │  │  Lambda     │
       └─────────────┘  └─────────────┘  └─────────────┘
              ↕                ↕                ↕
                (control-plane sync via Mortar Center)

Each region is a complete, isolated deploy. The Mortar binary is the same; only the env-var-injected backends differ. Adding a region is a single env-config + helm install.

Mortar Center is the shared control plane — billing, account management, audit-log aggregation. A single global Postgres replicates to each region as read-only so the local Mortar can look up tenant metadata (region pin, plan tier, credit balance) without a cross-region round-trip per request.

Data residency

Every Mortar project carries a region field (set at creation, immutable without a paid migration). All data for the project lives ONLY in that region’s Postgres / OSS / FC.

Configuration:

MORTAR_PROJECT_DEFAULT_REGION=cn-north  # what new projects default to
MORTAR_REGION=cn-north                  # what THIS Mortar instance serves

Requests for a project pinned to a different region than the Mortar instance receives them get a 307 redirect to the correct regional endpoint. The SDK transparently follows; the dashboard does too.

Why redirect, not proxy: proxying a cn-south read through a cn-north Mortar would cross the Great Firewall twice (once in, once out) and add ~100 ms + a compliance surface area we don’t want. Redirect lets clients hit the right region directly.

Replication strategy per primitive

Database

Independent per-region Postgres — no cross-region replication.

Postgres streaming replication across China–international boundaries incurs 200–400 ms lag minimum (long-fat network), which would break our RLS isolation model on writes (replica thinks the project doesn’t exist yet) and our credit-gate accuracy. Logical replication would also need network paths through Aliyun Express Connect to AWS Direct Connect — expensive + brittle.

Cross-region migrations are a paid one-shot operation (admin-only; tooling planned): mortar-migrate --project=... --target-region=sg dumps the project’s data, ships it through Mortar Center, restores on the target region. Hours of downtime for the project; pre-flighted.

Storage

OSS cross-region replication enabled for hot objects — the user-facing mortar_files bucket replicates async to the other Aliyun region for read-near-user. AWS S3 cross-region replication mirrors the same pattern for sg.

This is read-only fanout: writes always go to the project’s home region. Reads against a non-home region serve from the replicated bucket (~10 ms faster than fetching cross-region).

Cache

Independent per-region Redis — cache is ephemeral; replicating it adds cost without value. Rate-limit counters are per-region anyway (the limit is “60 req/min from this user TO THIS region’s Mortar”, which is what we want).

Realtime pub/sub does NOT cross region. A subscriber in cn-north only sees publishes to cn-north’s Redis; a subscriber in sg only sees sg publishes. Acceptable for the initial multi-region rollout — real cross-region realtime needs a global pub/sub plane (planned).

Compute

Each region runs its own FC pool / Lambda pool. Function code is deployed independently per region (the mortar.compute.deploy() SDK call hits the project’s home region only).

Cold starts are per-region; warm pool is per-region. Cross-region compute would mean shipping the function code + invocation payload across the WAN — too slow + too expensive.

Network egress

Per-region egress accounting. Each region’s egress meter tracks its own bytes-out; the credit ledger rolls them up against the project’s single credit pool. Project tier caps apply globally (not per-region) — a Tiny tier project shares its 25 GB egress cap across whatever regions it spans.

Failure modes + RPO / RTO

ScenarioRPORTONotes
Single Mortar binary crash030 ssystemd / k8s restart
Single AZ failure within region02 minRDS Multi-AZ failover
Full region outagevarieshours+Project pinned to that region is unavailable
Great Firewall disruption (cn ↔ sg)0durationCross-border requests degrade; in-region traffic unaffected
Mortar Center outage0minutesPer-region operation continues with cached metadata

Note that “full region outage” recovery is project-dependent: a paid migration (~hours) restores the project to a different region. Most tenants either accept the outage (cheaper) or pay the migration fee upfront for active-passive geo-redundancy (planned enterprise tier).

Cost

Single-region baseline today: ~¥30k/mo for production capacity at our current tenant count.

Three-region baseline (planned): ~¥80k/mo (≈ 2.7× single-region, not 3×, because the sg region is smaller — international demand is currently a fifth of cn-*).

The premium is justified by:

  • PIPL compliance: mainland tenants’ data legally must stay in mainland Postgres — region pin enforces this.
  • Latency: a Shenzhen user hitting cn-south instead of cn-north saves ~30 ms round-trip.
  • Customer requirements: state-owned enterprise + government / enterprise customers want to see the region tag in audit logs.

Acceptance criteria

  • Helm chart parameterizes every MORTAR_*_REGION knob for the target region.
  • project.region column added; migration backfills existing projects to cn-north.
  • DNS routing: Route 53 geo-DNS lives + cuts over 50% of sg traffic.
  • Per-region health check + status page (status.mortar.cloud).
  • mortar-migrate tool reviewed + tested against a fixture project.
  • Cross-region request 307 redirect works end-to-end from the JS SDK.
  • Audit-event aggregation in Mortar Center shows the region tag.

Edit this page on GitHub →