Scaling Financial Data Consistency: Razorpay’s Journey from Monolith to Microservices

In fintech and payment ecosystems, data consistency is paramount. A single missed transaction, phantom balance update, or partial write directly impacts customer trust and regulatory compliance. At the same time, maintaining high availability and low latency during extreme traffic surges—such as Indian Premier League (IPL) matches—makes traditional distributed locking and ACID transactions across systems virtually impossible.

This architecture breakdown covers the journey of Razorpay (India’s leading payments infrastructure provider) as explained by Arjun Tomar from their Payments Platform team: moving from a lean Day-0 monolith to a resilient, high-throughput microservices architecture while preserving strict financial correctness.

---

1. Day-0 Architecture: The Lean Monolith

Like many high-growth startups, Razorpay prioritized time-to-market and feature velocity in its initial phase. Engineering focused on shipping payment integrations quickly without prematurely optimizing for massive distributed scale.

The Initial Tech Stack

• Application Layer: PHP with Laravel running on Apache Web Server.
• Compute & Ingress: AWS Route 53 for DNS, AWS Application Load Balancers (ALBs), and EC2 instances.
• Persistence Layer: A single monolithic MySQL Amazon RDS instance running with ACID guarantees.

  [ Client / Merchant Checkout ]
               │
               ▼
        [ AWS Route 53 ]
               │
               ▼
            [ ALB ]
               │
       ┌───────┴───────┐
       ▼               ▼
   [ Apache ]      [ Apache ]
   [ PHP App]      [ PHP App]
       └───────┬───────┘
               │ (Direct DB Connections)
               ▼
      [ MySQL RDS Monolith ]

The Problem with Database-Level Constraints

To ensure relational correctness, the initial relational schema enforced strict foreign key (FK) constraints and unique constraints directly in the database engine.

While foreign keys guarantee referential integrity natively, they introduce severe bottlenecks under scale:

• Write Stalls & Lock Escalation: Every INSERT, UPDATE, or DELETE on a child table requires row-level or gap-level shared locks on the parent table. This introduces serialization bottlenecks.
• Cascading Latency: Background processes, transaction purges, and index updates degrade under high write throughput, driving query latency spikes.

Moving Constraints into Application Code

To recover database throughput, Razorpay systematically dropped foreign key constraints at the database level.

Instead of offloading referential integrity to the DB engine, they moved constraints into an application-level validation layer using a custom repository pattern and ORM extensions:

• Application-Level Referential Integrity: Before executing an insert into payments, custom repository managers verify that the corresponding order_id exists in the orders entity.
• Custom Validation Enforcements: Check constraints (e.g., column lengths, valid state transitions, non-negative monetary balances) were encoded directly inside application-level entity validators.
• Database Transactions: Mutations remained wrapped inside standard ACID transactions at the monolith database level, preserving transactional boundaries without database-enforced foreign key locks.

---

2. The Breaking Point: IPL 2019 and Connection Starvation

By 2019, Razorpay experienced rapid, hockey-stick volume growth. The catalyst that exposed the architectural limits of the PHP monolith was the Indian Premier League (IPL).

The “IPL Effect” and Second-Order Traffic Spikes

During major sporting events like the IPL, viewer behavior drives downstream consumer transactions:

• In the 30 minutes preceding a match, food delivery apps (Zomato, Swiggy) and fantasy sports/gaming platforms (Dream11, My11Circle) experience massive, synchronized traffic spikes.
• Razorpay, sitting as an upstream payment aggregator, absorbs the concentrated transaction burst of all these merchants simultaneously.

The Mechanics of Connection Exhaustion

Under this burst, the monolith faced connection limits:

1. Thread-per-Request Architecture: PHP processes ran on Apache without native connection pooling.
2. Slow Downstream Bank Gateways: When banking partners slowed down under load, transaction execution times lengthened.
3. Holding Database Connections Open: Because a PHP worker held an active MySQL connection throughout the full lifecycle of a web request, long-running bank API calls caused MySQL connections to sit idle yet blocked.
4. Database Saturation: MySQL hit its maximum active connection limits (max_connections). The application could no longer open connections, causing cascading failures and forcing Razorpay to throttle merchants.

[ Bursty Request ] ──▶ [ Apache/PHP Worker ] ──(Holds Open Conn)──▶ [ MySQL RDS ]
                             │                                            ▲
                             │ (Awaiting Bank Gateway)                    │
                             ▼                                    Connection Pool
                     [ External Bank API ]                           Exhausted!
                     (High Latency: 5-10s)                       (Writes Rejected)

This incident became the wake-up call to transition to a scalable microservices architecture.

---

3. Designing the Microservice Split

Moving a critical payment processor from a monolith to microservices cannot happen overnight. It was executed as a multi-quarter, multi-year migration, requiring zero downtime, high availability, and absolute data consistency.

Core Entities in the Monolith

The monolithic schema was anchored around five primary entities:

1. Payments: Payment attempts, states, metadata, and payment instrument details.
2. Orders: Commercial orders initiated by merchants.
3. Merchants: Merchant account configurations and credentials.
4. Transactions: Ledger movements representing debits and credits.
5. Balances: Current virtual account balances for merchants.

Service Decomposition

The monolith was split along domain and payment method boundaries:

• Order Service: Extracted to own order creation and merchant checkout lifecycle.
• Ledger Service: Absorbed transactions and balance tables to form an isolated double-entry accounting engine.
• Method-Specific Payment Services: Extracted into standalone microservices based on payment rails (e.g., Cards Service, Net Banking Service [NB+], UPI Service).
• Smart Routing Service: A dedicated intelligence layer running on MongoDB to analyze bank gateway health in real-time and dynamically route transactions to the gateway with the highest success rate.

                      [ Ingress / Edge Gateway ]
                                  │
                 ┌────────────────┴────────────────┐
                 ▼                                 ▼
          [ Order Service ]             [ Payment Router / Edge ]
                                                   │
                      ┌────────────────────────────┼────────────────────────────┐
                      ▼                            ▼                            ▼
             [ Card Microservice ]         [ NB+ Microservice ]        [ UPI Microservice ]
                      │                            │                            │
                      └────────────────────────────┬────────────────────────────┘
                                                   ▼
                                         [ Ledger Service ]

---

4. Safe Migration: Dual Writing and the Outbox Pattern

Migrating active live payment flows requires keeping old and new persistence stores synchronized without data loss.

Phase 1: Dual Writing with Transactional Outbox

Direct dual-writing to two disparate databases from application code introduces the dual-write problem: if the first write succeeds and the second fails, the data drifts into an inconsistent state. Implementing distributed transactions (like Two-Phase Commit / 2PC) was rejected due to latency and throughput overhead.

Razorpay resolved this using the Transactional Outbox Pattern:

1. When the monolith processed a transaction, it updated its primary relational table and inserted an event record into an outbox table within the exact same database transaction.
2. If the local database transaction failed, both the state mutation and the outbox record rolled back atomically.
3. An asynchronous worker or CDC connector polled the outbox table and published the mutation events to Apache Kafka.
4. Consumers on the microservice side consumed from Kafka and wrote the records into the new microservice database.

┌── Monolith DB Transaction ───────────────┐
│  1. INSERT INTO payments (status='paid') │
│  2. INSERT INTO outbox (payload='...')   │
└──────────────────────────────────────────┘
                      │ (Local Commit)
                      ▼
            [ Outbox Table ]
                      │
             (Tail Transaction Log / CDC)
                      ▼
               [ Apache Kafka ]
                      │
                      ▼
         [ Microservice Consumer ]
                      │
                      ▼
        [ Microservice Target DB ]

Phased Traffic Migration: “Splits”

To shift traffic systematically without a binary big-bang cutover, Razorpay developed an internal experiment platform called Splits:

• Traffic was routed based on granular criteria (e.g., merchant ID, payment method, percentage buckets).
• Traffic shifted incrementally: 0% → 1% → 5% → 25% → 100%.
• Once a service reached 100% traffic and completed validation runs, legacy monolith paths were deprecated.

---

5. Eventual Consistency in Financial Ledgers

In a monolithic architecture, a payment and its merchant balance update occur in a single ACID transaction block. In a microservices architecture, the Payment Service and the Ledger Service have separate databases.

To balance strict correctness with high throughput, Razorpay decoupled payment processing from balance crediting using Eventual Consistency.

Why Eventual Consistency Works for Ledgers

• Merchant settlements are not instantaneous; standard domestic bank payouts settle on a $T+1$ or $T+2$ business-day schedule.
• The internal ledger represents a virtual merchant balance. Recording this balance can safely tolerate a latency of a few seconds without violating commercial constraints, provided it is guaranteed to be processed eventually and strictly once (idempotently).

The Asynchronous CDC Pipeline Architecture

To power asynchronous ledger entries at scale:

1. Payment Processing: The payment microservice completes payment execution with an external gateway.
2. Local Commit: The payment state is written to the service’s primary table, and a corresponding ledger event is written to a local outbox table in a single local ACID transaction.
3. Debezium CDC: Change Data Capture (CDC) via Debezium monitors the database binlogs/transaction logs and pushes events to an Apache Kafka topic in real-time.
4. Ledger Ingestion: The Ledger Service consumes events from Kafka and writes debit/credit operations to its balance store.

[ Payment Service ] ──▶ [ Payment DB (Tx: Outbox + Payment) ]
                                     │
                              (DB Transaction Log)
                                     ▼
                          [ Debezium CDC Pipeline ]
                                     │
                                     ▼
                              [ Apache Kafka ]
                                     │
                                     ▼
                            [ Ledger Service ]
                                     │
                                     ▼
                           [ Ledger/Balance DB ]

The Two-Way Handshake Mechanism

Pushing data to Kafka alone is insufficient for financial integrity; networks partition, messages can drop, or consumers can lag. Razorpay implemented a Two-Way Handshake via CDC:

1. Forward Leg: Payment Service → Outbox → CDC → Kafka → Ledger Service writes balance.
2. Reverse Leg (Acknowledgment): Upon updating the ledger, the Ledger Service writes an acknowledgment record into its own outbox table.
3. Reverse CDC: A separate CDC pipeline captures the Ledger’s outbox and publishes the acknowledgment back to a dedicated Kafka acknowledgment topic.
4. Payment Service Reconciliation: The Payment Service consumes the acknowledgment and records the associated transaction_id or flips a is_ledger_synced = TRUE flag on the original payment record.

┌─────────────────┐                                  ┌────────────────┐
│ Payment Service │                                  │ Ledger Service │
└────────┬────────┘                                  └────────┬───────┘
         │ 1. Write Payment + Outbox                          │
         │    (Local ACID Transaction)                        │
         ▼                                                    │
   [ Payment DB ] ──(CDC)──▶ [ Kafka: payment-events ] ──────▶│ 2. Process Balance
                                                              │    Write Ack to Outbox
                                                              ▼
   [ Payment DB ] ◀── [ Kafka: ledger-acks ] ◀──(CDC)── [ Ledger DB ]
         │ 3. Consume Ack & Mark Payment Synced
         ▼

Fallback Reconciliation: The Sweeper Cron

To handle edge-case failures (e.g., CDC connection drops, poison-pill messages, prolonged consumer lag), a Sweeper Cron Job continuously scans the Payment database for payment records where the ledger acknowledgment has timed out:

• It queries for payments completed past a specific time threshold that lack an acknowledgment ID.
• It executes a direct synchronous REST/gRPC fallback call to the Ledger Service to force consistency.
• This dual async/sync fallback model guarantees zero lost ledger entries.

Handling Instant Settlements ($T+0$)

Certain businesses (such as mutual funds, stock brokers, or instant-payout merchants) require $T+0$ real-time balance settlement:

• Razorpay uses Priority Kafka Topics.
• The outbox message stores an event priority flag (P0 vs P1/P2).
• The CDC pipeline inspects the priority flag and routes P0 events to dedicated high-partition, high-consumer Kafka topics.
• While standard events replicate end-to-end in 2 to 3 seconds, P0 events are processed within 1 second.

---

6. Resilience, Idempotency, and Testing

Idempotency Guarantees

Because distributed pipelines guarantee at-least-once delivery, duplicate events inevitably occur (e.g., when Debezium or Kafka consumers restart from a previously committed offset).

• The Ledger Service enforces strict idempotent operations by indexing transactions using unique payment IDs.
• If an incoming event references an already-recorded payment transaction ID, the Ledger ignores the mutation and re-emits the acknowledgment.

Chaos and Failure Mode Testing (FMEA)

To validate that the system survives worst-case outages without ledger drift, the team subjected the pipeline to structured Failure Mode and Effects Analysis (FMEA) and manual chaos engineering:

• Severing Debezium connectors.
• Killing Kafka brokers and consumer clusters.
• Inducing deliberate Ledger Service outages to verify consumer lag handling and the Sweeper Cron fallback.

SLIT: Service-Level Integration Testing

To prevent regressions in inter-service communication as microservices proliferated, Razorpay built SLIT (Service Level Integration Testing):

• Unlike broad end-to-end test suites (which are brittle and slow), SLIT focuses strictly on upstream-to-downstream API and schema contracts.
• It validates contract schemas between producer and consumer services, ensuring that an upstream schema change in payment entities does not break the downstream Ledger or Settlement processing.

---

7. Key Architecture Takeaways

Challenge	Monolith Limitation	Microservices Resolution
Data Integrity	Enforced via DB foreign keys, leading to write stalls and locks.	Dropped DB foreign keys; referential integrity moved to application validation layers.
Traffic Bursts	PHP/Apache thread-per-request model saturated MySQL connections.	Decoupled method services (Cards, UPI, Net Banking) with independent connection pools.
Dual-Write Drift	Writing across split services risked partial failure.	Implemented Transactional Outbox Pattern with local ACID boundaries.
Ledger Consistency	Monolithic sync transactions provided ACID guarantees but throttled throughput.	Adopted Eventual Consistency using Debezium CDC, Kafka, and a Two-Way Handshake.
Edge-Case Drift	Silent failures if message queues stall.	Implemented Sweeper Cron Jobs as a synchronous fallback layer.
Low-Latency SLA	Uniform processing queues delayed critical settlements.	Dynamic priority routing via dedicated P0 Kafka Topics for instant ($T+0$) settlements.