How Razorpay Scaled Their Notification System to Handle High-Throughput Fintech Workloads

In fintech and payment infrastructure, outbound notifications are not passive updates—they are mission-critical transactional events. When a customer initiates a payment through an infrastructure provider like Razorpay, multiple outbound communication channels must fire reliably and synchronously with the payment state:

• Customers require instantaneous payment confirmations and invoices via SMS or Email.
• Merchants expect real-time mobile push notifications and webhook calls to update internal ledger systems, trigger inventory allocations, or ping internal tools like Slack.

If notification latency spikes, user trust immediately deteriorates. A merchant who does not receive an alert assumes the transaction failed; a customer who is debited without immediate confirmation calls their bank. Maintaining strict delivery Service Level Agreements (SLAs) during regular operations and massive transaction surges (such as IPL matches or holiday flash sales) is a hard distributed systems problem.

Here is a technical deep dive into how Razorpay diagnosed scaling bottlenecks in their initial architecture and re-engineered their notification system to handle thousands of transactions per second (TPS) while preserving sub-two-second latency.

---

1. The Initial Architecture: Direct SQS and Synchronous Writes

The initial design followed a standard decoupled worker-queue model.

  [Payment Flow]
        │
        ▼
  ┌───────────┐         ┌───────────┐         ┌───────────┐
  │  API      │ ──────> │  Amazon   │ ──────> │  Worker / │
  │  Server   │ (Event) │  SQS      │         │  Executor │
  └───────────┘         └───────────┘         └─────┬─────┘
                                                    │
                         ┌──────────────────────────┴──────────────────────────┐
                         ▼                                                     ▼
              ┌─────────────────────┐                               ┌─────────────────────┐
              │ Dispatch Outbound   │                               │ Synchronous Write   │
              │ (SMS/Email/Webhook) │                               │ to MySQL DB         │
              └─────────────────────┘                               └──────────┬──────────┘
                                                                               │
                                                                    ┌──────────▼──────────┐
                                                                    │ Scheduler (Retries) │
                                                                    └─────────────────────┘

The Workflow

1. Event Ingestion: When a payment transaction succeeds, the API server publishes an event payload to an Amazon Simple Queue Service (SQS) FIFO/standard queue.
2. Consumption & Execution: A pool of background consumer workers polls messages from SQS and invokes an Executor service.
3. Outbound Dispatch: The Executor communicates with downstream gateway providers (SMS aggregators, Email SMTP providers, merchant webhook URLs, or mobile push gateways).
4. Synchronous Audit Logging: To guarantee message tracking and recovery, the Executor makes a synchronous write to a centralized MySQL database containing the delivery state, timestamp, and recipient details.
5. Retry Mechanism: An out-of-band Scheduler periodically scans the MySQL database for failed or timed-out notifications and re-enqueues them into SQS for a retry attempt.

Why Persist Every Notification?

In consumer tech, dropping a notification can occasionally be tolerated. In fintech, state reconciliation requires auditable delivery logs. If an external SMS gateway fails or a merchant’s webhook server experiences a temporary 5xx error, Razorpay must retry the event. Persisting the outbound dispatch state in an ACID-compliant database is essential for deterministic retry scheduling and auditability.

---

2. The Bottleneck: Scaling Past 1,000 TPS

As transaction volumes surpassed 1,000 TPS, the system encountered severe degradation:

• P99 latency doubled from 2 seconds to 4 seconds.
• Worker threads began timing out or stalling.
• Outbound delivery queues backed up across all merchants.

Profiling the system revealed three core failure modes:

Bottleneck A: Connection Pool and I/O Saturation in MySQL

Relational databases are bound by discrete limits: max concurrent connections and disk I/O operations per second (IOPS).

• When incoming traffic spiked, the auto-scaler added more worker instances (e.g., scaling from 100 to 500 workers).
• If MySQL is provisioned to comfortably handle 300 concurrent connection threads, running 500 workers causes connection pool starvation.
• Because the Executor wrote synchronously to MySQL on every single outbound dispatch, the workers spent most of their CPU cycles blocked waiting for database I/O confirmations.

Bottleneck B: Inelasticity of Database Scaling

Compute workers are horizontally elastic; relational databases are not. Scaling a database vertically requires provisioning for peak traffic that might only happen during rare annual events. Running an over-provisioned primary database continuously introduces excessive infrastructure costs. Furthermore, adding read replicas does not alleviate synchronous write contention.

Bottleneck C: Queue Starvation (No Priority Isolation)

In a unified queue, all notification types are treated identically. If a merchant triggers a batch marketing campaign (e.g., a promotional message sent to 10 million users on New Year’s Eve), those millions of bulk messages flood the SQS queue. Critical P0 transactional messages—such as OTPs or post-payment receipts—get queued behind millions of non-urgent marketing payloads, leading to massive SLA violations for core payment flows.

---

3. The Re-Architected Notification Engine

To address connection saturation, resource contention, and queue starvation, Razorpay introduced three fundamental engineering shifts: Priority Queues, Multi-dimensional Rate Limiting, and Asynchronous Persistence via Message Streaming.

                                        ┌──────────────────────────────┐
                                        │     Rate-Limited Queue       │
                                   ┌──> │   (Secondary / Low Pri)      │
                                   │    └──────────────────────────────┘
                                   │
  ┌──────────┐     ┌─────────────┐ │    ┌──────────────────────────────┐
  │   API    │ ──> │    Rate     │ ├──> │ Priority 0: Critical (Txn)   │
  │  Server  │     │   Limiter   │ │    └──────────────┬───────────────┘
  └──────────┘     └─────────────┘ │                   │
                                   │    ┌──────────────┴───────────────┐
                                   ├──> │ Priority 1: Default          │
                                   │    └──────────────┬───────────────┘
                                   │                   │
                                   │    ┌──────────────┴───────────────┐
                                   └──> │ Priority 2: Marketing / Bulk │
                                        └──────────────┬───────────────┘
                                                       │
                                                       ▼
                                            ┌─────────────────────┐
                                            │   Worker Pool &     │
                                            │     Executor        │
                                            └──────────┬──────────┘
                                                       │
                           ┌───────────────────────────┴───────────────────────────┐
                           ▼                                                       ▼
                ┌─────────────────────┐                                 ┌─────────────────────┐
                │  Outbound Provider  │                                 │ Amazon Kinesis Data │
                │ (SMS / Push / Webh) │                                 │       Stream        │
                └─────────────────────┘                                 └──────────┬──────────┘
                                                                                   │
                                                                                   ▼ (Batch Consumer)
                                                                        ┌─────────────────────┐
                                                                        │      MySQL DB       │
                                                                        │   (Audit/Retries)   │
                                                                        └──────────▲──────────┘
                                                                                   │
                                                                        ┌──────────┴──────────┐
                                                                        │ Scheduler (Retries) │
                                                                        └─────────────────────┘

Strategy 1: Prioritized Traffic Isolation

Notifications were segmented by criticality into separate physical queues:

• Priority 0 (P0): Mission-critical transactional flows (Payment confirmations, OTPs, immediate charge alerts).
• Priority 1 (P1): Standard operational notifications (Dashboard events, password updates, routine alerts).
• Priority 2 (P2): Low-priority batch/marketing communications (Promotional blasts, seasonal greetings).

Workers are allocated proportionately across these queues. Even if the P2 queue backs up with 50 million promotional messages, P0 workers remain completely isolated, processing transactional confirmations within milliseconds.

Strategy 2: Multi-Dimensional Rate Limiting

Priority isolation alone does not solve multi-tenant abuse. If Merchant A triggers a bulk broadcast of 2 million notifications within P2 (or attempts to misclassify them), other merchants sharing that queue would face starvation (noisy neighbor problem).

Razorpay implemented a centralized rate-limiting layer sitting before queue ingress:

• Limits are enforced across three dimensions: Per Customer $\times$ Per Queue $\times$ Per Event Type.
• Non-destructive spillover: Instead of rejecting requests with HTTP 429 Too Many Requests, events exceeding the configured threshold are redirected to a dedicated Rate-Limited Events Queue.
• Secondary worker pools consume from the rate-limited queue at a controlled, steady throughput. This prevents traffic bursts from impacting overall platform health while guaranteeing eventual delivery.

Strategy 3: Asynchronous Persistence via Streaming (Amazon Kinesis)

To eliminate the primary bottleneck—synchronous database writes during dispatch—Razorpay decoupled the Executor from MySQL using Amazon Kinesis Data Streams.

1. Decoupled Write Path: When an Executor successfully dispatches an SMS or webhook, it writes a dispatch record to a high-throughput Amazon Kinesis stream (a distributed append-only log similar to Apache Kafka).
2. Controlled Ingestion: Dedicated stream ingestion workers consume records from Kinesis in micro-batches and persist them to MySQL at a steady, sustainable rate.
3. Traffic Smoothing: During massive peak bursts, Kinesis acts as a shock absorber. While thousands of outbound messages are fired per second, the database writer controls the write pressure, staying well within MySQL’s IOPS limits and connection pool capacity.
4. Trade-off Analysis: This design introduces eventual consistency to the database state. However, because retry schedulers run periodically (e.g., checking for unconfirmed deliveries older than 1–2 hours), a sub-minute lag in persisting log records to MySQL has zero impact on retry semantics.

---

4. Observability and SLA Governance

Operating a multi-queue, asynchronous notification engine requires proactive observability to prevent silent consumer drift and unnoticed SLA degradation.

Key Metrics and Telemetry

1. Latency Bucketing: Average latency hides critical tail degradation. Razorpay monitors execution latency distributions in discrete SLA buckets: $\text{Ratio} = \frac{\text{Deliveries within } t \text{ seconds}}{\text{Total Deliveries}} \quad \text{for } t \in \{1\text{s}, 2\text{s}, 3\text{s}, 4\text{s}\}$
2. Queue Age and Lag: Monitoring raw message counts in SQS is insufficient; the primary metric is ApproximateAgeOfOldestMessage. A high message age indicates that workers are stalling or failing.
3. Downstream Gateway Health: Tracking provider error codes (e.g., telecom aggregator drops, webhook connection timeouts) to distinguish internal platform issues from external network failures.
4. Tenant Anomaly Detection: Real-time triggers detect aberrant message publishing behavior by specific merchants before it causes platform-wide backpressure.

---

5. Architectural Comparison: Before vs. After

Dimension	Initial Architecture	Re-Architected Architecture
Ingress Queuing	Single shared SQS queue	Multi-tier prioritized queues (P0, P1, P2)
Traffic Isolation	None (Batch marketing starves transactional events)	Strictly isolated worker pools per priority level
Abuse Handling	No rate limiting; bulk events overwhelm workers	Multi-dimensional rate limiter with spillover queues
Database Persistence	Synchronous writes by Executor to MySQL	Asynchronous streaming via Amazon Kinesis to batch DB writers
Worker Scaling	Constrained by MySQL connection pool limits	Horizontally elastic; unconstrained by database connection ceilings
Throughput & SLA	P99 latency degraded from 2s to 4s at >1k TPS	Consistent sub-2-second P99 latency under surge loads

---

6. Key Takeaways

1. Decouple Fast-Path Delivery from Audit Paths: In high-throughput messaging, synchronous database logging on the critical execution path creates hard limits on horizontal scalability. Streaming platforms (Kinesis/Kafka) act as write buffers to protect the underlying datastore.
2. Never Treat All Traffic Equally: Segregating queues by business criticality (P0/P1/P2) prevents low-value, high-volume workloads from compromising business-critical flows.
3. Shape Traffic Instead of Dropping It: Combining granular rate limits with spillover queues allows systems to absorb abusive bursts gracefully without degrading third-party developer experience with hard errors.
4. Tail Latency Is the Only True Metric: Aggregate metrics like mean latency obscure systemic failures. True SLA compliance requires strict visibility into tail distributions (P95/P99) through continuous latency bucketing.