How PayPal’s Braintree Solved the Thundering Herd Problem and Simplified Architecture

In distributed systems, seemingly simple problems can lead to cascading failures if not addressed correctly. This article dissects how PayPal’s Braintree team tackled the notorious “Thundering Herd Problem” and simultaneously refined their system architecture, offering valuable lessons in system design and simplicity.

Understanding the Thundering Herd Problem

The Thundering Herd Problem occurs when a large number of processes or requests, often after a failure, attempt to access a shared resource or service simultaneously. This synchronized surge of activity can overwhelm the resource, leading to further failures and a vicious cycle of retries and overloads.

The Cycle of Failure:

1. A service becomes overwhelmed and starts failing requests.
2. Clients retry these failed requests.
3. The retries add to the existing load, further overwhelming the service.
4. More requests fail, leading to more retries, creating a cascading failure.

A common initial thought for handling retries is exponential backoff, where clients wait for increasingly longer intervals (e.g., 1s, 2s, 4s, 8s) before retrying. While better than fixed-interval retries, exponential backoff alone is often insufficient, especially if many clients start retrying at roughly the same time, leading to synchronized retry waves. This was precisely the challenge PayPal’s Braintree faced.

PayPal’s Braintree: The Dispute Processing System

PayPal’s Braintree processes merchant disputes, which are highly irregular in traffic. To handle this, they initially designed an asynchronous processing flow:

Original Architecture (Problematic)

graph TD
    A[Merchant SDK] --> B(Dispute API);
    B --> C{SQS / Active Job};
    C --> D[Worker Node];
    D --> E(Processor Service);
    E --> F{Processor Service Database};
    F -- Cron Job --> G[Batching & SFTP];
    G --> H[Payment Processors];

Flow Breakdown:

1. Merchant SDK: Merchants use PayPal’s SDK to register a dispute, making an HTTP call to the Dispute API.
2. Dispute API: Receives the request and enqueues it into an SQS queue using Active Job (a background job processing framework similar to Sidekiq in Ruby or Celery in Python).
3. Worker Node: A worker picks up the message from the SQS queue.
4. Processor Service: The worker makes an HTTP call to the Processor Service.
   • The Processor Service does not immediately process the dispute.
   • Instead, it stores the dispute in its internal database.
   • Periodically (every few minutes or hours), a cron job wakes up.
   • This cron job retrieves recent submissions from the database, batches them into large zip files, and sends them via SFTP to external Payment Processors.

The Core Problem: Cascading Failures and DLQ Overload

This architecture, particularly the Processor Service, contributed to cascading failures:

• Job Queue Buildup: If the Processor Service was under load or failing, messages would remain in the SQS queue or be retried by workers.
• Synchronized Retries: Workers, upon failure to connect to or process with the Processor Service, would retry. If many workers failed simultaneously, their retries would converge, creating a synchronized load spike.
• Service Overload: The Processor Service would become overloaded, leading to connection failures and further retries.
• Dead Letter Queue (DLQ) Overflow: Messages that failed after multiple retries (e.g., three attempts) were moved to a Dead Letter Queue. DLQs are typically for messages requiring manual intervention. However, due to the Thundering Herd, the DLQ would fill up rapidly, making it impossible for humans to triage effectively, as the root cause was systemic overload, not individual message errors.

The Processor Service itself was identified as a key contributor to the problem due to its buffering and batching logic, which could have been handled differently.

The Root Cause and Solution 1: Introducing Jitter

The fundamental cause of the Thundering Herd Problem in this scenario was synchronized retries. When many clients retry at the same time, they create a “herd” that overwhelms the target service.

The solution is to desynchronize these retry attempts. This is achieved by adding a random delay to the exponential backoff, known as Jitter.

What is Jitter?

Jitter is a random variation added to the delay in retry attempts. Its purpose is to:

• Break Retry Convergence: Prevent multiple clients from retrying at precisely the same time.
• Scatter Retry Attempts: Distribute retries over a time window, reducing peak load.
• Increase Success Rate: By reducing peak load, the chances of individual retries succeeding increase.

How Jitter Works (Types):

Let base_delay be the exponential backoff delay (e.g., 1s, 2s, 4s).

1. Full Jitter: The retry delay is a random number between 0 and base_delay.

   • delay = random(0, base_delay)
   • This provides maximum desynchronization but can lead to very short delays, potentially still hitting the service too quickly.

2. Half Jitter (or Decorrelated Jitter concept): The retry delay is base_delay / 2 plus a random number between 0 and base_delay / 2.

   • delay = (base_delay / 2) + random(0, base_delay / 2)
   • This ensures a minimum wait time while still introducing randomness. The video describes it as “whatever your backup delay was that plus half of the difference between the back of delay,” which aligns with ensuring a minimum wait.

The core idea is to introduce enough randomness to scatter the retries, regardless of the specific jitter algorithm used. Even a simple random delay within a small window (e.g., 0-5 seconds) can significantly alleviate the problem.

Solution 2: Architectural Simplification

Beyond addressing the Thundering Herd with jitter, PayPal’s Braintree identified a critical architectural anti-pattern: the Processor Service. This service was an unnecessary abstraction.

The Anti-Pattern: Microservices for Everything

A common misconception in modern system design is that every distinct piece of logic requires its own microservice. This often leads to:

• Increased Complexity: More services mean more deployment, monitoring, networking, and inter-service communication overhead.
• Unnecessary Abstraction: Services that merely buffer data or orchestrate simple tasks can often be consolidated.
• Maintenance Burden: Each service adds to the operational load.

The Processor Service was a prime example. Its job was to accept HTTP requests, buffer them in a database, and then batch and send them via SFTP. This entire logic could be handled by the existing worker nodes.

Simplified Architecture

graph TD
    A[Merchant SDK] --> B(Dispute API);
    B --> C{SQS / Active Job};
    C --> D[Worker Node (Batches & SFTP)];
    D --> H[Payment Processors];

New Simplified Flow:

1. Merchant SDK calls Dispute API.
2. Dispute API enqueues the dispute into SQS via Active Job.
3. Worker Node:
   • Picks up the message from SQS.
   • Now directly handles the logic previously performed by the Processor Service:
     • Buffers the dispute internally (or in a shared, simpler storage if needed).
     • Batches disputes.
     • Sends batches via SFTP directly to Payment Processors.

By consolidating the Processor Service’s responsibilities into the worker nodes, Braintree eliminated an entire microservice, significantly simplifying their architecture.

Key Lessons Learned

This case study from PayPal’s Braintree offers two profound lessons for system architects and engineers:

1. Simple Solutions for Complex Problems: The Thundering Herd Problem, a seemingly complex distributed systems challenge, was effectively mitigated by a simple, yet powerful, technique: adding jitter to retry mechanisms. This desynchronized requests, reducing peak load and preventing cascading failures.
2. Embrace Simplicity, Avoid Unnecessary Abstraction: The Processor Service was an example of an unnecessary microservice. Not every piece of logic requires its own service. Architects should strive for the simplest possible solution, consolidating functionality where it makes sense. Simple systems are easier to build, maintain, monitor, and, crucially, scale. Complex systems are easy to build but hard to scale.