Real-Time Aggregations in DynamoDB: How Deliveroo Computes Favorite Counts

Amazon DynamoDB is a fully managed, highly scalable NoSQL database optimized for predictable single-digit millisecond latency at any scale. It offers robust support for primary key lookups, batch operations, conditional writes, and TTL-based item expirations.

However, DynamoDB intentionally avoids complex relational operations: it lacks native aggregation support (such as SQL COUNT, SUM, or GROUP BY). Performing aggregations dynamically requires custom architectural patterns, especially for real-time customer-facing features.

This article examines how food delivery platforms like Deliveroo solve this problem to power features like listing the most-favorited restaurants in real time—without degrading API latency or ballooning cloud infrastructure costs.

---

The Use Case: Favorite Restaurants

In food delivery applications (such as Deliveroo, Swiggy, or Zomato), users can bookmark or “favorite” a restaurant. The system must fulfill two primary requirements:

1. Point Queries ($O(1)$): Check whether a specific user has marked a specific restaurant as a favorite, and allow them to toggle this state.
2. Ranked/Aggregated Lists: Render a leaderboard or discoverability feed displaying restaurants ordered by total favorite counts across different time windows (e.g., all-time, monthly, or daily).

+-------------------------------------------------------------------------+
|                              Deliveroo App                              |
|                                                                         |
|  [♥] Joe's Pizza      Total Favorites: 1,420                            |
|  [ ] Sushi Den        Total Favorites: 980                              |
|  [♥] Burger Joint     Total Favorites: 750                              |
+-------------------------------------------------------------------------+

---

Data Modeling for Point Operations

To handle point lookups efficiently, the underlying data store requires an $O(1)$ access pattern.

Primary Table: UserFavorites

• Partition Key (PK): Composite string restaurant_id#user_id (or restaurant_id as PK and user_id as Sort Key).
• Attributes: user_id, restaurant_id, created_at.

{
  "PK": "REST_101#USER_505",
  "restaurant_id": "REST_101",
  "user_id": "USER_505",
  "created_at": 1664784000
}

Operational Complexity

• Check Favorite Status: GetItem(PK="REST_101#USER_505") $\rightarrow O(1)$
• Favorite a Restaurant: PutItem(PK="REST_101#USER_505") $\rightarrow O(1)$
• Unfavorite a Restaurant: DeleteItem(PK="REST_101#USER_505") $\rightarrow O(1)$

While point operations are optimal, generating a leaderboard requires finding the sum of all favorites for every restaurant. Running a table-wide Scan operation on DynamoDB to count occurrences on-the-fly is computationally prohibitive, unacceptably slow, and exorbitantly expensive.

---

Evaluating Potential Aggregation Approaches

1. Scheduled Full-Table Scans (Batch Processing)

• Mechanism: A cron job runs once every 24 hours, scanning the entire UserFavorites table, aggregating counts in memory, and writing the summary back to an aggregate table.
• Drawbacks:
  • Stale Data: User actions take up to 24 hours to reflect on the platform.
  • High Read Capacity Unit (RCU) Costs: A full table scan consumes massive throughput, which scales linearly with the total number of historical favorites.

2. Synchronous Dual-Writes via the API

• Mechanism: When a user taps the favorite button, the application API server executes two operations synchronously:
  1. Insert into UserFavorites.
  2. Increment counter in RestaurantAggregates.
• Drawbacks:
  • Elevated Latency: The client experiences the cumulative latency of multiple database round trips.
  • Partial Failure & Inconsistency: If the second write fails due to network partitions or throttling, counts drift out of sync unless managed via distributed 2PC or multi-table ACID transactions, both of which increase latency and cost.

---

The Architecture: CDC via DynamoDB Streams and AWS Lambda

To combine low latency, strict data consistency, and near real-time updates, Deliveroo uses a Change Data Capture (CDC) pattern leveraging DynamoDB Streams and AWS Lambda.

flowchart LR
    Client([Mobile / Web Client])
    API[API Gateway / Service]
    FavoritesTable[(DynamoDB: UserFavorites)]
    Stream[[DynamoDB Stream]]
    LambdaFunction[AWS Lambda Aggregator]
    AggregateTable[(DynamoDB: RestaurantAggregates)]

    Client -->|1. Toggle Favorite| API
    API -->|2. PutItem / DeleteItem| FavoritesTable
    FavoritesTable -->|3. CDC Event| Stream
    Stream -->|4. Batched Events| LambdaFunction
    LambdaFunction -->|5. Atomic ADD| AggregateTable

Step-by-Step Flow

1. Client Action: A user favorites or unfavorites a restaurant.
2. Fast-Path API Write: The API performs an asynchronous/independent single-item write (PutItem or DeleteItem) to the UserFavorites table and immediately returns an HTTP 200 to the client.
3. Change Data Capture: DynamoDB Streams captures row-level changes (INSERT, MODIFY, REMOVE) in strict chronological order, behaving like a distributed transaction log (similar to MySQL binlogs or Kafka partitions).
4. Event Processing: AWS Lambda polls the stream and ingests batches of records.
5. Atomic Aggregation: Lambda filters for INSERT and REMOVE events, groups them, and updates the aggregated counts table.

---

Aggregations Table Design

To support aggregations across different time granularities without redesigning schemas later, the aggregate table uses a composite primary key structure.

Schema: RestaurantAggregates

• Partition Key (PK): restaurant_id (String)
• Sort Key (SK): time_window (String: e.g., ALL_TIME, YEAR#2022, MONTH#2022-10, DAY#2022-10-03)
• Attributes: favorite_count (Number), updated_at (Timestamp)

{
  "restaurant_id": "REST_101",
  "time_window": "ALL_TIME",
  "favorite_count": 1420,
  "updated_at": 1664784025
}

This schema allows querying a restaurant’s all-time score, monthly trend, or daily velocity simply by altering the sort key condition in a single Query API call.

---

Addressing Concurrency and Atomicity

When multiple users favorite the same restaurant simultaneously, AWS Lambda may spin up multiple concurrent executions. A naive Read-Modify-Write strategy introduces serious race conditions:

Execution A reads: count = 10
Execution B reads: count = 10
Execution A writes: count = 11
Execution B writes: count = 11  <-- Lost Update! Count should be 12

To resolve this, the system relies on two critical DynamoDB primitives:

1. Atomic Counter Updates (ADD / UpdateExpression)

Instead of reading the count, calculating count + 1, and issuing a PutItem, the Lambda function uses an UpdateItem with an atomic expression:

response = dynamodb_client.update_item(
    TableName='RestaurantAggregates',
    Key={
        'restaurant_id': {'S': restaurant_id},
        'time_window': {'S': 'ALL_TIME'}
    },
    UpdateExpression="ADD favorite_count :delta SET updated_at = :now",
    ExpressionAttributeValues={
        ':delta': {'N': '1'},       # Use -1 for REMOVE events
        ':now': {'N': str(current_timestamp)}
    }
)

DynamoDB serializes atomic updates at the partition replica level, guaranteeing mathematical correctness even under heavy write contention.

2. Transactional Batching (TransactWriteItems)

When a Lambda processes a batch of stream records, multiple aggregate records can be updated within a single transaction using TransactWriteItems (up to 100 actions per transaction). If an individual item failure occurs, the transaction ensures atomicity, preventing the aggregate view from drifting into a corrupted state.

---

Cost and Capacity Planning

A common reservation about serverless pipelines is runtime cost: Will triggering a Lambda function for every stream event become expensive?

Understanding access patterns and user behavior informs capacity planning:

• User Behavior: Unlike social media feeds (where users click “Like” on dozens of posts per session), users rarely favorite or unfavorite restaurants. It is an occasional, high-intent action.
• Write Volume: Deliveroo observed an average of approximately 7,000 write/delete events per day on favorites.
• Execution Metrics:
  • Batching Window: Lambda processes records in batches (e.g., 50–100 items per batch or buffered over small time windows), drastically reducing total invocations.
  • Execution Duration: ~25 ms per invocation.
  • Memory Footprint: 64 MB to 128 MB (the minimum allocation).

Cost Breakdown

• 7,000 executions / day $\approx$ 210,000 executions / month.
• Total compute time: $210,000 \times 0.025\text{s} = 5,250\text{ seconds of compute}$.
• At 128 MB RAM, total cost is less than $1.00 USD per month.

Running dedicated EC2 instances or container tasks (ECS/EKS) 24/7 to continuously poll a queue or process stream events would cost significantly more in baseline idle compute alone.

---

Key Architecture Trade-Offs

Attribute	Synchronous Dual-Write	Batch Job (Daily Scan)	CDC via Streams + Lambda
Consistency	Strong (if 2PC/Transactions)	Periodic Batch ($t+24\text{h}$)	Eventual (sub-second lag)
API Latency Impact	High (double write latency)	None ($0\text{ms}$)	None ($0\text{ms}$)
DynamoDB Cost	Moderate	High (Full table RCU scan)	Extremely Low
Failure Blast Radius	High (Aggregator failure blocks user)	Low	Isolated (Buffered in stream)

---

Architectural Takeaways

1. Respect DynamoDB’s Native Strengths: Use DynamoDB for fast $O(1)$ key-value lookups, but do not force it to execute ad-hoc aggregation queries.
2. Decouple Ingestion from Processing: Offload non-critical side effects (aggregations, notifications, indexing) from the primary API request path to an asynchronous stream.
3. Enforce Atomic Primitives: When updating pre-computed metrics asynchronously, always use atomic expressions (ADD) to prevent lost updates from concurrent executions.
4. Align Serverless with Traffic Profiles: For low-frequency, bursty write operations (like bookmarking or favoriting), serverless CDC pipelines provide near-zero operational overhead at a negligible cost.