Understanding Storage-Compute Separation in Cloud-Native Databases: An Amazon Aurora Deep Dive (Part 1)

Amazon Aurora represents a fundamental re-evaluation of how relational databases should operate in a cloud environment. It’s not merely an incremental improvement but a ground-up redesign. This article, based on the introduction section of the Amazon Aurora research paper, explores the key problems with traditional databases, provides a glimpse into cloud-native architectures, and delves deeply into the concept of storage-compute separation—a paradigm common across almost all modern distributed databases.

The Rise of Distributed Cloud Databases

Distributed cloud databases have gained immense popularity primarily due to two critical offerings:

1. Resiliency: The ability of the system to maintain availability even if individual nodes fail. This includes how easily a failed node can be replaced without impacting service.
2. Scalability: The capacity of the database to handle increasing workloads, such as a surge in requests or queries, without requiring extensive manual operations or performance degradation.

To achieve these, almost all distributed databases adopt two fundamental architectural principles:

1. Decoupling Storage and Compute: Separating the processing logic from the data persistence layer.
2. Replicating Storage: Distributing multiple copies of data partitions across various nodes. This not only facilitates handling more read requests but also provides mechanisms for managing writes and ensuring data durability.

Prominent examples of distributed cloud databases include Amazon Aurora, CockroachDB, Google Spanner, and Amazon DynamoDB.

Deep Dive into Storage-Compute Separation

Traditional Database Architecture

Traditionally, when setting up a database, one would provision a virtual machine (e.g., an EC2 instance) and install a database system like MySQL or PostgreSQL directly on it. In this setup, both the storage (the disk where data resides) and the compute (the process that accepts requests and performs operations on the disk) are tightly coupled on the same physical or virtual node. This high coupling means that scaling one component often necessitates scaling the other, even if not strictly required.

The Concept of Separation

Storage-compute separation involves placing the storage layer on a network-attached storage system and the compute layer on a separate machine. This architecture intentionally introduces a network call to access storage, which might seem counterintuitive due to the inherent latency. However, the benefits derived from this separation are substantial.

Roles in a Separated Architecture:

• Compute Node: Responsible for accepting and parsing queries, creating query execution plans, deciding which storage nodes to access, performing joins, aggregations, and other arithmetic operations on fetched data, and finally responding to the client. Compute nodes are typically CPU-heavy.
• Storage Node: Primarily responsible for data replication, durability, persistence, and performing I/O operations as fast and efficiently as possible. The storage layer is often designed to be highly available and fault-tolerant.

Why Storage-Compute Separation?

The decision to decouple storage and compute is driven by several compelling advantages:

1. Cost Efficiency (Serverless Model):

   • Storage is generally much cheaper than compute resources (CPUs). In many cloud scenarios, storage costs can be considered almost negligible at certain scales.
   • By separating, cloud providers can bill users only for the compute power consumed when queries are actively fired. If no queries are executed, the compute cost can drop to zero, while only minimal storage costs are incurred.
   • This model enables serverless databases like Neon and PlanetScale, which charge based on query usage rather than continuous machine uptime. Without separation, a single node running both storage and compute would need to be active constantly, incurring continuous costs.

2. Simpler Operations and Infrastructure Management:

   • Node Replacement: If a compute node misbehaves, it can be easily replaced because compute nodes are typically stateless. Other compute nodes can seamlessly take over requests, ensuring availability.
   • Storage Node Recovery: If a storage node fails, data replication ensures that other nodes hold copies. A new node can be brought online, and data can be replicated to it, simplifying recovery.
   • Adding Read Replicas: Increasing the number of read replicas becomes straightforward. New compute nodes can be added to handle more read requests without affecting the storage layer.
   • Failover: The stateless nature of compute nodes and the replicated nature of storage simplify failover mechanisms.
   • Scaling: Both compute and storage can be scaled independently. If more processing power is needed, more compute nodes can be added. If more storage capacity is required, the storage layer can be expanded without impacting compute.

Trade-offs and Bottlenecks

While storage-compute separation offers significant benefits, it introduces new challenges and bottlenecks:

1. Network Bottleneck: The most prominent drawback is the reliance on the network for every data access. This can lead to:

   • Increased Data Transfer: Compute nodes often need to fetch large amounts of data from storage to perform operations like joins and aggregations, leading to substantial network I/O.
   • Network Bandwidth and TCP Connections: High query loads can saturate network bandwidth or exhaust the number of available TCP connections, impacting performance.

2. IOPS on Storage Nodes: The storage layer must be provisioned with very high IOPS (Input/Output Operations Per Second) capabilities to handle the aggregated demands from all compute nodes.

3. Fan-out Writes and Latency: When a write operation needs to be replicated across multiple storage nodes for durability and consistency, the response time of the query is bounded by the slowest component. If one replica is slow, the entire write operation’s latency increases, potentially impacting throughput, especially for synchronous operations.

4. Expensive Transactional Guarantees: Implementing strong transactional guarantees (like ACID properties or distributed commits) becomes significantly more complex and expensive in a distributed, separated environment:

   • Two-Phase Commit (2PC): Achieving atomicity across multiple storage nodes typically requires protocols like 2PC, which involve a locking phase and a commit phase. This coordination over the network is inherently slower.
   • Coordination Overhead: The need for distributed coordination adds latency and complexity.
   • Failure Intolerance: Distributed transactions are often more susceptible to failures. If a participant fails during the transaction, recovery and retry mechanisms can be complex and costly, potentially cascading errors back to the client.

In essence, while distributed setups offer operational simplicities, they make transactional guarantees more explicit and costly, requiring careful design and consideration.

Amazon Aurora’s Innovative Approach

Amazon Aurora addresses many of these challenges by leveraging a key mechanism: the redo log. While the deep dive into its implementation will be covered in subsequent sections of the paper, the introduction highlights several advantages Aurora achieves:

• Independent, Fault-Tolerant, Self-Healing Storage: Aurora’s storage layer is designed to be highly resilient, spanning multiple data centers and automatically recovering from failures.
• No Performance Variance: It aims to provide consistent performance regardless of the load or underlying failures in the compute, network, or storage tiers.
• Reduced Network IOPS: By primarily transferring redo logs (which are typically small) instead of full data pages, Aurora significantly reduces the amount of data transferred over the network. This leads to a substantial throughput improvement over traditional MySQL.
• High Throughput: Less work for the underlying nodes due to optimized I/O operations results in higher overall throughput.
• Continuous Backups: The continuous streaming of redo logs to storage nodes enables continuous backups, eliminating the need for expensive, one-time backup operations.
• Near-Instant Crash Recovery: With logs continuously streamed and applied, Aurora can achieve near-instant crash recovery without requiring traditional checkpointing processes.
• Unaffected Foreground Processing: Unlike traditional databases where operations like MySQL dump or PG dump can stall ongoing requests, Aurora’s asynchronous log streaming ensures that foreground processing is not affected by backup or recovery operations.

Aurora essentially takes a MySQL-compatible core and re-architects its storage layer using a redo-log-based approach to achieve its distributed, cloud-native capabilities. The subsequent sections of the Aurora paper delve into the specifics of how these mechanisms are implemented to deliver such robust performance and operational simplicity.