Compute-Compute Separation: How Rockset Achieves Real-Time Latency and Workload Isolation

Rockset is a cloud-native search and analytics database built to serve low-latency queries over streaming, real-time data. To deliver predictable sub-second queries across diverse client applications while continuously ingesting high-throughput streams, a database must solve two fundamental engineering challenges:

1. Horizontal Scalability: The ability to scale query processing and storage independently as data and query volumes grow.
2. Workload Isolation: Ensuring that heavy analytical queries, operational lookups, and continuous streaming ingestion do not contend for the same compute resources.

While cloud data warehouses pioneered compute-storage separation, this paradigm introduces query lag that breaks real-time freshness guarantees. Rockset introduces an evolution beyond this pattern: Compute-Compute Separation.

---

The Evolution of Database Architectures

To understand why compute-compute separation is necessary, we must trace how architectural patterns have evolved and identify where each breaks down under real-time requirements.

Traditional Monolithic DB
       │ (Resource contention, no isolation)
       ▼
Data Sharding
       │ (Compute contention per shard, rebalancing overhead)
       ▼
Read Replicas
       │ (Storage multiplication, ingest contention per replica)
       ▼
Compute-Storage Separation
       │ (Network hop overhead, stale local caches)
       ▼
Rockset Compute-Compute Separation

---

1. Traditional Monolithic Database (Single Node)

In standard relational systems (such as single-node PostgreSQL or MySQL):

• Mechanism: Writes (ingestion) and reads (queries) execute on the exact same node and share CPU, RAM, disk I/O, and buffer pools.
• Strengths: Strong consistency and immediate query availability (real-time visibility of writes).
• Failures at Scale:
  • Resource Contention: A sudden burst of data ingestion throttles query performance; conversely, complex analytical queries saturate CPU and memory, stalling ingestion pipelines.
  • Zero Workload Isolation: If Application A runs an unoptimized scan, Application B’s operational queries immediately degrade.

---

2. Sharding Across Multiple Nodes

Sharding divides the global dataset into distinct partitions (shards) distributed across multiple machines.

• Mechanism: Each node owns a slice of the dataset and handles both reads and writes for that subset.
• Failures at Scale:
  • Localized Compute Contention: If multiple applications query data residing on the same shard, that specific shard node faces severe CPU and memory saturation.
  • Inelastic Rebalancing: Adding a new node requires rehashing and migrating large volumes of data across the network, making dynamic horizontal auto-scaling slow and operationally risky.
  • Cross-Shard Overhead: Analytical queries spanning multiple shards incur cross-network merge penalties, causing high tail latencies.

---

3. Read Replicas

To achieve read isolation, systems introduce read replicas fed by replication streams (e.g., MySQL binary logs or PostgreSQL WAL shipping).

• Mechanism: The primary node handles writes, while read replicas asynchronously apply the log and handle queries. Different consumer applications can point to dedicated read replicas.
• Failures at Scale:
  • Ingestion Contention on Replicas: Each replica must still execute the ingest/write path to replay the log, meaning compute contention between query execution and write replay persists on the replica nodes.
  • Storage Multiplication: Storing full dataset duplicates on every replica becomes cost-prohibitive.
  • Slow Scaling Lifecycle: Provisioning a new read replica requires taking a snapshot, transferring gigabytes or terabytes over the network, applying backlogged logs, and warming up caches.

---

4. Compute-Storage Separation (Cloud Data Warehouses)

Cloud data warehouses (like Snowflake or BigQuery) decouple compute engines from the storage layer.

+-------------------------------------------------------------+
|                     Compute Layer                           |
|  [ Compute Cluster A ]           [ Compute Cluster B ]     |
+-------------------------------------------------------------+
                               │ Reads via Network
                               ▼
+-------------------------------------------------------------+
|                     Shared Storage                          |
|           (Distributed Object Store: S3 / GFS)              |
+-------------------------------------------------------------+

• Mechanism: Query execution nodes are stateless. The source of truth resides in a shared distributed file or object store (e.g., Amazon S3, Google Cloud Storage, HDFS). Compute nodes fetch file blocks over the network on demand.
• Strengths:
  • Infinite, independent horizontal scaling of compute nodes.
  • True workload isolation: Application A uses Cluster A; Application B uses Cluster B.
  • Single source of truth with no duplicated storage overhead.
• The Real-Time Dilemma:
  • Latency Penalties: Accessing files over remote storage on every query introduces high latency.
  • The Stale Cache Trade-off: Adding a local disk/memory cache to compute nodes reduces network hops, but breaks data freshness. When writes occur, updating or invalidating distributed caches across remote compute clusters introduces replication delay. As a result, data warehouses typically suffer from minutes to hours of ingestion lag, failing the requirements for real-time applications.

---

The Rockset Approach: Compute-Compute Separation

Rockset solves the latency vs. isolation trade-off by decoupling ingest compute from query compute while preserving sub-second data freshness.

Architectural Foundations: RocksDB-Cloud

Rockset builds upon RocksDB-Cloud, an open-source, cloud-optimized extension of Facebook’s embedded Key-Value store (RocksDB):

• RocksDB is an Log-Structured Merge-tree (LSM) storage engine.
• Writes and updates are appended to an in-memory buffer called the MemTable (and written to a Write-Ahead Log for durability).
• When a MemTable reaches capacity, it is frozen, flushed to local disk as an immutable SST (Sorted String Table) file, and subsequently uploaded to persistent cloud storage (Amazon S3).

flowchart LR
    Writes[Incoming Writes] --> MT[MemTable in RAM]
    MT -->|Flush| SST[Immutable SST Files]
    SST -->|Async Backup| S3[Persistent Cloud Storage S3]

---

The Compute-Compute Separation Topology

Instead of making compute nodes pull everything through object storage or wait for periodic batch file updates, Rockset divides compute into Ingest Compute and Query Compute, connected via a dual-path data synchronization model.

flowchart TB
    subgraph IngestPlane [Ingest Plane]
        IW[Ingest Compute Node] 
        MT[In-Memory MemTable]
        IW --> MT
    end

    subgraph ReplicationChannel [Low-Latency Path]
        MT -->|Async In-Memory Replication| CA_MT[Cluster A MemTable Mirror]
        MT -->|Async In-Memory Replication| CB_MT[Cluster B MemTable Mirror]
    end

    subgraph QueryPlane [Query Plane (Isolated Clusters)]
        subgraph ClusterA [Query Cluster A]
            CA_MT
            QN1[Query Node 1]
        end
        subgraph ClusterB [Query Cluster B]
            CB_MT
            QN2[Query Node 2]
        end
    end

    subgraph StorageHierarchy [Storage Hierarchy]
        SHS[(Shared Hot Storage / SSD Cache Layer)]
        S3[(Durable Object Store: Amazon S3)]
    end

    IW -->|Flush SSTs| SHS
    SHS -->|Tier Out / Backup| S3
    SHS -.->|Ultra-Fast Reads| QN1
    SHS -.->|Ultra-Fast Reads| QN2

1. The Real-Time Path: Streaming MemTable Replication

To bypass the latency of writing to S3, waiting for manifest updates, and invalidating caches, Rockset streams raw in-memory changes:

• As the Ingest Compute Node receives writes, updates, and deletes, it buffers them in its active MemTable.
• Instead of waiting for SST flushes, the ingest node asynchronously replicates raw MemTable mutations directly over the network to the MemTable mirrors of all active Query Compute Clusters.
• Because this involves streaming small, in-memory updates over memory-to-memory channels, the replication delay between ingestion and query visibility is virtually zero (sub-second / low-millisecond range).
• The query nodes merge in-flight MemTable data with underlying historical files at read time, guaranteeing strong read-your-writes semantics.

2. The Historical Path: Shared Hot Storage (SSD Tier)

For historical or flushed data, compute nodes cannot afford to make cold S3 API calls on every analytical join or filter:

• Flushed SST files are stored in a Shared Hot Storage Layer—a distributed, high-throughput file system backed by high-speed NVMe SSDs.
• This layer functions as a global, shared distributed cache in front of Amazon S3.
• S3 remains the authoritative, infinitely durable cold storage layer, but active compute nodes rarely touch it directly.

---

Performance and Operational Benefits

Architectural Vector	Traditional Data Warehouse	Rockset Compute-Compute Separation
Workload Isolation	Compute clusters isolated; writes centralized	Complete physical isolation between Ingest & Query clusters
Data Freshness (Lag)	Minutes to hours (batch or micro-batch commits)	Sub-second (streaming MemTable replication)
Cache Efficiency	Nodes manage local caches with invalidation lags	Shared Hot SSD Cache tier backed by immutable SSTs
Storage Scalability	Elastic object storage	Elastic object storage (S3) + hot SSD tier
Horizontal Scaling	Adding nodes requires network warm-up	Compute nodes spin up quickly by attaching to shared hot cache

High Cache-Hit Ratio

Because query nodes read immutable SSTs from the fast Shared Hot Storage tier:

• Rockset reports an operational cache-hit ratio of 99.997% on SSD reads.
• Remote Amazon S3 fetch requests occur so infrequently that object store access happens roughly once every several days per node under regular access distributions.
• SST immutability completely avoids the cache invalidation problem: once an SST is written and cached, it is never mutated, only unlinked during background compaction.

True Horizontal Scalability and Isolation

• Isolated Compute Units: An organization can provision one compute cluster exclusively for critical customer-facing API queries, a second cluster for ad-hoc internal business intelligence, and a separate ingest cluster for streaming pipelines (e.g., Apache Kafka).
• Zero Resource Contention: A massive scan run by data analysts on Cluster B has zero CPU, memory, or thread-pool impact on Cluster A, and cannot throttle ingestion on the ingest node.
• Fast Elastic Scaling: Query clusters are essentially stateless query engines with an attached local/shared SSD tier; scaling out a cluster does not require resharding or heavy data balance operations.

---

Summary

Rockset’s Compute-Compute Separation combines the isolation and elasticity of decoupled storage systems with the low data latency of operational databases:

1. Ingest Compute Nodes handle document parsing, indexing, and buffering.
2. In-Memory MemTable Replication bypasses disk and object storage bottlenecks, streaming fresh changes to query nodes in near real time.
3. Shared Hot SSD Storage caches immutable RocksDB SST files to deliver a 99.997% cache-hit ratio, bypassing S3 network latency.
4. Query Compute Clusters remain isolated, preventing noisy-neighbor contention across distinct enterprise workloads.