Architecture Deep-Dive of a Real-Time Analytics Database: Rockset Internals

Modern applications often demand low-latency, sub-second analytical queries over high-velocity, real-time data streams. Traditional transactional databases (OLTP) crumble under high-throughput analytical scans, while standard data warehouses (OLAP) often involve heavy ETL batch windows that introduce minutes or hours of data staleness.

Rockset was designed specifically to bridge this gap as a cloud-native real-time search and analytics database. It enables low-latency SQL queries over streaming and semi-structured data without requiring upfront schema definitions.

In this architectural deep dive, we break down Rockset’s core design principles, its Aggregator-Leaf-Tailer (ALT) architecture, and how it achieves independent scaling, fault tolerance, and high write throughput.

---

Where Rockset Fits in the Modern Data Stack

Rockset functions as a serving layer optimized for operational analytics and search. Because it is a fully managed cloud service, it connects natively to various operational and analytical data sources:

• Transactional Databases: MongoDB, MySQL, PostgreSQL, DynamoDB (via Change Data Capture / CDC).
• Event Streams: Apache Kafka, AWS Kinesis.
• Data Lakes & Warehouses: Amazon S3, Google Cloud Storage, Snowflake.

Once ingested, data becomes queryable within milliseconds, powering use cases such as:

• E-commerce: Real-time fraud detection, spam mitigation, and anomaly detection.
• Gaming: High-throughput live leaderboards and player analytics.
• SaaS Applications: Customer-facing operational analytics dashboards.
• Content & Media: Vector search and instant real-time personalization.

[ OLTP Databases / Event Streams / Data Lakes ]
                     │
                     ▼ (CDC / Push / Pull)
            ┌─────────────────┐
            │  ROCKSET ENGINE │
            └─────────────────┘
                     │
                     ▼ (Standard SQL / Low Latency)
[ Dashboards / Fraud Engines / Vector Search / Leaderboards ]

One of Rockset’s standout capabilities is schemaless ingestion: it allows users to fire standard SQL queries directly over semi-structured nested JSON without predefining a rigid relational schema or running destructive DDL migrations.

---

The ALT (Aggregator-Leaf-Tailer) Architecture

Rockset decouples computation and storage using an architecture known as ALT (Aggregator, Leaf, Tailer). Rather than relying on monolithic nodes that handle ingestion, indexing, and query execution all at once, Rockset divides these responsibilities across three specialized tiers.

flowchart TD
    subgraph Ingestion Tier
        Sources[OLTP / Kafka / S3] --> T[Tailer Nodes]
        T --> DLog[(Distributed Write Log)]
    end

    subgraph Storage Tier
        DLog --> L1[Leaf Node 1]
        DLog --> L2[Leaf Node 2]
        DLog --> L3[Leaf Node N]
        L1 <--> S3[(Cloud Storage: S3 / SST Files)]
        L2 <--> S3
        L3 <--> S3
    end

    subgraph Query Tier
        Client[SQL Client / App] --> A[Aggregator Nodes]
        A -->|Scatter Subqueries| L1
        A -->|Scatter Subqueries| L2
        A -->|Scatter Subqueries| L3
        L1 -->|Gather Results| A
        L2 -->|Gather Results| A
        L3 -->|Gather Results| A
        A --> Client
    end

1. Tailer Nodes (Ingestion Layer)

• Role: Pulls data from external sources (streams, databases, object stores) or accepts direct programmatic writes via Rockset’s REST Write API.
• Durability Guarantee: Tailers do not write directly to storage nodes. Instead, upon receiving a record, the Tailer node appends it to a durable Distributed Log.
• Failure Isolation: If downstream indexing nodes stall or crash, incoming writes remain safely persisted in the distributed log, preventing data loss and backpressure on the origin systems.

2. Leaf Nodes (Storage and Indexing Layer)

• Role: Leaf nodes subscribe to partitions of the distributed log, pull incoming mutation records, and index them.
• Storage Engine: Leaf nodes use an optimized cloud variant of RocksDB (RocksDB-Cloud).
• Data Organization: Each document has an internal unique identifier. Rockset determines the responsible Leaf node using hash-based partitioning (hash-based ownership) over this document ID. This produces an even distribution across nodes, avoiding write hotspots.

3. Aggregator Nodes (Query Layer)

• Role: Serves user SQL queries.
• Query Execution (Scatter-Gather):
  1. The Aggregator parses the SQL query, compiles it, and breaks it down into disjoint subqueries.
  2. It scatters the subqueries in parallel across the relevant Leaf nodes holding the required data shards.
  3. Leaf nodes execute the localized filter/index lookups and return the intermediate sets.
  4. The Aggregator gathers the partial outputs, performs global sorting/aggregations, and streams the final result back to the client.

---

Storage Deep-Dive: RocksDB-Cloud and Amazon S3

Leaf nodes serve both writes and reads. To achieve high write throughput while maintaining low-latency point lookups and scans, Rockset relies on RocksDB-Cloud, an open-source extension developed by Rockset built on top of Meta’s RocksDB.

LSM-Tree Write Path

RocksDB is an LSM-Tree (Log-Structured Merge-tree) storage engine:

1. Incoming writes are appended to an in-memory buffer called the MemTable.
2. Because writes are sequential in-memory operations, write throughput is exceptionally fast.
3. When a MemTable reaches capacity, it is frozen and flushed sequentially to disk as an immutable SST (Sorted String Table) file.
4. Background compaction threads periodically merge overlapping SST files to maintain read performance and clean up deleted/updated records.

Decoupling Persistence with Cloud Storage

Standard RocksDB relies entirely on local disks (NVMe/SSD). If a node fails, its data must be recovered from replicas, incurring heavy network replication overhead.

Rockset modifies this model:

• Leaf nodes flush SST files to local persistent SSDs for low-latency caching, but asynchronously replicate and persist these SST files directly to Amazon S3.
• This architecture leverages S3’s $99.999999999\%$ (11 9s) durability guarantee without requiring a complex, self-managed disk replication protocol.

Incoming Ingestion Record
         │
         ▼
   [ Leaf Node ]
         │ (In-Memory MemTable)
         ▼
  [ Local SSD: SST File ]
         │ (Background Flush)
         ▼
[ Amazon S3 (Cold / Durable SST Storage) ]

---

Elastic Scalability and Failure Handling

The fundamental advantage of the ALT architecture is that each dimension of scale can be addressed independently, without over-provisioning unused resources.

Bottleneck Type	Symptoms	ALT Scaling Strategy
Ingestion Bursts	Massive spike in incoming CDC events or bulk loads	Scale Tailers: Add more Tailer nodes to pull data faster and push to the distributed log. Leaf nodes continue processing without being overwhelmed.
Dataset Growth	Storage footprint expands significantly	Scale Leaf Nodes: Add Leaf instances. Hash partitions rebalance, spreading SST files and memory pressure across more hardware.
Query Concurrency Spikes	Large volume of incoming user analytics queries	Scale Aggregators: Add stateless Aggregator nodes to compile, plan, and scatter queries across existing storage.
Read Bottleneck at Storage	Leaf nodes max out CPU/Memory during heavy scans	Spin Up Replicated Leaf Nodes: Provision new Leaf nodes that hydrate their local caches directly by downloading SST files from S3, enabling parallel query processing without rewriting data.

---

Summary of Key Architectural Benefits

1. Independent Scaling: Ingestion compute (Tailers), storage/indexing compute (Leafs), and query compute (Aggregators) scale orthogonally.
2. Zero Ingestion Impact on Query Latency: Tailers buffer mutations into a distributed log. Bursty data writes do not degrade the CPU or bandwidth required by Aggregators to serve real-time user queries.
3. Cloud-Native Durability: By offloading persistent SST files to Amazon S3, Leaf nodes stay relatively lightweight and can be recovered or horizontally cloned rapidly.
4. No Hotspots: Hash-based partitioning over document IDs ensures that incoming ingestion and data distributions remain uniformly scattered across the Leaf tier.