Optimizing Jira: How Atlassian Migrated from JSON to Protobuf for Massive Performance and Cost Benefits

Atlassian’s Jira, a widely used project management tool, embarked on a significant architectural evolution by migrating its core “issues” data from a monolithic application to a dedicated microservice. This transition, while offering modularity, also presented challenges, particularly concerning data exchange efficiency. This article delves into how Jira successfully moved from JSON to Protobuf, achieving remarkable improvements in performance, resource utilization, and cost efficiency.

The Context: Migrating Jira Issues to a Microservice

The primary motivation was to decouple the “issues” functionality from the Jira monolith into a dedicated “issues microservice.” This microservice was designed with a simple key-value store database, treating each issue as a document, initially stored in JSON format. The migration from the monolith to the issues microservice was executed in three distinct phases to ensure a smooth, controlled rollout.

Phase 1: Issues Service as an Eventually Consistent Cache

In the initial phase, the issues microservice acted as an eventually consistent cache for issues data.

• Request Flow: All requests continued to hit the Jira monolith.
• Writes: Write requests were directed to the primary database, which remained the source of truth for issues.
• Reads:
  1. A read request for an issue came to the Jira monolith.
  2. The monolith called the issues microservice to retrieve the data.
  3. The issues microservice checked its local database.
  4. If the data was present (cache hit), it served the data.
  5. If it was a cache miss, the issues microservice would then query the Jira monolith, which in turn fetched the data from the primary database.
  6. The data was then cached in the issues microservice’s database for subsequent reads.

This phase established the issues service as a read-through cache, gradually populating its data store without disrupting the primary write path.

Phase 2: Dual Writes to Issues Service and Monolith

To reduce the issues microservice’s dependency on the monolith for cache misses, Phase 2 introduced dual writes.

• Writes: All write operations were now directed to both the primary database (monolith’s source of truth) and the issues microservice’s database.
• Reads: Reads primarily went to the issues microservice. For critical reads requiring strong consistency, the monolith’s primary database could still be consulted. This ensured that the issues microservice’s database was always up-to-date, eliminating the need to fall back to the monolith for data retrieval.

Phase 3: Full Transition to Issues Service

The final phase completed the transition, making the issues microservice the authoritative source for issues data.

• Writes: All writes for issues data were directed solely to the issues microservice.
• Reads: All reads for issues data were served directly from the issues microservice. This marked the complete decoupling of the issues domain, with the microservice handling both reads and writes independently.

The Need for a New Data Format: From JSON to Protobuf

During the migration, a critical observation emerged: issues documents were often very bulky, leading to significant network bandwidth consumption. JSON, while human-readable and widely adopted for web communication, proved to be inefficient for high-volume, performance-critical internal service communication due to:

• Large Data Size: JSON’s verbose nature results in larger payloads.
• Serialization/Deserialization Overhead: Converting JSON to and from objects is computationally expensive.
• Network Bandwidth: Larger data sizes consume more network resources, increasing latency and cost.

Recognizing these limitations, Atlassian sought a more efficient data serialization format. After evaluating options like Avro, Kryo, and Thrift, Protobuf (Protocol Buffers) emerged as the clear winner.

Why Protobuf?

Protobuf offered several compelling advantages:

• Super Compact Format: Highly compressed binary format significantly reduces data size.
• Efficient Marshalling/Unmarshalling: Faster serialization and deserialization times compared to JSON.
• Schema Evolution: Supports both backward and forward compatibility, simplifying schema changes without breaking existing clients.
• Widespread Adoption: A mature and widely used standard in the industry.

No-Downtime Migration from JSON to Protobuf

Migrating the data format in a live production system without downtime required a meticulous, phased approach. The strategy involved introducing Protobuf incrementally while maintaining JSON compatibility for external clients.

Step 1: Monolith Provides Protobuf to Issues Service

The first step focused on enabling the Jira monolith to provide issue data in Protobuf format to the issues microservice.

• A new endpoint was created in the Jira monolith.
• This endpoint returned issue data specifically in Protobuf format.
• The issues microservice was updated to consume this new Protobuf endpoint whenever it needed to fetch data from the monolith (e.g., during a cache miss in Phase 1 or for initial data population).
• The issues microservice then stored this data in its mcash database in Protobuf format.

Step 2: Issues Service Stores and Returns Protobuf

Next, the issues microservice was enhanced to handle Protobuf internally and serve it to the monolith.

• A new endpoint was created in the issues microservice.
• This endpoint was designed to return issue data in Protobuf format.
• The Jira monolith was updated to call this new Protobuf endpoint on the issues microservice when requesting issue data.
• Internally, the issues microservice’s mcash database now consistently stored issue data serialized as Protobuf.

Step 3: Monolith Converts Protobuf to JSON for Clients

The final step ensured that end-users remained unaffected by the internal format change.

• When a user request came to the Jira monolith, the monolith would call the new Protobuf endpoint on the issues microservice.
• It received the issue data in Protobuf format.
• The monolith then converted this Protobuf data back into JSON format.
• Finally, the JSON data was served to the client (user).

This multi-step process allowed for a gradual, controlled transition, where internal communication became efficient with Protobuf, while external APIs continued to serve JSON, ensuring no downtime or user impact.

Incremental Rollout with Feature Flags

To manage the risk and ensure correctness during the rollout of new logic and data formats, Atlassian employed a robust incremental rollout strategy using feature flags. This allowed for testing in production without affecting all users and provided an easy rollback mechanism. A feature flag was introduced with three possible values: old, check, and new.

• old:

  • Executes only the existing, proven logic (e.g., using JSON, calling the old endpoint).
  • Serves the result from the old logic to the user.
  • This is the default or fallback state, ensuring stability.

• new:

  • Executes only the new logic (e.g., using Protobuf, calling the new endpoint).
  • Serves the result from the new logic to the user.
  • Used for a small percentage of users or internal testing once confidence is high.

• check:

  • Executes both the old logic and the new logic concurrently.
  • Compares the results from both executions.
  • If any differences are found, these are reported (e.g., logged for engineers to investigate).
  • Crucially, it always serves the result from the old logic to the user.
  • This mode is invaluable for identifying bugs and inconsistencies in the new logic in a production environment without impacting the end-user experience.

This old-check-new pattern is a common and highly effective strategy for safe, incremental deployments and easy reversibility. If any issues arose with the new logic, simply switching the flag back to old would immediately revert the system to its stable state.

Impact and Results

The migration from JSON to Protobuf for Jira’s issues microservice yielded astounding results, demonstrating the power of critically evaluating fundamental design choices like data serialization formats.

• CPU Utilization: The mcash cluster’s CPU consumption was reduced by 75%.
• Data Size: The data stored in mcash was 80% smaller than its JSON equivalent.
• Serialization/Deserialization Speed:
  • Deserialization time was 33 times faster.
  • Serialization time was 4 times faster.
• Cost Optimization: The significant reduction in resource usage (CPU, data size) allowed Atlassian to reduce their mcash cluster size by 55%, leading to substantial cost savings.

These metrics highlight a massive improvement across performance, resource efficiency, and operational costs, all stemming from a strategic shift in data serialization.

Key Takeaways

Jira’s journey from JSON to Protobuf is a classic example of how challenging common design decisions can unlock immense value. It underscores the importance of:

• Performance-Driven Design: For high-volume, internal service communication, the efficiency of data serialization formats can have a profound impact.
• Phased Migration: Complex architectural changes and data format migrations can be executed safely and without downtime through careful planning and incremental rollouts.
• Robust Rollout Strategies: Techniques like feature flags with old, check, and new states are crucial for testing new logic in production, identifying bugs, and ensuring easy reversibility.
• Holistic Impact: Optimizations in one area (e.g., data format) can cascade into significant benefits across CPU, memory, network, and ultimately, infrastructure costs.

By critically evaluating their data format, Atlassian not only enhanced Jira’s performance but also achieved substantial operational efficiencies and cost reductions, proving that even seemingly small technical choices can lead to monumental improvements in large-scale distributed systems.