Architecture and Internals of Blacksmith: Accelerating GitHub Actions with Hardware-First Optimization

Continuous Integration and Continuous Deployment (CI/CD) pipelines test code, build container images, compile artifacts, and deploy services. For engineering teams, CI/CD speed directly dictates developer velocity. Slow builds lead to blocked pull requests, delayed deployments, and significant cloud infrastructure costs.

Blacksmith is an alternative execution engine for GitHub Actions that runs workflows roughly twice as fast while cutting billing costs in half—often requiring only a one-line change in workflow configuration. To understand how this is achieved, we must examine its high-level pipeline, its execution engine, and the counter-intuitive hardware trade-offs driving its cost efficiency.

---

1. High-Level System Architecture

The life cycle of a Blacksmith-driven GitHub Action consists of webhook ingestion, queue management, agent dispatching, and microVM lifecycle control.

flowchart TD
    A[GitHub Actions Event Trigger] -->|Webhook with Job Metadata| B[Blacksmith Ingestion Backend]
    B -->|Validate & Extract Payload| C[(Redis Job Queue)]
    
    subgraph Host Machine [Bare-Metal Host: AMD Ryzen 7950X + Local NVMe]
        D[Blacksmith Agent (Go Process)] -->|Poll Job| C
        D -->|Spawn via cgroups| E[Firecracker MicroVM]
        E -->|Execute Build & Test Tasks| E
        E -->|Read/Write Artifacts 400 MB/s| F[(Collocated Cache)]
        D -->|Report Status| G[GitHub Actions API]
    end

Pipeline Stages

1. Webhook Ingestion: When a GitHub Actions workflow is triggered (e.g., on a pull request or merge), GitHub dispatches a webhook containing the job metadata to Blacksmith’s backend.
2. Validation and Enqueuing: Blacksmith’s backend service validates the signature, parses the job requirements and matrix variables, and pushes the job payload onto an in-memory Redis queue.
3. Agent Polling: Dedicated agent worker daemons, written in Go, continuously poll the Redis queue for pending jobs.
4. Execution and Reporting: Upon receiving a task, the host agent spins up an isolated virtual environment, executes the workflow steps, streams the execution logs, and returns the status back to GitHub via its check-runs/job status APIs.

---

2. The Core Hardware Trade-Off: High-End Hardware vs. Cloud Billing

At first glance, providing significantly higher-tier compute resources while charging less seems economically impossible. However, the fundamental economics of cloud infrastructure explain this discrepancy.

The Cloud Economic Rule: Public cloud infrastructure is billed per-second of consumption. If utilizing hardware that is $2\times$ more expensive allows a job to complete in $0.25\times$ or $0.5\times$ the time, the aggregate cost per run drops significantly.

$\text{Total Cost} = \text{Hardware Unit Price per Second} \times \text{Execution Duration}$

If the reduction in execution time outpaces the hardware price premium, the overall pipeline cost decreases.

Standard Runner:  [ Moderate Hardware Cost ($)  ] x [ 20 Minutes Duration ] = High Net Cost
Blacksmith:       [ Premium Hardware Cost ($$) ] x [  8 Minutes Duration ] = Lower Net Cost

This pattern mirrors the architectural shift from Hadoop MapReduce to Apache Spark. Hadoop performed computations by reading and writing intermediate states to mechanical disk arrays. Spark opted for substantially more expensive RAM for in-memory transformations. While RAM was far costlier than magnetic disks per gigabyte, Spark workflows finished orders of magnitude faster, dramatically reducing the cluster footprint and total operating cost.

---

3. Dissecting the Hardware Bottlenecks

CI/CD pipelines are almost exclusively bound by CPU compute (compiling, transpiling, linting, running unit test suites) and Disk I/O (fetching dependencies, unpacking Docker images, writing build caches).

Dimension	Standard GitHub-Hosted Runners	Blacksmith Runners	Architectural Advantage
Processor	AMD EPYC (Server-grade)	AMD Ryzen 7950X (High-frequency Consumer/Desktop)	Vastly superior single-core boost clock (up to 5.7 GHz)
Disk Storage	Network-Attached Storage (e.g., AWS EBS)	Locally Attached NVMe SSD	Zero network latency; raw PCIe speeds for I/O-heavy builds
Cache Layer	Centralized Network Storage (~100 MB/s)	Host-Collocated Cache (~400 MB/s)	$4\times$ faster dependency and Docker layer hydration

A. CPU: Server-Grade vs. High-Clock Desktop Processors

• Standard Cloud Runners: Typically run on virtualization platforms backed by high-core-density server processors such as the AMD EPYC line. While EPYC processors provide massive multi-tenant core counts, their base and boost clock frequencies are lower to manage thermal limits in dense server racks.
• Blacksmith Runners: Utilize processors like the AMD Ryzen 7950X. Built with architectures geared toward gaming and high-performance client workstations, these processors prioritize single-thread throughput and high operating frequencies (clocking up to 5.7 GHz). Because most CI compilation tasks, test runners, and script interpreters are single-thread bound or poorly parallelized, higher single-core IPC (instructions per cycle) directly cuts execution times in half.

B. Disk: Network-Attached Storage vs. Local NVMe

• The Problem with Network-Attached Disks (AWS EBS): Standard cloud VMs attach storage over the internal data center network. While the operating system surfaces a standard POSIX filesystem interface, every read and write incurs a network hop across the hypervisor fabric. For CI jobs that pull thousands of small files (node_modules, Maven caches, Go package modules), network round-trip overhead compounds quickly.
• The Transient Advantage of NVMe: Network-attached volumes exist so disks can survive VM terminations and detach/reattach during node outages. However, CI/CD jobs are completely transient. If an underlying host dies, the CI build simply fails and is re-run from scratch; persistent disk state is not required. By acknowledging this characteristic, Blacksmith utilizes direct-attached PCIe NVMe SSDs, bypassing network hops and providing near-instantaneous I/O.

C. Artifact and Cache Collocation

Workflows frequently spend 30–50% of their lifespan pulling and pushing dependency caches (npm cache, Docker layers, Gradle dependencies).

• Standard runners fetch these caches from external object stores or network filesystems, typically saturating around 100 MB/s.
• Blacksmith collocates cache artifacts closer to the physical host nodes, delivering transfer rates reaching 400 MB/s. This significantly reduces the hydration phase of the build.

---

4. Execution Sandbox: AWS Firecracker MicroVMs

Speed cannot compromise multi-tenant security. CI/CD runners frequently execute untrusted code submitted via open-source pull requests. Running these directly on the host OS is unsafe, while traditional hypervisors (QEMU/KVM) introduce slow boot times and heavy memory footprints.

To balance isolation, security, and velocity, Blacksmith uses AWS Firecracker microVMs.

flowchart TD
    subgraph Host Bare-Metal OS
        Agent[Blacksmith Agent - Go Daemon]
        Cgroups[Linux cgroups & namespaces]
        
        subgraph Firecracker MicroVM [Ephemeral Firecracker MicroVM]
            Kernel[Minimal Guest Linux Kernel]
            Task[CI/CD Build & Test Workload]
        end
        
        Agent -->|Spawns in milliseconds| Firecracker MicroVM
        Cgroups -->|Enforces CPU, Memory & I/O quotas| Firecracker MicroVM
    end

Why Firecracker?

1. Sub-Second Boot Times: Firecracker strips away legacy hardware emulators and unnecessary peripheral drivers, booting a minimal Linux kernel in approximately 5 milliseconds.
2. Hardware-Level Isolation: Unlike standard container engines (like vanilla Docker) which share the host Linux kernel, Firecracker leverages Linux Kernel-based Virtual Machine (KVM) technology. A kernel panic, exploit, or runaway container inside the runner cannot access or compromise the host machine.
3. Granular Resource Accounting with cgroups: Host agents wrap Firecracker microVMs in Linux control groups (cgroups). This enforces hard ceilings on memory usage, CPU quota allocations, and disk I/O, preventing any single job from starving neighboring microVMs.

---

5. Architectural Takeaways for Systems Designers

1. Tailor Hardware to Workload Constraints: Traditional infrastructure defaults to general-purpose cloud servers with network-attached block storage. By identifying that CI/CD workloads are transient, bursty, and single-thread intensive, Blacksmith selected unconventional hardware (high-clock desktop CPUs and bare-metal NVMe) that directly addresses those constraints.
2. The Economics of Velocity: In usage-based billing models, expensive, high-performing resources can be substantially more economical than cheaper, slower resources if they reduce total execution time.
3. Security Without Overhead: Ephemeral workloads do not require full-blown virtual machines with lengthy boot sequences. MicroVM technologies (such as Firecracker) paired with kernel-level cgroups deliver hypervisor-grade isolation at container-level startup speeds.