Designing API Clients: A Deep Dive into Requestly’s Low-Level Architecture and Data Models

Building a full-fledged API client like Requestly involves much more than just wrapping cURL commands. It requires careful consideration of various features, their interactions, and robust data modeling to support a seamless user experience. This document delves into the low-level design and database schema modeling for such a client, exploring key functionalities like variable interpolation, request chaining, collection runners, and integrated test assertion frameworks.

Beyond a cURL Wrapper: Core Features of an API Client

An API client provides a user-friendly interface to interact with APIs, allowing users to define requests, send them, and inspect responses. While it might seem like a simple wrapper around HTTP requests, a comprehensive client offers advanced features that significantly enhance productivity and testing capabilities:

1. Scripting Runtime: Allows users to write pre-request and post-response scripts to manipulate requests, responses, or environment variables.
2. Scoped Variables: Enables dynamic values in requests that change based on environment, collection, or runtime context.
3. Test Assertion Framework: Provides a mechanism to write and execute tests against API responses, ensuring expected behavior and data structures.
4. Collection Runner: Facilitates the sequential execution of multiple requests within a collection, often used for workflows or automated testing.
5. Migration Layer: Supports importing and exporting API definitions from various sources (e.g., cURL commands, other API clients) for interoperability.

Let’s deep dive into the design and data models for some of these critical features.

Variable Interpolation: Dynamic Values in API Requests

Variable interpolation is a fundamental feature that allows users to define placeholders in their requests (e.g., URLs, headers, body) that are resolved at runtime. This is crucial for managing environment-specific configurations (e.g., dev.example.com vs. prod.example.com) without duplicating requests.

The Challenge of Scoping and Precedence

While simple variable substitution might seem straightforward, a robust system needs to handle complex scenarios involving variable scopes and precedence:

• Scoping: Variables can be defined at different levels, each with its own accessibility:

  • Global: Accessible across all collections and environments (e.g., request_timeout, default_user_id).
  • Environment: Specific to an environment (e.g., base_url for dev, staging, production).
  • Collection: Specific to a particular collection (e.g., default_payment_amount for a ‘Payments’ collection).
  • Runtime: Temporary, session-scoped overrides for a specific execution, not persisted or synced.

• Precedence: When the same variable key is defined at multiple scopes, a clear hierarchy determines which value is used. The typical precedence order is:

  1. Runtime (highest)
  2. Collection
  3. Environment
  4. Global (lowest)

• Resolution Timing: Some variables (dynamic variables) are resolved at the time of invocation, generating a fresh value for each execution (e.g., a random UUID or current timestamp).

Data Model for Variables

To support scoped variables, the database schema would involve several tables:

• variables Table:

  • id: Primary key
  • workspace_id: Links to the user’s workspace/organization.
  • scope: Enum (GLOBAL, ENVIRONMENT, COLLECTION)
  • scope_id: Foreign key to environments.id or collections.id based on scope.
  • key: The variable name (e.g., base_url).
  • initial_value: The default or initial value of the variable. This value is typically synced across devices/cloud.
  • is_secret: Boolean, indicates if the variable contains sensitive information that should be masked by default.

• environments Table:

  • id: Primary key
  • workspace_id: Links to the workspace.
  • name: Environment name (e.g., Dev, Staging, Production).

• collections Table:

  • id: Primary key
  • workspace_id: Links to the workspace.
  • parent_id: Self-referencing foreign key for nested collections (tree view).
  • name: Collection name.

• runtime_variables Table (or in-memory store):

  • id: Primary key
  • session_id: Links to the current user session.
  • key: Variable name.
  • current_value: The value overridden for the current session. This is not synced and remains local to the machine.

Initial Value vs. Current Value

A crucial distinction in variable storage is between initial_value and current_value:

• initial_value: The default value, which can be synced across a user’s workspace (e.g., to a cloud service). This represents the agreed-upon default for a variable.
• current_value: A local override for the initial_value, specific to the current machine or session. This value is not synced and allows users to temporarily test different values without affecting others or the synced configuration.

Dynamic Variables

Dynamic variables (e.g., $random_uuid, $timestamp) are special variables that are not stored but rather evaluated at the time of request invocation, providing a fresh value for each execution. These are typically recognized by a specific prefix (e.g., $ or {{$) and resolved by the client’s execution engine.

Request Chaining: Linking API Calls

Request chaining allows the output of one API request to be used as input for a subsequent request. A common use case is extracting an authentication token from a login response and using it in all subsequent authenticated requests.

Mechanism: The RQ Object and Environment

API clients typically expose a global object (e.g., RQ in Requestly) within the scripting runtime. This object provides access to the current request, response, and a shared environment where data can be stored and retrieved across requests.

• RQ Object Structure:
  • RQ.request: Represents the current request being processed.
  • RQ.response: Represents the response received for the current request.
  • RQ.environment: A key-value store accessible by all scripts within a collection run. Scripts can set values into RQ.environment and get values from it.
  • RQ.globals: Provides access to global variables.

Scripting Hooks

Request chaining is implemented using scripting hooks, which are JavaScript code blocks executed at specific points during a request’s lifecycle:

• Pre-request Script: Executed before the API request is sent. This is where you might retrieve a token from RQ.environment and add it to the current request’s headers.
• Post-response Script: Executed after the API response is received. This is where you would parse the response (e.g., extract a JWT token) and store it in RQ.environment for subsequent requests.

Collection Runner: Automating Workflows

A collection runner enables users to execute all requests within a specified collection in a defined sequence. This is invaluable for testing entire API workflows, integration tests, or data setup/teardown processes.

Execution Flow

When a collection is run, the client iterates through each request. For each request, the following sequence of operations occurs:

1. Pre-request Script Execution: Any pre-request scripts defined for the current request are executed.
2. Variable Resolution: All variables in the request (URL, headers, body, query parameters) are resolved based on their scope and precedence.
3. Request Execution: The actual HTTP request is sent to the API endpoint.
4. Post-response Script Execution: Any post-response scripts are executed, potentially storing data in RQ.environment or performing initial response processing.
5. Test Assertion Execution: The test assertion framework runs tests against the received response.
6. Result Recording: The outcome (status, response time, body, headers, test results) is recorded.

Data Model for Collection Runs and Results

To store the history and results of collection runs, the schema needs to capture the run itself, individual request results, and the requests themselves.

• collections Table (revisited):

  • id, workspace_id, parent_id, name (as defined earlier).

• collection_runs Table:

  • id: Primary key.
  • collection_id: Foreign key to collections.id.
  • environment_id: Foreign key to environments.id (which environment was used for this run).
  • iteration_count: How many times the collection was run (e.g., for data-driven tests).
  • status: Enum (PASSED, FAILED, IN_PROGRESS, etc.).
  • start_time, end_time.

• run_results Table:

  • id: Primary key.
  • collection_run_id: Foreign key to collection_runs.id.
  • request_id: Foreign key to requests.id (which request was executed).
  • iteration_index: If the collection run had multiple iterations.
  • status: Enum (PASSED, FAILED, SKIPPED).
  • response_time_ms: Time taken for the request in milliseconds.
  • response_body_json: The full response body (potentially truncated or stored separately for large responses).
  • response_headers_json: Response headers.
  • test_results_json: JSON blob of individual test assertion outcomes.

• requests Table:

  • id: Primary key.
  • collection_id: Foreign key to collections.id.
  • name: User-defined name for the request.
  • method: HTTP method (e.g., GET, POST).
  • url_template: The URL string, potentially containing variable placeholders.
  • headers_json: JSON representation of request headers, potentially with variables.
  • query_params_json: JSON representation of query parameters, potentially with variables.
  • body_type: Enum (NONE, RAW, FORM_DATA, X_WWW_FORM_URLENCODED, GRAPHQL).
  • body_content: The raw body content or reference to it.
  • auth_info_json: JSON blob containing authentication details (e.g., API key, Bearer token config, OAuth details).

• scripts Table:

  • id: Primary key.
  • request_id: Foreign key to requests.id.
  • phase: Enum (PRE_REQUEST, POST_RESPONSE).
  • source_code: The actual JavaScript code for the script.
  • timeout_ms: Maximum execution time for the script to prevent infinite loops.

Test Assertion Framework: Ensuring API Reliability

An integrated test assertion framework allows users to define tests that validate the structure and content of API responses. This is crucial for catching breaking changes early and ensuring API contracts are met.

Purpose and Mechanism

Tests are typically grouped into named test cases. For example, a test case named