Understanding Network Protocols: Deconstructing HTTP and Designing Custom Protocols from Scratch

When two machines communicate over a network, establishing a transport connection (such as a TCP handshake) is only the first step. While TCP ensures reliable, in-order delivery of raw bytes, it has no intrinsic understanding of what those bytes mean. For meaningful communication, both the client and server must agree upon an application protocol—a shared language and grammar that dictates how commands, data, and responses are formatted, delimited, and parsed.

By peeling back the abstractions of web browsers, curl, and framework request objects, we can observe HTTP directly as structured text over a raw TCP socket. This deep dive covers how protocols function, how HTTP/1.1 parses streams using headers like Host and Content-Length, and how developers can engineer custom protocols from scratch.

---

1. What Exactly is a Protocol?

At its core, a network protocol is nothing more than an agreed-upon contract for encoding and decoding messages.

Consider human conversation: English has grammar, spacing, and punctuation. If one party speaks English and the other only understands Mandarin, communication fails despite sound waves successfully traveling between them. In computing, TCP represents the sound waves, and the application protocol represents the language.

A Simple Toy Protocol

Imagine defining a custom calculation protocol over TCP:

1. Every command is a space-separated ASCII string terminated by a newline character ( ).
2. The first token is the operation (e.g., add).
3. Subsequent tokens are operands (e.g., 2, 3).
4. The server responds with the scalar result followed by .

Client Request:   add 2 3

Server Response:  5

Because both sides know the delimiter ( ) and the token separator ( ), the server can parse incoming bytes deterministically:

sequenceDiagram
    autonumber
    participant Client
    participant Server
    Client->>Server: Connect TCP Socket
    Client->>Server: "add 2 3\n"
    Note over Server: Read stream until '\n'<br/>Split tokens by ' '<br/>Execute 2 + 3 = 5
    Server->>Client: "5\n"

This simple model is not just theoretical. It is almost identical to RESP v1 (Redis Serialization Protocol version 1), where commands such as set mykey myvalue were handled as simple inline space-separated text.

---

2. Deconstructing the HTTP/1.1 Specification

HTTP (Hypertext Transfer Protocol) is an application-level protocol running on top of TCP. In HTTP/1.1, the communication paradigm is strictly client-initiated request/response.

Structure of an HTTP Request

An HTTP request consists of three distinct segments:

1. The Request Line: Defines the HTTP verb, the target URI, and the protocol version.
2. Headers: Key-value pairs containing metadata about the request, connection, or client.
3. Message Delimiter: A mandatory empty line (\r ) indicating the end of the headers.
4. Optional Message Body: Raw payload data (JSON, form data, binary, etc.).

METHOD URI PROTOCOL_VERSION\r

Header-Name-1: Value1\r

Header-Name-2: Value2\r

\r

[Optional Request Body]

Why `\r

(CRLF)? The HTTP specification mandates Carriage Return (\r, ASCII 13) followed by Line Feed ( , ASCII 10`), often abbreviated as CRLF, as the standard line terminator.

---

3. Dissecting HTTP via a Raw TCP Connection

To observe the protocol without helper abstractions, we can start a basic Go web server listening on port 1729 and connect to it using a raw TCP utility (such as nc, telnet, or a direct socket program).

Scenario A: A Basic GET Request

Connecting directly to the port:

nc localhost 1729

If we transmit a naive request line:

GET /foo HTTP/1.1

(Followed by an empty newline)

In Go’s standard net/http implementation, the server responds with an error:

HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close

missing required Host header

The Requirement of the Host Header

Under HTTP/1.0, the Host header was optional. However, HTTP/1.1 made Host mandatory. This allows a single web server with a single IP address to host multiple domain names (Name-Based Virtual Hosting). Without the Host header, the server cannot determine which virtual host should process the request.

Scenario B: A Successful GET Request

Supplying the Host header and terminating the header section with a double newline:

GET /foo HTTP/1.1
Host: localhost

The server immediately parses the request and returns a fully compliant HTTP response:

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Date: Sat, 09 Mar 2024 12:00:00 GMT
Content-Length: 3

bar

If arbitrary, non-HTTP text (e.g., foo bar abcd\r ) is sent over the same connection, the server parser halts and returns 400 Bad Request because the byte sequence violates the HTTP grammar.

---

4. Handling Request Bodies: The Critical Role of Content-Length

TCP is a continuous byte stream with no internal concept of message boundaries. When sending a POST request with a payload, how does the server know where headers end and where the body starts? More importantly, how does it know when the body is complete?

The Delimitation Problem

Consider sending this over raw TCP:

POST /login HTTP/1.1
Host: localhost

user=alice&password=secret

If you type this manually without declaring body boundaries, the server might read the double \r , conclude that the request has completed with an empty body, and return a response before you even begin transmitting the body bytes.

The Content-Length Header

To resolve this ambiguity, HTTP provides two primary framing mechanisms:

1. Content-Length: Explicitly states the size of the payload in bytes.
2. Chunked Transfer Encoding (Transfer-Encoding: chunked): Used when the payload size is unknown ahead of time (streaming).

When using Content-Length, the server reads the headers until it encounters the double \r . It then reads precisely the specified number of bytes from the TCP socket buffer before dispatching the payload to the application handler:

POST /login HTTP/1.1
Host: localhost
Content-Length: 28
Content-Type: application/x-www-form-urlencoded

user=arpit&password=pass1234

flowchart TD
    A[TCP Stream Ingestion] --> B[Read Line by Line until CRLF CRLF]
    B --> C[Extract Headers: Content-Length = N]
    C --> D{N > 0?}
    D -- Yes --> E[Read exactly N bytes from socket]
    E --> F[Construct Request Object]
    D -- No --> F
    F --> G[Dispatch to Route Handler]

What About Content-Type?

While Content-Length tells the lower-level transport parser how many bytes to read, Content-Type tells the application layer how to deserialize those bytes:

• application/json: Parsed by JSON deserializers (e.g., json.Unmarshal in Go, request.json in Flask, req.body with body-parser in Express).
• application/x-www-form-urlencoded: Parsed into key-value query structures.
• multipart/form-data: Handled using boundary markers for binary file uploads.

---

5. Designing Custom Protocols: Why and How

HTTP is universally supported by web browsers and proxies, but it carries overhead:

• Verbose ASCII header parsing.
• Repetitive text metadata in every frame.
• Strict request-response semantics (in HTTP/1.1).

When performance, low latency, or compact binary serialization is required, systems engineers often design custom application protocols.

The Building Blocks of a Custom Protocol

To build a protocol over TCP, you must define:

1. Framing & Framing Boundaries: How the receiver distinguishes messages (e.g., delimiter-based like , or length-prefixed where the first 4 bytes indicate the payload size).
2. Type/Command Identification: An opcode or string specifying the action (0x01 for READ, 0x02 for WRITE).
3. Payload Encoding: Binary (Protobuf, MessagePack, Avro) or Text (JSON, custom CSV-like formats).
4. State Management: Whether requests are synchronous (wait-for-reply) or multiplexed asynchronously (using Correlation IDs).

Why Databases Build Custom Protocols

Databases like MySQL, PostgreSQL, and Redis do not use standard HTTP for their primary query engine:

• MySQL / PostgreSQL Wire Protocols: Use optimized binary packet structures containing length prefixes, sequence numbers, parameter type markers, and raw byte buffers for tabular data.
• Redis (RESP): Uses prefix bytes (+ for simple strings, - for errors, : for integers, $ for bulk strings, * for arrays) to allow fast, allocation-free parsing in C without complex regex or string searches.

# Example RESP for: SET key val
*3\r

$3\r

SET\r

$3\r

key\r

$3\r

val\r

Because these protocols are proprietary to the storage engines, developers cannot query MySQL or Redis directly via a standard web browser. Instead, engineers install dedicated database drivers or client SDKs.

What is a Database Driver?

A driver is simply a client-side encoder/decoder library that:

1. Accepts an API call or query string in the host programming language.
2. Serializes that call into the database’s specific wire protocol specification.
3. Writes the bytes over a raw TCP socket.
4. Reads the response bytes from the socket, parses them according to the wire format, and returns native objects to your application.

---

6. Summary: Key Takeaways

Concept	Responsibility
TCP	Establishes the connection, guarantees reliable byte ordering, and manages packet retransmissions and flow control.
Application Protocol	Defines message semantics, token framing, field delimiters, and interpretation rules for raw bytes.
HTTP/1.1 Request Line	Identifies the verb (GET), the URI (/path), and the version (HTTP/1.1).
**Delimiters (`\r\n\r
`)**	Marks the boundary separating request headers from the optional body payload.
Content-Length	Directs the TCP socket reader on precisely how many payload bytes follow the header section.
Custom Wire Protocols	Allow high-performance, compact framing tailored for internal systems (e.g., Redis, Kafka, MySQL, gRPC).