Understanding TCP TIME_WAIT and Resolving "Address Already in Use" Errors

The Problem: The Mystery of the Ghost Port

A common issue encountered when developing or operating network services (such as in-memory databases like DiceDB, web servers, or custom daemon processes) is the immediate failure to restart a stopped service:

Error: bind: address already in use (EADDRINUSE)

When checking for active processes bound to that specific port using commands like lsof -i :<port> or netstat -tulnp, no running process appears. Despite the process having terminated, the operating system kernel still prevents re-binding to that socket address.

The root cause lies in the transport layer: the socket is lingering in the TCP TIME_WAIT state.

---

The Anatomy of TCP Connection Termination

TCP is a full-duplex, connection-oriented protocol where both sides must independently close their respective send channels. Closing a connection requires a four-way handshake involving FIN (Finish) and ACK (Acknowledgment) segments.

The 4-Way Teardown Flow

sequenceDiagram
    autonumber
    participant A as Client / Initiator (Active Closer)
    participant B as Server / Receiver (Passive Closer)
    
    Note over A,B: Connection Established
    
    A->>B: FIN (seq = u)
    Note over A: State: FIN_WAIT_1
    Note over B: State: CLOSE_WAIT
    B->>A: ACK (ack = u + 1)
    Note over A: State: FIN_WAIT_2
    
    Note over B: Flushes remaining data...
    B->>A: FIN (seq = v)
    Note over B: State: LAST_ACK
    
    A->>B: ACK (ack = v + 1)
    Note over A: State: TIME_WAIT
    Note over B: State: CLOSED
    
    Note over A: Waits 2 * MSL (typically 60s)
    Note over A: State: CLOSED

1. Initiation (Active Close): Endpoint $A$ decides to terminate the connection and sends a FIN packet to Endpoint $B$. Endpoint $A$ transitions from ESTABLISHED to FIN_WAIT_1.
2. Acknowledgment: Endpoint $B$ receives the FIN and immediately responds with an ACK. Endpoint $B$ moves to CLOSE_WAIT, while Endpoint $A$ transitions to FIN_WAIT_2 upon receipt.
3. Passive Teardown: Once Endpoint $B$ completes transmitting any pending outbound buffers, it issues its own FIN to Endpoint $A$ and transitions to LAST_ACK.
4. Final Acknowledgment & TIME_WAIT: Endpoint $A$ sends the final ACK to Endpoint $B$ acknowledging the termination. Endpoint $B$ transitions directly to CLOSED. However, Endpoint $A$ does not immediately close; it transitions into TIME_WAIT.

---

Why Does TIME_WAIT Exist?

The party that initiates an active close enters the TIME_WAIT state for two primary safety guarantees mandated by the TCP specification (RFC 793 / RFC 1122):

1. Reliable Final ACK Delivery

Network links are lossy. If the final ACK sent from Endpoint $A$ to Endpoint $B$ is dropped in transit, Endpoint $B$ remains stuck in LAST_ACK and will retransmit its FIN packet.

If Endpoint $A$ had completely torn down its state and immediately freed the socket:

• Endpoint $A$ would respond to the retransmitted FIN with a RST (Reset) packet.
• Endpoint $B$ would interpret this as an abnormal, ungraceful connection reset rather than a clean termination.

By preserving state in TIME_WAIT, Endpoint $A$ can absorb and retransmit the final ACK when receiving duplicate FIN frames, ensuring both peers finish in a clean CLOSED state.

2. Preventing Old Duplicate Segments from Corrupting New Connections

Packets can be delayed, duplicated, or routed through suboptimal paths on the Internet. If a socket pair (source_ip, source_port, dest_ip, dest_port) could be reused instantly, delayed packets from the old, terminated connection might arrive at the new connection using the same tuple, leading to silent data corruption.

To prevent this, the socket pair is quarantined for $2 \times \text{MSL}$ (Maximum Segment Lifetime):

• MSL is the maximum duration an IP packet can exist on the network before being discarded by router TTL mechanics.
• Common kernel defaults set MSL to 30 to 60 seconds, meaning TIME_WAIT typically lasts between 60 to 120 seconds.

---

Why This Triggers EADDRINUSE on Restart

When you stop a server process (e.g., stopping an instance of DiceDB), the server process calls close() on its client connections or the listener socket, acting as the active closer.

Consequently, the operating system holds the socket tuple in the kernel’s connection table in TIME_WAIT. When you immediately restart the binary and call bind() on the exact same port, the kernel refuses by default, raising EADDRINUSE to safeguard against potential segment overlap.

---

Solutions and Best Practices

1. Enabling SO_REUSEADDR

The idiomatic way to handle rapid server restarts across production systems is to configure the SO_REUSEADDR socket option before binding the listener socket.

SO_REUSEADDR informs the kernel that if a port is currently held by a socket in the TIME_WAIT state, the operating system can allow a new socket to bind to that exact address and port.

C Implementation Example

int sockfd = socket(AF_INET, SOCK_STREAM, 0);
int optval = 1;

// Allow reuse of local addresses in TIME_WAIT
if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &optval, sizeof(optval)) < 0) {
    perror("setsockopt(SO_REUSEADDR) failed");
    exit(EXIT_FAILURE);
}

// bind() will now succeed even if sockets linger in TIME_WAIT
bind(sockfd, (struct sockaddr *)&server_addr, sizeof(server_addr));

Go Implementation Example

import (
    "net"
    "syscall"
)

// Using a ListenConfig to set socket control flags
lc := net.ListenConfig{
    Control: func(network, address string, c syscall.RawConn) error {
        var opErr error
        err := c.Control(func(fd uintptr) {
            opErr = syscall.SetsockoptInt(int(fd), syscall.SOL_SOCKET, syscall.SO_REUSEADDR, 1)
        })
        if err != nil {
            return err
        }
        return opErr
    },
}

listener, err := lc.Listen(context.Background(), "tcp", ":8080")

2. Tuning Kernel Parameters (Avoid Disabling Blindly)

Under high-load proxy or microservice architectures, thousands of outbound connections cycling through TIME_WAIT can lead to port exhaustion.

Linux provides sysctl tuning flags:

• net.ipv4.tcp_tw_reuse: Allows the kernel to safely reuse TIME_WAIT sockets for outgoing client connections when timestamps (net.ipv4.tcp_timestamps) are enabled.
• Warning on tcp_tw_recycle: Historical kernels provided tcp_tw_recycle, which has been completely removed in modern Linux kernels (v4.12+) because it broke connections behind NAT.

---

Summary

Observation	Mechanism	Purpose / Solution
Server crashes/stops, fails to rebind	Kernel retains connection in TIME_WAIT	Protects TCP reliability and prevents old packet collisions
Initiator of close() gets stuck	Active Closer responsibility	Must ensure the passive closer receives final ACK
Fix for listener restart	SO_REUSEADDR	Allows immediate re-binding of local listening address
Fix for outbound port exhaustion	tcp_tw_reuse + TCP Timestamps	Allows safe reuse of outgoing client sockets