⚡ Module 02: TCP vs UDP & Transport Layer Mastery

Core Focus: 3-Way Handshake, 4-Way Termination, TIME_WAIT vs CLOSE_WAIT Sockets, Flow Control (Sliding Window), Congestion Control (Slow Start, AIMD, Fast Retransmit/Recovery, BBR), TCP Header Anatomy, and Low-Latency Tuning for Payment Gateways.

📑 Table of Contents

1. Transport Layer Overview & Core Responsibilities
2. Definitive Comparison: TCP vs UDP
3. TCP 3-Way Handshake (Connection Establishment)
   • 3.1 Step-by-Step Packet Flow & Sequence Numbers
   • 3.2 SYN Flood Attack & SYN Cookies Mitigation
4. TCP 4-Way Handshake (Connection Termination)
   • 4.1 Step-by-Step Termination Sequence
   • 4.2 The TIME_WAIT State & 2MSL Timer Rationale
   • 4.3 Production Disasters: TIME_WAIT vs CLOSE_WAIT Socket Leaks
5. Flow Control: The Sliding Window Protocol
   • 5.1 Window Size (rwnd) Mechanics
   • 5.2 Zero Window Probe & Silly Window Syndrome
6. Congestion Control: Protecting the Global Network
   • 6.1 The 4 Phases: Slow Start, Congestion Avoidance, Fast Retransmit, Fast Recovery
   • 6.2 TCP Tahoe vs TCP Reno vs Modern Google BBR
7. UDP Internals & When to Use It
8. Juspay & Payment Systems Real-World Context
9. High-Yield Interview Questions & Verbatim Answers

1. Transport Layer Overview & Core Responsibilities

The Transport Layer (OSI Layer 4) is responsible for Process-to-Process delivery (end-to-end communication) using Port Numbers.

A network connection is uniquely identified by a 5-Tuple: $$\text{Socket Connection} = (\text{Source IP}, \text{Source Port}, \text{Destination IP}, \text{Destination Port}, \text{Protocol})$$

2. Definitive Comparison: TCP vs UDP

3. TCP 3-Way Handshake (Connection Establishment)

Before any data is exchanged, client and server must agree on Initial Sequence Numbers (ISN) and establish buffer parameters.

3.1 Step-by-Step Packet Flow & Sequence Numbers

1. Packet 1 (SYN): Client sends a packet with SYN=1, random Initial Sequence Number $\text{Seq} = x$, and options (MSS, Window Scale). State changes from CLOSED $\rightarrow$ SYN_SENT.
2. Packet 2 (SYN-ACK): Server receives SYN, allocates TCB (Transmission Control Block) buffer, sends back SYN=1, ACK=1, $\text{Ack} = x+1$, and its own sequence number $\text{Seq} = y$. State changes from LISTEN $\rightarrow$ SYN_RCVD.
3. Packet 3 (ACK): Client receives SYN-ACK, acknowledges server sequence number with $\text{Ack} = y+1$. State changes to ESTABLISHED. Server receives ACK and enters ESTABLISHED.

[!NOTE] Why random ISN?
If sequence numbers always started at 0, an attacker could predict sequence numbers and inject malicious packets into an active TCP connection (TCP sequence prediction attack). Furthermore, delayed duplicate packets from a previous closed connection could corrupt a new connection.

3.2 SYN Flood Attack & SYN Cookies Mitigation

• Vulnerability: Server allocates memory in the SYN Queue for every SYN received while waiting for the final ACK. A flood of fake SYNs exhausts memory and prevents legitimate users from connecting.
• Solution — SYN Cookies:
  • The server does not allocate memory in the backlog queue upon receiving a SYN.
  • Instead, the server encodes the connection state inside the server's Initial Sequence Number $y$: $$y = \text{Hash}(\text{Client IP}, \text{Client Port}, \text{Server Secret}, \text{Timestamp}) + \text{MSS Index}$$
  • When the client sends the 3rd packet (ACK with $\text{Ack} = y+1$), the server extracts $y$, re-computes the cryptographic hash, verifies its authenticity, and only then allocates the connection memory!

4. TCP 4-Way Handshake (Connection Termination)

TCP is full-duplex (data flows both directions independently). Closing a connection requires closing both channels independently.

4.2 The TIME_WAIT State & 2MSL Timer Rationale

The endpoint that performs the Active Close (usually the client, or a reverse proxy like Nginx) must enter the TIME_WAIT state and remain there for $2 \times \text{MSL}$ (Maximum Segment Lifetime, typically 60 to 120 seconds).

4.3 Production Disasters: TIME_WAIT vs CLOSE_WAIT Socket Leaks

5. Flow Control: The Sliding Window Protocol

Flow Control ensures a fast sender does not overwhelm a slow receiver's memory buffer.

5.1 Window Size (rwnd) Mechanics

• In every ACK packet, the receiver advertises its available buffer space via the Receive Window (rwnd) header field.
• The sender is mathematically constrained: $$\text{Bytes In-Flight} \le \text{rwnd}$$
• If the receiver's application is slow reading data from the buffer, rwnd drops towards 0.

5.2 Zero Window Probe & Silly Window Syndrome

• Zero Window: When rwnd = 0, sender halts data transmission. The sender starts a Persist Timer and periodically transmits a 1-byte Zero Window Probe to prompt the receiver to advertise its updated rwnd.
• Silly Window Syndrome:
  • Occurs when receiver application reads data 1 byte at a time and advertises a window of 1 byte, causing sender to transmit tiny packets with huge header overhead (40 bytes header for 1 byte payload = 2.5% efficiency).
  • Receiver Solution (Clark's Algorithm): Receiver does not advertise a non-zero window until free space is $\ge \min(\text{MSS}, \text{Buffer}/2)$.
  • Sender Solution (Nagle's Algorithm): Sender buffers small outgoing messages until a full MSS segment is ready or until an ACK for previous data is received.

6. Congestion Control: Protecting the Global Network

While Flow Control protects the receiver, Congestion Control protects the intermediate network routers and switches from being overwhelmed.

The sender computes the effective sending window: $$\text{Max Send Window} = \min(\text{cwnd}, \text{rwnd})$$ Where $\text{cwnd}$ is the sender's dynamically calculated Congestion Window.

6.1 The 4 Phases:

1. Slow Start:
   • Starts with $\text{cwnd} = 1$ (or 10 in modern Linux).
   • For every ACK received, $\text{cwnd} = \text{cwnd} + 1 \implies \text{cwnd}$ doubles every RTT ($1 \rightarrow 2 \rightarrow 4 \rightarrow 8 \dots$).
   • Continues until $\text{cwnd} \ge \text{ssthresh}$ (Slow Start Threshold).
2. Congestion Avoidance (AIMD — Additive Increase Multiplicative Decrease):
   • When $\text{cwnd} \ge \text{ssthresh}$, growth switches from exponential to linear.
   • Increases $\text{cwnd}$ by 1 MSS per RTT ($\text{cwnd} = \text{cwnd} + \frac{1}{\text{cwnd}}$ per ACK).
3. Fast Retransmit:
   • If a single packet is lost while subsequent packets arrive, the receiver sends Duplicate ACKs.
   • Upon receiving 3 Duplicate ACKs, the sender retransmits the missing segment immediately without waiting for the slow Retransmission Timeout (RTO) timer.
4. Fast Recovery:
   • Instead of dropping $\text{cwnd}$ back to 1 (like legacy Tahoe), TCP Reno sets $\text{ssthresh} = \frac{\text{cwnd}}{2}$ and $\text{cwnd} = \text{ssthresh}$, staying in Congestion Avoidance.

6.2 TCP Tahoe vs TCP Reno vs Modern Google BBR

• TCP Tahoe: On any loss (Timeout or 3 Dup ACKs), drops $\text{cwnd} = 1$ and enters Slow Start. (Severe throughput collapse).
• TCP Reno: Drops $\text{cwnd} = 1$ on Timeout; on 3 Dup ACKs, cuts $\text{cwnd}$ in half (Fast Recovery).
• Google BBR (Bottleneck Bandwidth and RTT):
  • Loss-based algorithms (Reno/Cubic) falsely interpret any dropped packet as network congestion, causing bufferbloat and performance degradation on high-speed fiber/mobile networks.
  • BBR measures real-time maximum delivery rate and minimum round-trip time to model the physical bottleneck, maximizing throughput without filling router buffers.

7. UDP Internals & When to Use It

• Why UDP Wins in Real-Time Applications:
  • Zero handshake overhead (immediate data transmission).
  • No Head-of-Line (HoL) blocking — if packet 2 drops, packet 3 is delivered to the app immediately.
  • Custom congestion control: Applications can decide their own retransmission policy (e.g., in WebRTC, drop old video frames instead of retransmitting them).
  • Basis for HTTP/3 (QUIC).

8. Juspay & Payment Systems Real-World Context

1. TCP_NODELAY (Disabling Nagle's Algorithm):
   • By default, Nagle's algorithm buffers small TCP packets. For payment APIs (where JSON payloads are small, e.g., 500 bytes), Nagle can introduce an artificial 40ms delay waiting for ACKs.
   • Juspay disables Nagle via setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, 1) to transmit transaction payloads instantly.
2. Persistent Connection Pooling:
   • Creating a new TCP connection takes 1 RTT (3-way handshake) + 2 RTTs (TLS 1.2 handshake) = 3 RTTs (~150ms over cellular networks).
   • Juspay maintains warm, long-lived TCP connection pools to merchant servers and banking APIs, eliminating connection handshake latency on payment critical paths.
3. Socket Exhaustion Protection:
   • High TPS payment bursts can exhaust local ephemeral ports (65,535 max). Juspay uses multiple outbound NAT gateways and IP-binding configurations to scale past single-interface port limitations.

9. High-Yield Interview Questions & Verbatim Answers

Q1: "Explain the TCP 3-Way Handshake. Why is it 3 steps and not 2?"

Spoken Answer:
*"The TCP 3-way handshake establishes a reliable, full-duplex byte stream connection.

1. The client sends a SYN packet with an initial sequence number $x$.
2. The server acknowledges $x$ by sending ACK = x + 1 and synchronizes its own channel by sending SYN with sequence number $y$.
3. The client responds with ACK = y + 1 to confirm receipt of the server's sequence number.

It cannot be 2 steps because TCP is bidirectional. With only 2 steps, the client knows the server received its SYN, but the server has no confirmation that the client received the server's SYN. Both sides must independently announce their sequence numbers and receive an explicit acknowledgment to guarantee synchronization."*

Q2: "What is the TIME_WAIT state in TCP? Why does it wait for 2MSL?"

Spoken Answer:
*"The TIME_WAIT state is entered by the endpoint that initiates the active close after sending the final ACK. It lasts for $2 \times \text{MSL}$ (Maximum Segment Lifetime), which is typically 60 to 120 seconds.
It exists for two fundamental reasons:

1. To ensure graceful termination: If the final ACK is lost in the network, the remote server will retransmit its FIN. The active closer must remain in TIME_WAIT to resend the ACK; otherwise, it would reply with an RST error.
2. To drain stale packets from the network: It ensures that delayed duplicate packets from this connection expire in intermediate router queues before a new connection reuses the same 5-tuple socket, preventing data corruption."*

Q3: "What is the difference between Flow Control and Congestion Control in TCP?"

Spoken Answer:
*"Flow Control is end-to-end between sender and receiver. It prevents a fast sender from overwhelming a slow receiver's memory buffer. It is implemented using the Sliding Window Protocol, where the receiver advertises its available buffer space (rwnd) in every ACK.

In contrast, Congestion Control is global between the sender and the intermediate network. It prevents excessive data injection from causing router buffer overflows and network collapse. It is managed by the sender dynamically adjusting its Congestion Window (cwnd) using algorithms like Slow Start, AIMD, and Fast Retransmit."*

Q4: "What is a SYN Flood attack and how do SYN Cookies mitigate it?"

Spoken Answer:
*"A SYN flood is a Denial of Service attack where an attacker sends a high volume of TCP SYN requests with spoofed IP addresses and never sends the completing ACK. The server allocates memory in its SYN backlog queue for each half-open connection, quickly exhausting RAM and denying service to legitimate users.

SYN Cookies solve this by removing the need for server-side state allocation during the SYN phase. The server cryptographically encodes the connection parameters into the server's Initial Sequence Number ($y$) using a hash of the client IP, port, server secret, and timestamp. The server only allocates memory after receiving the client's final ACK and validating the decoded sequence number."*