03-http-https-evolution.md

🚀 Module 03: HTTP/HTTPS & Protocol Evolution

Core Focus: HTTP Methods & Safety/Idempotency Matrix, Exhaustive Status Codes (200, 301, 302, 304, 400, 401, 403, 404, 409, 429, 500, 502, 503, 504), Protocol Evolution (HTTP/1.0 $\rightarrow$ HTTP/1.1 $\rightarrow$ HTTP/2 $\rightarrow$ HTTP/3 QUIC), Head-of-Line Blocking at Application vs Transport Layers, and Payment Gateway API Semantics.

📑 Table of Contents

  1. HTTP Fundamentals & Anatomy of a Request/Response
  2. HTTP Methods: The Idempotency & Safety Matrix
  3. Mastery of HTTP Status Codes
  4. The Evolution of HTTP: 1.0 to 3.0
  5. Evolution Comparison Matrix
  6. Juspay & Payment Systems Real-World Context
  7. High-Yield Interview Questions & Verbatim Answers

1. HTTP Fundamentals & Anatomy of a Request/Response

HTTP (Hypertext Transfer Protocol) is a stateless, application-layer request-response protocol.

HTTP REQUEST ANATOMY: POST /v1/orders HTTP/1.1 <-- Request Line (Method, URI, Version) Host: api.juspay.in <-- Headers (Host, Auth, Content-Type) Authorization: Bearer token_xyz123 Content-Type: application/json Idempotency-Key: e6db1049-365e-4e60-8b5b Content-Length: 42 <-- Empty Line (CRLF) Separator {"amount": 149900, "currency": "INR"} <-- Request Body (Payload) HTTP RESPONSE ANATOMY: HTTP/1.1 201 Created <-- Status Line (Version, Code, Reason) Content-Type: application/json Date: Sun, 30 Aug 2026 06:10:00 GMT Content-Length: 68 <-- Empty Line (CRLF) Separator {"order_id": "ord_9981", "status": "NEW"} <-- Response Body

2. HTTP Methods: The Idempotency & Safety Matrix

Understanding whether a method is Safe or Idempotent is fundamental for designing fault-tolerant distributed payment systems.

┌────────────────────────────────────────────────────────────────────────────────────────┐ │ SAFETY vs IDEMPOTENCY DEFINITIONS │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ SAFE: Does NOT modify any server-side state (Read-Only). Cacheable. │ │ IDEMPOTENT: Making N identical requests produces the EXACT SAME server state as 1. │ │ $f(f(x)) = f(x)$. Retrying over the network is inherently safe. │ └────────────────────────────────────────────────────────────────────────────────────────┘

2.1 Complete HTTP Method Matrix

MethodSafe?Idempotent?Request Body?Standard Semantic Purpose
GET✅ Yes✅ Yes❌ NoRetrieve resource state
HEAD✅ Yes✅ Yes❌ NoRetrieve headers only (check file size / ETag without downloading payload)
OPTIONS✅ Yes✅ Yes❌ NoPre-flight CORS check; query supported HTTP methods on server
POST❌ No❌ No✅ YesCreate new subordinate resource; process transactions
PUT❌ No✅ Yes✅ YesReplace the target resource entirely with provided payload
PATCH❌ No❌ * (Usually)✅ YesPartial update to a resource (e.g., update only user's phone number)
DELETE❌ No✅ Yes❌ (Optional)Delete target resource (deleting an already deleted ID leaves state unchanged)

2.2 Deep Comparison: PUT vs PATCH vs POST

PUT vs PATCH vs POST EXAMPLE Resource: /users/123 -> Current DB Record: {"name": "Om", "email": "om@test.com", "role": "DEV"} 1. POST /users: -> Creates a brand new user with auto-generated ID (e.g., /users/124). -> Calling it 3 times creates 3 distinct users! (Non-Idempotent). 2. PUT /users/123 with body: {"name": "Omkumar"} -> FULL REPLACEMENT: The DB record becomes {"name": "Omkumar", "email": null, "role": null}. -> Calling it 3 times results in the same single modified user. (Idempotent). 3. PATCH /users/123 with body: {"name": "Omkumar"} -> PARTIAL UPDATE: Only the "name" field changes. Result: {"name": "Omkumar", "email": "om@test.com", "role": "DEV"}.

3. Mastery of HTTP Status Codes

Interviewers frequently test subtle distinctions between status codes:

1xx: Informational | 2xx: Success | 3xx: Redirection | 4xx: Client Error | 5xx: Server Error

3.1 2xx Success Codes

  • 200 OK: Standard success for GET, PUT, PATCH. Response contains requested data.
  • 201 Created: Resource successfully created (standard for POST /orders). Includes Location header pointing to new resource.
  • 204 No Content: Action succeeded, but the server deliberately returns an empty body (common for DELETE or PUT updates).

3.2 3xx Redirection Codes

┌────────────────────────────────────────────────────────────────────────────────────────┐ │ 301 vs 302 vs 307 vs 308 EXPLAINED │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ 301 Moved Permanently: │ │ - Target URL has permanently changed. Browsers cache this redirect aggressively. │ │ - Historical Quirk: Browsers often rewrite POST to GET on redirect. │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ 302 Found (Temporary Redirect): │ │ - Temporary move. Browsers do not cache indefinitely. │ │ - Historical Quirk: Browsers often rewrite POST to GET on redirect. │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ 307 Temporary Redirect (HTTP/1.1 Strict): │ │ - Exact same as 302, but STRICTLY GUARANTEES the HTTP Method is NOT changed │ │ (POST stays POST on redirection). │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ 308 Permanent Redirect (HTTP/1.1 Strict): │ │ - Exact same as 301, but STRICTLY GUARANTEES the HTTP Method is NOT changed │ │ (POST stays POST on redirection). │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ 304 Not Modified: │ │ - Conditional GET (`If-None-Match: "etag_hash"` or `If-Modified-Since`). │ │ - Client uses local cache. Zero response body sent, saving network bandwidth. │ └────────────────────────────────────────────────────────────────────────────────────────┘

3.3 4xx Client Error Codes

  • 400 Bad Request: Malformed JSON syntax, invalid schema validation, or missing required fields.
  • 401 Unauthorized vs 403 Forbidden:
    • 401 Unauthorized: Authentication Missing / Invalid ("I do not know who you are"). Missing or invalid Bearer token / API key. Must include WWW-Authenticate header.
    • 403 Forbidden: Authorization Denied ("I know who you are, but you do not have permission to access this resource"). E.g., merchant trying to read another merchant's ledger.
  • 404 Not Found: Resource does not exist at the requested URI.
  • 405 Method Not Allowed: Endpoint exists, but does not support the HTTP verb (e.g., POST on a read-only /health endpoint). Server returns Allow: GET, HEAD header.
  • 409 Conflict: State conflict in the backend. (e.g., Concurrent payment request with the same idempotency key currently in-flight, or editing a stale version in optimistic locking).
  • 429 Too Many Requests: Client exceeded rate-limiting thresholds (Token Bucket / Leaky Bucket). Includes Retry-After: 30 header.

3.4 5xx Server Error Codes

  • 500 Internal Server Error: Unhandled exception, null pointer, or crash in the application code.
  • 502 Bad Gateway vs 503 Service Unavailable vs 504 Gateway Timeout:
┌────────────────────────────────────────────────────────────────────────────────────────┐ │ 502 vs 503 vs 504 (THE PROXY TRIO) │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ [ Client ] <────> [ Reverse Proxy / Nginx / ALB ] <────> [ Upstream Payment Service ] │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ 502 Bad Gateway: │ │ - Nginx connected to upstream service, but upstream returned an INVALID response or │ │ dropped the TCP connection abruptly (e.g., upstream Node/Java process crashed). │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ 503 Service Unavailable: │ │ - Upstream server is alive, but deliberately overloaded or undergoing maintenance. │ │ - Circuit Breaker open, or thread pool saturated. Often includes `Retry-After`. │ ├────────────────────────────────────────────────────────────────────────────────────────┤ │ 504 Gateway Timeout: │ │ - Nginx sent the request to the upstream service, but the upstream failed to reply │ │ within the configured timeout window (e.g., DB query stuck in deadlock). │ └────────────────────────────────────────────────────────────────────────────────────────┘

4. The Evolution of HTTP: 1.0 to 3.0

HTTP/1.0 ────► HTTP/1.1 ────────────► HTTP/2 ──────────────────────► HTTP/3 (1 Conn/Req) (Keep-Alive, Pipeline) (Multiplexing, Binary, HPACK) (QUIC over UDP, 0-RTT)

4.1 HTTP/1.0 & HTTP/1.1 (Keep-Alive & HTTP HoL Blocking)

  • HTTP/1.0: Every single asset (HTML, CSS, image) required establishing a fresh TCP connection (3-way handshake + TLS handshake + Teardown). Extreme latency overhead.
  • HTTP/1.1 Innovations:
    • Connection: keep-alive: Reuses an open TCP connection for sequential requests.
    • Chunked Transfer Encoding: Stream responses without pre-calculating Content-Length.
  • The Core Flaw — Application-Level Head-of-Line (HoL) Blocking:
    • In HTTP/1.1, requests on a single TCP connection must be processed strictly sequentially in FIFO order.
    • If Request #1 (a heavy report generation) is slow, Requests #2 and #3 are blocked behind it on the same socket.
    • Browser Workaround: Browsers opened up to 6 parallel TCP connections per domain, causing connection bloat on servers.
HTTP/1.1 FIFO Processing on 1 TCP Connection: [Req 1 (Slow 2000ms)] ──────► [Req 2 (Fast 10ms - BLOCKED!)] ──────► [Req 3]

4.2 HTTP/2 (Binary Framing, Multiplexing, HPACK)

Published in 2015 (RFC 7540), HTTP/2 overhauled the transport mechanics while keeping HTTP semantics (methods, headers, status codes) identical:

HTTP/2 BINARY FRAMING LAYER ┌────────────────────────────────────────────────────────────────────────────────────────┐ │ HTTP/2 Connection (Single TCP Socket) │ │ ├── Stream 1 (GET /index.html): [HEADERS Frame] [DATA Frame 1] [DATA Frame 2] │ │ ├── Stream 3 (GET /style.css) : [HEADERS Frame] [DATA Frame] │ │ └── Stream 5 (POST /payment) : [HEADERS Frame] [DATA Frame] │ └────────────────────────────────────────────────────────────────────────────────────────┘

Key Innovations:

  1. Binary Framing Layer: Replaces plaintext ASCII with lightweight binary frames (HEADERS, DATA, SETTINGS, RST_STREAM).
  2. True Multiplexing: Multiple bidirectional streams are interleaved concurrently over a single TCP connection. No HTTP HoL blocking.
  3. Stream Prioritization: Critical assets (JS/CSS) can be prioritized ahead of images.
  4. HPACK Header Compression:
    • HTTP/1.1 repeatedly sent huge identical headers (User-Agent, Cookies) on every request.
    • HPACK uses a shared static dictionary (61 common headers) + dynamic table to compress header overhead by over 85%.

The Remaining Bottleneck — Transport-Level (TCP) HoL Blocking:

  • Because HTTP/2 multiplexes all streams over one single TCP stream, if a single TCP packet drops on a lossy cellular network:
    • TCP halts delivery of all multiplexed streams until the missing packet is retransmitted and acknowledged!
    • Under packet loss ($\ge 2%$), HTTP/2 can actually perform worse than HTTP/1.1 with 6 separate TCP connections.

4.3 HTTP/3 & QUIC (UDP Foundation, 0-RTT, Connection Migration)

HTTP/3 solves TCP HoL blocking by ditching TCP entirely and building upon QUIC (Quick UDP Internet Connections) over UDP.

HTTP/2 STACK HTTP/3 STACK ┌────────────────────────┐ ┌────────────────────────┐ │ HTTP/2 Logic │ │ HTTP/3 Logic │ ├────────────────────────┤ ├────────────────────────┤ │ TLS 1.2/1.3 │ │ QUIC (Transport + │ ├────────────────────────┤ │ Native TLS 1.3) │ │ TCP │ ├────────────────────────┤ ├────────────────────────┤ │ UDP │ │ IP │ ├────────────────────────┤ └────────────────────────┘ │ IP │ └────────────────────────┘

Breakthrough Advantages of HTTP/3:

  1. Independent Stream Loss Recovery (Zero HoL Blocking):
    • QUIC handles stream multiplexing at the transport layer. If a packet belonging to Stream 3 drops, Stream 1 and Stream 5 continue receiving data with zero delay.
  2. 0-RTT Connection Resumption:
    • Because QUIC combines transport and cryptographic handshakes into a single protocol, returning clients can send HTTP data in the very first packet (0 Round-Trip Time)!
  3. Connection Migration (Mobile Resilience):
    • TCP connections are bound to the 5-tuple (Client IP, Client Port, Server IP, Server Port, TCP). When a user leaves home and switches from WiFi to 4G/5G, their Client IP changes $\implies$ active TCP connection breaks and must reconnect.
    • QUIC identifies connections via a 64-bit Connection ID (CID) independent of IP address. When switching networks, the payment or video stream continues uninterrupted!

5. Evolution Comparison Matrix

FeatureHTTP/1.1HTTP/2HTTP/3 (QUIC)
Underlying ProtocolTCPTCPUDP (QUIC)
Data FormatPlaintext ASCIIBinary FramesBinary Frames
Multiplexing❌ No (Sequential FIFO)✅ Yes (Single TCP Socket)✅ Yes (Independent QUIC Streams)
App-Level HoL Blocking❌ Present✅ Solved✅ Solved
TCP-Level HoL BlockingN/A (Multiple TCP Conns)❌ Present on packet loss✅ Solved completely
Header Compression❌ None✅ HPACK✅ QPACK
Handshake Latency2–3 RTTs (TCP + TLS)2–3 RTTs (TCP + TLS)1 RTT initial / 0 RTT resumption
Connection Migration❌ Breaks on IP change❌ Breaks on IP change✅ Seamless via Connection ID

6. Juspay & Payment Systems Real-World Context

PAYMENT API ERROR CONTRACT ┌──────────────────────────────────────┬────────────────────────────────────────────────────────┐ │ Response Condition │ Status Code & Handling Strategy │ ├──────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ Payment Created Successfully │ `201 Created` + JSON `{"status": "PENDING"}` │ │ Card Declined (Insufficient balance) │ `200 OK` or `422 Unprocessable` + Domain Error Code │ │ Concurrent Duplicate Transaction │ `409 Conflict` (Existing transaction in-flight) │ │ Upstream Bank Gateway Timeout │ `504 Gateway Timeout` -> Trigger Reconciliation Poller │ │ Rate Limit Exceeded │ `429 Too Many Requests` + `Retry-After: 5` │ └──────────────────────────────────────┴────────────────────────────────────────────────────────┘
  1. 200 OK vs 4xx/5xx in Payment Statuses:
    • Technical failures (e.g., bad token, timeout) use HTTP status codes (401, 504).
    • Business/Domain outcomes (e.g., card declined by issuer bank) are delivered with 200 OK or 422 containing a structured business status: {"status": "DECLINED", "reason": "INSUFFICIENT_FUNDS"}, preventing intermediate HTTP proxies from assuming infrastructure failure.
  2. HTTP/2 & gRPC Internal Microservices:
    • Inside Juspay's payment mesh, microservices communicate over gRPC / HTTP/2 binary streams with protobuf schemas, achieving sub-millisecond serialization and high multiplexed throughput.
  3. Idempotent Retry Policy for 502 / 504 Errors:
    • When a payment SDK receives a 504 Gateway Timeout or 502 Bad Gateway, it must never blindly retry with a fresh ID. It retries with the same Idempotency-Key using exponential backoff with jitter to query or complete the existing transaction.

7. High-Yield Interview Questions & Verbatim Answers

Q1: "What is the difference between PUT and PATCH? Which one is idempotent?"

Spoken Answer:
*"Both PUT and PATCH modify existing resources, but their semantics differ:

  • PUT performs a full replacement of the resource. The client sends the entire representation; any omitted fields are typically reset to null/defaults. Because replacing an entity with the exact same payload N times results in the same final state, PUT is strictly idempotent.
  • PATCH applies a partial update (modifying only the specified fields, leaving untouched fields intact). While many PATCH operations are idempotent, PATCH is technically non-idempotent by RFC specification because patches can include relative mutations (e.g., {"increment_by": 5})."*

Q2: "What is the difference between 401 Unauthorized and 403 Forbidden?"

Spoken Answer:
*"The core difference is Authentication vs Authorization:

  • 401 Unauthorized indicates that the client lacks valid authentication credentials (the server does not know who the client is). The client can authenticate and retry.
  • 403 Forbidden indicates that the client is authenticated and identified, but lacks the necessary permissions to access the target resource (the server knows who you are, but says you are not allowed). Retrying with the same credentials will fail."*

Q3: "What is Head-of-Line (HoL) Blocking, and how does HTTP/2 solve it? Does HTTP/2 eliminate it completely?"

Spoken Answer:
*"In HTTP/1.1, Head-of-Line blocking occurs at the application layer because requests on a single TCP connection must be processed sequentially in FIFO order; a slow response blocks all subsequent responses on that socket.

HTTP/2 solves application-layer HoL blocking by introducing a Binary Framing Layer that multiplexes independent bidirectional streams over a single TCP connection.

However, HTTP/2 does not eliminate HoL blocking completely—it shifts it to the transport layer (TCP). Because all HTTP/2 streams share one TCP connection, if a single TCP segment is dropped, TCP halts delivery of all multiplexed streams until that packet is retransmitted. HTTP/3 solves this by using QUIC over UDP, isolating packet loss to the individual stream."*

Q4: "Explain the difference between 502 Bad Gateway and 504 Gateway Timeout."

Spoken Answer:
*"Both errors originate from a reverse proxy or API gateway acting as an intermediary to upstream backend servers:

  • 502 Bad Gateway means the proxy received an invalid or corrupted response from the upstream server, or the upstream server abruptly terminated the TCP connection (e.g., process crash or port closed).
  • 504 Gateway Timeout means the proxy connected to the upstream server, but the upstream server failed to respond within the configured timeout limit (e.g., slow database query or deadlock)."*