06-idempotency-in-payments.md

💳 Module 06: Idempotency in Payment Systems (Juspay Core)

Core Focus: Why Idempotency is Mission-Critical in Financial Systems, The Double-Debit Disaster, Anatomy of Idempotency-Key, Distributed Locking with Redis & Relational Database State Machines, Handling In-Flight Concurrent Collisions (409 Conflict), Payload Hash Verification, and Webhook Deduplication.

📑 Table of Contents

  1. What is Idempotency & Mathematical Foundation
  2. Why Idempotency is Non-Negotiable in Payment Gateways
  3. Architecture of an Idempotent Payment System
  4. Step-by-Step Request Lifecycle & Collision Handling
  5. Database Schema & Implementation Design
  6. Webhook Deduplication & Bank Reconciliation
  7. High-Yield Interview Questions & Verbatim Answers

1. What is Idempotency & Mathematical Foundation

In mathematics and computer science, an operation $f$ is idempotent if applying it multiple times produces the exact same outcome as applying it once:

$$f(f(x)) = f(x) \quad \forall x$$

NON-IDEMPOTENT OPERATION: bankAccount.deduct(100) -> Run 1: Balance = 900 bankAccount.deduct(100) -> Run 2: Balance = 800 (CORRUPTED! Double debited!) IDEMPOTENT OPERATION: paymentEngine.processPayment(txnId: "tx_998", amount: 100) -> Run 1: Deducts 100 paymentEngine.processPayment(txnId: "tx_998", amount: 100) -> Run 2: Returns existing txn receipt, NO deduction!

2. Why Idempotency is Non-Negotiable in Payment Gateways

Juspay processes hundreds of millions of transactions daily across UPI, Credit Cards, Netbanking, and Wallets. In distributed networks, network failures are inevitable.

2.1 The Two-Generals Problem & Network Timeouts

[ User App / Merchant ] ──────── (1) POST /pay (Rs. 1000) ───────► [ Juspay Payment Core ] │ (2) Forward to Bank / NPCI ▼ [ User App / Merchant ] ◄── (4) TIMEOUT! (Socket Dropped) ───────── [ Bank Debits Rs. 1000 ] │ (3) Bank Success ACK (Lost in transit!)

What happened here?

  1. The merchant backend or mobile app sent a charge request.
  2. Juspay instructed the bank to debit ₹1,000.
  3. The bank successfully debited the customer's account and sent a response.
  4. Due to cellular network jitter, the HTTP connection between Juspay and the merchant timed out (504 Gateway Timeout).
  5. The merchant does not know if the payment succeeded or failed!

[!CAUTION] If the merchant's automated retry system or a nervous user clicking "Pay Again" fires a second POST /pay request, a non-idempotent system will charge the user a second time (₹2,000 total). Idempotency guarantees that automatic retries are 100% safe.

2.2 Real-World Failure Scenarios at Juspay Scale

  1. The Double-Click Frenzy: A user on a slow 3G connection taps the "Pay ₹4,999" button twice within 50ms.
  2. Network Flapping / TCP RST: The mobile client drops from 5G to 4G during the bank redirect, prompting the client SDK to retry.
  3. Upstream Bank Gateway Timeouts: The card network takes 15 seconds to reply. The client-side HTTP client times out at 10 seconds and retries.
  4. Duplicate Webhook Delivery: NPCI / Visa sends the exact same transaction success webhook 4 times over 10 minutes to guarantee at-least-once delivery.

3. Architecture of an Idempotent Payment System

IDEMPOTENT PAYMENT ARCHITECTURE [ Merchant / Client SDK ] │ │ Headers: │ - Idempotency-Key: UUIDv4 │ - Authorization: Bearer key │ Body: {"amount": 1000, ...} ▼ ┌─────────────────────────┐ │ Juspay API Gateway │ └────────────┬────────────┘ │ ▼ ┌────────────────────────────────────────────────┐ │ Redis Distributed Lock (SET key NX EX 30) │ │ - Prevents in-flight concurrent execution race │ └────────────────────────┬───────────────────────┘ │ ▼ ┌────────────────────────────────────────────────┐ │ PostgreSQL DB (`idempotency_keys` table) │ │ - UNIQUE constraint on `(merchant_id, key)` │ │ - Tracks status: STARTED -> SUCCESS / FAILED │ └────────────────────────┬───────────────────────┘ │ ▼ [ Core Banking Gateway ]

3.1 The Idempotency-Key Protocol

  • Clients generate a unique string (typically a UUIDv4, e.g., 8f3b6c41-9a72-4d2a-b68e-128e08d62634) and pass it as an HTTP header: Idempotency-Key: 8f3b6c41-9a72-4d2a-b68e-128e08d62634
  • The server scopes idempotency by Tenant / Merchant ID + Idempotency-Key: $$\text{Storage Key} = \text{merchant_id} + \text{":"} + \text{idempotency_key}$$

3.2 The Complete Transaction State Machine

┌────────────────────────┐ │ Client Request │ └───────────┬────────────┘ │ ▼ ┌─────────────────┐ ┌───────┤ Does Key Exist? ├───────┐ │ └─────────────────┘ │ [ NO ] [ YES ] │ │ ▼ ▼ ┌─────────────────────┐ ┌────────────────────┐ │ Insert DB Record: │ │ Check State: │ │ Status = 'STARTED' │ └──────────┬─────────┘ └──────────┬──────────┘ │ │ ┌───────────────────────┼───────────────────────┐ ▼ ▼ ▼ ▼ ┌─────────────────────┐ [ STARTED / PENDING ] [ SUCCESS ] [ FAILED ] │ Execute Downstream │ (Concurrent Race!) (Replay Cache) (Replay Cache) │ Bank / Card Gateway │ │ │ │ └──────────┬──────────┘ ▼ ▼ ▼ │ Return 409 Conflict Return 200 OK Return 4xx/5xx ▼ or poll until done Cached Payload Cached Error ┌─────────────────────┐ │ Update DB Record: │ │ Status = 'SUCCESS' │ │ Cache Response Body │ └─────────────────────┘

4. Step-by-Step Request Lifecycle & Collision Handling

4.1 Case 1: First-Time Execution (Clean Path)

  1. Request arrives with Idempotency-Key: key_123 and payload {"amount": 1000}.
  2. Server computes payload_hash = SHA256(request_body).
  3. Server attempts atomic insert into DB with status = 'STARTED'.
  4. Server makes external HTTP call to Bank API $\rightarrow$ Bank returns {"txn_id": "b_99", "status": "CHARGED"}.
  5. Server updates DB record: status = 'SUCCESS', response_status = 200, response_body = {"txn_id": "b_99", ...}.
  6. Server returns HTTP 200 OK to client.

4.2 Case 2: Concurrent In-Flight Collision (Race Condition)

Time | Thread 1 (T1) - Request A | Thread 2 (T2) - Duplicate Request B (Same Key) ------|------------------------------------|---------------------------------------------- t1 | Arrives with Key: `uuid_abc` | t2 | Acquires Redis Lock / Inserts DB | Arrives with Key: `uuid_abc` t3 | DB Status = `STARTED` | Attempts DB Insert -> Hits UNIQUE Constraint! t4 | Invoking Bank Gateway (Slow 800ms) | Queries DB -> Sees Status == `STARTED` t5 | ... | **Action**: Returns `HTTP 409 Conflict` | | "A transaction with this key is currently in-flight."

Alternative Enterprise Behavior: Thread 2 subscribes to a Redis Pub/Sub channel for uuid_abc with a 5-second timeout. When Thread 1 finishes, it publishes the completed response, allowing Thread 2 to return HTTP 200 without returning an error to the merchant.

4.3 Case 3: Replay After Completion (Cached Response Replay)

  1. Client retries request with Idempotency-Key: key_123 after receiving a client-side timeout.
  2. Server checks DB $\rightarrow$ Record exists with status = 'SUCCESS'.
  3. Server compares SHA256(new_body) == stored_payload_hash $\rightarrow$ Match verified!
  4. Server bypasses the banking gateway entirely.
  5. Server extracts cached_response_body and cached_status_code from DB/Redis and returns it immediately.
  6. Result: Zero duplicate charge, instant response.

4.4 Case 4: Key Reused with Mismatched Payload (Tamper Detection)

[!WARNING] Interviewer Trap: What if a merchant sends Idempotency-Key: key_123 with amount = 100, and 5 minutes later sends Idempotency-Key: key_123 with amount = 500?

  • Handling:
    1. Server hashes the new body: SHA256(amount=500).
    2. Server queries DB for key_123 and finds existing hash SHA256(amount=100).
    3. Hashes do NOT match!
    4. Server immediately rejects the request with HTTP 422 Unprocessable Entity or HTTP 400 Bad Request:
      { "error": "idempotency_key_payload_mismatch", "message": "Idempotency key 'key_123' was previously used with a different request payload." }

5. Database Schema & Implementation Design

5.1 PostgreSQL Table Design with Unique Constraints

CREATE TABLE idempotency_keys ( id BIGSERIAL PRIMARY KEY, merchant_id VARCHAR(64) NOT NULL, idempotency_key VARCHAR(128) NOT NULL, request_path VARCHAR(255) NOT NULL, request_hash VARCHAR(64) NOT NULL, -- SHA-256 of request payload status VARCHAR(32) NOT NULL, -- 'STARTED', 'SUCCESS', 'FAILED' response_code INT, -- HTTP status (200, 201, 400) response_body JSONB, -- Cached JSON response locked_until TIMESTAMP WITH TIME ZONE NOT NULL,-- Distributed lock expiry created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), -- MANDATORY UNIQUE CONSTRAINT FOR ATOMICITY: CONSTRAINT uk_merchant_idempotency_key UNIQUE (merchant_id, idempotency_key) ); CREATE INDEX idx_idempotency_lookup ON idempotency_keys(merchant_id, idempotency_key);

5.2 Redis Distributed Lock + Atomic Pipeline (Pseudocode)

import redis import hashlib import json def process_payment_idempotent(merchant_id, idempotency_key, payload): redis_client = redis.Redis() lock_key = f"lock:idemp:{merchant_id}:{idempotency_key}" payload_hash = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest() # Step 1: Attempt to acquire Redis Lock for 30 seconds acquired = redis_client.set(lock_key, "LOCKED", nx=True, ex=30) if not acquired: # Another concurrent thread is currently processing this exact request return {"status_code": 409, "body": {"error": "Concurrent request in-flight. Please wait."}} try: # Step 2: Query DB for existing key record record = db.query_idempotency_key(merchant_id, idempotency_key) if record: # Case 4: Verify payload hash matches if record.request_hash != payload_hash: return {"status_code": 422, "body": {"error": "Payload mismatch for existing idempotency key."}} # Case 3: Return cached response if already completed if record.status in ('SUCCESS', 'FAILED'): return {"status_code": record.response_code, "body": record.response_body} # Step 3: Insert initial 'STARTED' record in DB db.insert_idempotency_started(merchant_id, idempotency_key, payload_hash) # Step 4: Execute Sensitive Bank Call bank_response = call_upstream_bank_gateway(payload) # Step 5: Persist Final Response & Update Status to 'SUCCESS' db.update_idempotency_success( merchant_id=merchant_id, idempotency_key=idempotency_key, response_code=200, response_body=bank_response ) return {"status_code": 200, "body": bank_response} finally: # Step 6: Always release the distributed lock redis_client.delete(lock_key)

6. Webhook Deduplication & Bank Reconciliation

Payment gateways receive asynchronous webhooks from banks confirming transaction statuses.

WEBHOOK DEDUPLICATION FLOW [ Bank / NPCI ] ─── Webhook (txn_id: 9981, status: SUCCESS) ───► [ Juspay Webhook Ingress ] │ Atomic Update with Guard: UPDATE orders SET status = 'SUCCESS' WHERE id = 9981 AND status = 'PENDING'; │ ┌─────────────────┴─────────────────┐ ▼ ▼ [ Rows Affected = 1 ] [ Rows Affected = 0 ] First Delivery: Process Duplicate Delivery: merchant fulfillment Ignore & Return HTTP 200
  • Rule: All Webhook consumers must be idempotent. If a webhook for the same transaction arrives 5 times, only the first processing triggers order fulfillment.

7. High-Yield Interview Questions & Verbatim Answers

Q1: "What is Idempotency and why is it essential in a payment gateway like Juspay?"

Spoken Answer:
*"Idempotency means that performing the same operation multiple times produces the exact same side effects and final system state as performing it once ($f(f(x)) = f(x)$).

In payment systems, it is essential because distributed networks are unreliable. When a client initiates a payment and experiences a network timeout (e.g., dropped TCP connection or 504 Gateway Timeout), the client cannot know whether the upstream bank debited the user or failed.
Without idempotency, client retries would result in double debits (charging the customer multiple times). Idempotency ensures that retrying an operation with the same Idempotency-Key safely returns the cached transaction outcome without re-executing the financial charge."*

Q2: "How do you handle two identical payment requests with the same Idempotency-Key arriving simultaneously at the exact same millisecond?"

Spoken Answer:
*"To handle concurrent in-flight collisions:

  1. We acquire a distributed lock in Redis using SET lock_key unique_id NX EX 30 or rely on a database UNIQUE(merchant_id, idempotency_key) constraint.
  2. The first thread successfully acquires the lock, writes an initial record with status STARTED, and proceeds to invoke the banking switch.
  3. The second concurrent thread fails to acquire the lock (or hits a DB unique constraint violation) and detects that the transaction status is STARTED (in-flight).
  4. The server responds to the second request with HTTP 409 Conflict indicating that an identical request is already being processed, or alternatively holds the request and polls a Redis Pub/Sub channel for up to a short timeout window to return the completed response."*

Q3: "What happens if an API client sends the same Idempotency-Key but with a different payment amount or currency?"

Spoken Answer:
"When a request arrives, we compute a cryptographic hash of the normalized request payload (e.g., SHA-256(body)).
When checking an existing idempotency key in the database, we compare the incoming payload hash with the stored hash. If they do not match, this indicates client misbehavior or parameter tampering. The server must immediately reject the request with HTTP 422 Unprocessable Entity or HTTP 400 Bad Request with an explicit error message stating that the idempotency key cannot be reused with modified parameters."

Q4: "Where do you store idempotency keys and how do you handle their lifecycle / TTL?"

Spoken Answer:
*"We store idempotency keys in a high-availability relational database (like PostgreSQL) for strict ACID guarantees and unique constraints, alongside a Redis caching layer for high-throughput lookups.

Idempotency keys should not be retained forever. In payment architectures, idempotency keys typically have an agreed TTL of 24 to 48 hours. After 48 hours, financial transactions transition to settled states, and any subsequent interaction with that order uses dedicated order query or refund endpoints. Old idempotency records are partitioned by date and cleaned up via automated database table partitioning and Redis key expiry."*