🌐 Module 11: REST API Architecture & Systems Design for High-Scale Payments

Targeted for: Juspay Elimination Round, Technical Interviews, Hackathon Part B System Discussions, and High-Scale Backend Engineering.
Core Focus: Roy Fielding's Architectural Constraints, Richardson Maturity Model, HTTP Semantics & RFC Specifications, Idempotency Engineering, Optimistic Concurrency Control (ETags), Keyset vs Offset Pagination, API Versioning Strategies, Webhook Delivery Guarantees, Distributed Rate Limiting, and Juspay-Specific Payment Gateway Case Studies.

📑 Table of Contents

1. Foundational REST Architecture & Fielding Constraints
   • 1.1 What REST Is and What It Is Not
   • 1.2 The 6 Architectural Constraints
   • 1.3 The Uniform Interface Sub-Constraints & HATEOAS
2. The Richardson Maturity Model (RMM)
   • 2.1 The 4 Levels (Level 0 to Level 3)
   • 2.2 Why Real-World Payment Gateways Stop at Level 2
3. REST vs Alternative Architectural Paradigms
   • 3.1 REST vs GraphQL vs gRPC vs WebSockets
   • 3.2 The Juspay Microservices Protocol Split (North-South vs East-West)
4. Resource Modeling & URI Design Principles
   • 4.1 Plural Nouns, Hierarchies, and Path Depth
   • 4.2 Handling Non-CRUD Business Actions (Refund, Capture, Cancel, Verify)
5. Exhaustive HTTP Method Semantics: Safety, Idempotency & RFC Specifications
   • 5.1 Safe vs Idempotent vs Non-Idempotent Matrix
   • 5.2 PUT vs PATCH: JSON Merge Patch (RFC 7386) vs JSON Patch (RFC 6902)
   • 5.3 Payloads in GET and DELETE: The RFC 9110 Reality
6. HTTP Status Codes in Payment Gateways: Precision & Nuance
   • 6.1 2xx Success Semantics (200, 201 with Location, 202 Accepted, 204)
   • 6.2 3xx Redirections: 301/302 vs 307/308 (Why Payment POST Fails on 302)
   • 6.3 4xx Client Failures: 400 vs 422, 401 vs 403, 404 vs 410, 409, 412, 415, 429
   • 6.4 5xx Upstream Bank Failures: 500 vs 502 vs 503 vs 504
   • 6.5 Standardized Error Payloads: RFC 7807 (Problem Details)
7. High-Scale Idempotency Implementation in REST APIs
   • 7.1 The Two-Generals & Network Timeout Dilemma
   • 7.2 Architecture of an Idempotency Filter (Redis Lock + DB Store)
   • 7.3 Payload Hash Mismatch Detection & Concurrency Edge Cases
8. Optimistic Concurrency Control via HTTP Headers (ETags)
   • 8.1 The Lost Update Problem in Payment States
   • 8.2 Strong vs Weak ETags (If-Match vs If-None-Match)
9. Pagination for High-Volume Ledgers: Keyset (Cursor) vs Offset
   • 9.1 Why Offset Pagination Degrades to $O(N)$ and Duplicates Records
   • 9.2 Keyset / Cursor Pagination Mechanics & DB Index Seek
10. Distributed Rate Limiting & Throttling
    • 10.1 Algorithms: Token Bucket, Leaky Bucket, Sliding Window
    • 10.2 Standard Headers (Retry-After, X-RateLimit-*)
    • 10.3 Distributed Redis Lua Atomic Rate Limiter
11. API Versioning Strategies in Production
    • 11.1 URI Path vs Custom Header vs Content Negotiation vs Query Param
    • 11.2 The Stripe/Juspay Event Adapter & Backward Compatibility Pattern
12. Webhook Architecture: Reliable Asynchronous REST Callbacks
    • 12.1 Webhooks vs Long Polling vs Server-Sent Events
    • 12.2 Security: HMAC-SHA256 Signatures & Anti-Replay Timestamps
    • 12.3 Delivery Guarantees: Exponential Backoff, Jitter & Dead Letter Queues
13. API Security, Authentication & Hardening
    • 13.1 Publishable vs Secret API Keys (PCI-DSS Scoping)
    • 13.2 Mutual TLS (mTLS) in Core Banking Interfacing
    • 13.3 Request Signing (Method, URI, Timestamp, Digest)
14. Juspay-Specific Interview Scenarios & Verbatim Defense

1. Foundational REST Architecture & Fielding Constraints

1.1 What REST Is and What It Is Not

• What REST Is: REpresentational State Transfer. An architectural style formulated by Dr. Roy Fielding in his 2000 doctoral dissertation (Architectural Styles and the Design of Network-based Software Architectures).
• What REST Is Not:
  • REST is NOT a protocol. It is not a standard like HTTP, TCP, or SOAP.
  • REST is NOT tied exclusively to HTTP, though in 99.9% of software engineering implementations, HTTP is the underlying application protocol used.
  • REST is NOT simply returning JSON from a URL. A server returning JSON from POST /getUserDetails is an RPC-over-HTTP API, not a RESTful API.

1.2 The 6 Architectural Constraints

Fielding proved that an architecture adhering to these 6 constraints achieves scalability, simplicity, modifiability, visibility, portability, and reliability:

1. Client-Server Separation:

   • The user interface and user state concerns are separated from the data storage and business logic concerns.
   • Impact: Frontends (React web apps, Android SDKs, iOS apps) can evolve independently from backend payment microservices.

2. Statelessness (Crucial Interview Topic):

   • Definition: Every request from client to server must contain all the context, authentication credentials, and state parameters necessary for the server to understand and process the request.
   • The server must never store client session context in server memory (e.g., no in-memory Java HttpSession pinned to a single server).
   • Payment Relevance: If Server A handles request 1 and dies, Server B must be able to handle request 2 immediately because all auth (Bearer token) and resource identifiers are inside the request itself. This enables horizontal elasticity behind load balancers.

3. Cacheability:

   • Every response must explicitly or implicitly label itself as cacheable or non-cacheable (Cache-Control: no-store, no-cache, ETag, Expires).
   • Payment Relevance: Read requests (e.g., supported payment methods GET /v1/merchants/{id}/payment-methods) are heavily cached at CDNs/Redis. Payment mutations (POST /v1/payments) must strictly send Cache-Control: no-store, private to prevent intermediary proxies from caching sensitive transaction data.

4. Layered System:

   • A client cannot tell whether it is connected directly to the end application server or to an intermediary (API Gateway, Reverse Proxy, CDN, Hardware Load Balancer).
   • Payment Relevance: Juspay places NGINX/Envoy API gateways in front of merchant traffic to handle TLS termination, DDoS mitigation, mTLS validation, and distributed rate limiting before requests touch payment processing engines.

5. Uniform Interface:

   • The cornerstone constraint that differentiates REST from all other network paradigms. Elaborated in Section 1.3.

6. Code on Demand (Optional Constraint):

   • Servers can temporarily extend or customize client functionality by transferring executable code (e.g., compiled WebAssembly, JavaScript snippets like Juspay's hyper-sdk.js or Stripe.js injected into merchant checkout pages).

1.3 The Uniform Interface Sub-Constraints & HATEOAS

Fielding defined 4 sub-requirements for a Uniform Interface:

1. Identification of Resources:
   • Resources are conceptually identified using stable URIs (https://api.juspay.in/v1/orders/ord_1024). The URI identifies the resource, not its internal database table name or representation format.
2. Manipulation of Resources through Representations:
   • When a client holds a representation of a resource (including headers), it has enough information to modify or delete the resource on the server (given adequate permissions).
3. Self-Descriptive Messages:
   • Each message includes enough metadata for the receiver to know how to parse it.
   • Content-Type: application/json; charset=utf-8 specifies how to deserialize the payload.
   • Content-Encoding: gzip informs the receiver how to decompress the body.
4. HATEOAS (Hypermedia as the Engine of Application State):
   • A client interacts with a network application entirely through hypermedia provided dynamically by the server.
   • The client does not hardcode URL routes for future states; the server tells the client what states and transitions are legal right now via embedded hypermedia links.

If the order transitions to PAID, a subsequent GET will omit "pay" and "cancel" links and instead present a "refund" link.

2. The Richardson Maturity Model (RMM)

Leonard Richardson categorized web services into 4 progressive tiers of REST compliance:

2.2 Why Real-World Payment Gateways Stop at Level 2

Interviewers at Juspay frequently ask: "Why don't Stripe, Razorpay, or Juspay implement Level 3 (HATEOAS)?"

1. Client-Side Coupling to Strong Typed SDKs:
   • Modern payment integration is done via strongly typed SDKs (Java, Python, Node, Go, Swift, Kotlin). Merchants don't dynamically parse links; they invoke SDK functions (juspay.orders.create(...)).
2. Payload Bloat & Mobile Latency:
   • Adding _links metadata to millions of micro-transactions inflates JSON payload size by 30-50%, increasing serialization overhead, network bandwidth, and packet fragmentation on poor 3G/4G networks in India.
3. UI Determinism:
   • Frontend developers need to build static UI views (e.g., an iOS checkout button). If the UI depends on dynamic HATEOAS discovery, frontend logic becomes overly complex without tangible benefits.

3. REST vs Alternative Architectural Paradigms

3.1 REST vs GraphQL vs gRPC vs WebSockets

3.2 The Juspay Microservices Protocol Split (North-South vs East-West)

• North-South (External $\rightarrow$ Gateway): REST over HTTPS is used because thousands of external merchant developers consume it across dozens of languages. Universal debugging (cURL, Postman), standard HTTP status codes, and HTTP-native caching are mandatory.
• East-West (Internal Microservices): gRPC over HTTP/2 is used internally because Protobuf serialization is drastically faster than JSON parsing, connections are multiplexed, and strict code-generated interfaces eliminate runtime schema mismatch bugs.

4. Resource Modeling & URI Design Principles

4.1 Plural Nouns, Hierarchies, and Path Depth

1. Always Use Plural Nouns for Collections:
   • GET /v1/orders (List orders)
   • POST /v1/orders (Create order)
   • GET /v1/orders/ord_101 (Retrieve single order)
   • Never use verbs: GET /v1/getAllOrders or POST /v1/createPayment are anti-patterns.
2. Hierarchical Relationships (Parent $\rightarrow$ Child):
   • GET /v1/merchants/mer_99/transactions (All transactions under merchant 99)
   • GET /v1/orders/ord_101/payments/pay_505 (Specific payment attempt for order 101)
3. The "Rule of Two Levels" (Preventing URI Explosion):
   • When resource nesting exceeds 2 levels (e.g., /merchants/{m_id}/orders/{o_id}/payments/{p_id}/refunds/{r_id}), URI readability and routing become brittle.
   • Best Practice: Flatten child resources that have globally unique identifiers:
     • Instead of: GET /merchants/1/orders/2/payments/3/refunds/4
     • Use: GET /v1/refunds/rf_4 (Refund ID is globally unique).
     • Filter via query parameter if querying a collection: GET /v1/refunds?payment_id=pay_3.

4.2 Handling Non-CRUD Business Actions (Refund, Capture, Cancel, Verify)

In a pure CRUD world, you only create, read, update, or delete. But payment systems are driven by state machine transitions:

• Authorize $\rightarrow$ Capture
• Payment $\rightarrow$ Refund
• Order $\rightarrow$ Cancel
• VPA (UPI ID) $\rightarrow$ Verify

How do you model these in REST without violating Roy Fielding's principles?

5. Exhaustive HTTP Method Semantics: Safety, Idempotency & RFC Specifications

5.1 Safe vs Idempotent vs Non-Idempotent Matrix

5.2 PUT vs PATCH: JSON Merge Patch (RFC 7386) vs JSON Patch (RFC 6902)

Common Juspay Interview Trap: "Is PATCH idempotent? What is the difference between PUT and PATCH?"

1. PUT Semantics (Full Replacement):

   • Client sends the entire new representation.
   • If a field is omitted from the request body, the server must overwrite that field with null or its default value!
   • PUT is strictly idempotent: replacing resource X with state Y multiple times results in state Y.

2. PATCH Semantics (Partial Delta Update):

   • Client sends only the fields that need mutation.
   • Why PATCH is NOT guaranteed to be idempotent:
     • If the patch contains an operation like "increment_attempts" or "append_to_log", executing it 3 times results in +3 increments.

5.3 Payloads in GET and DELETE: The RFC 9110 Reality

• GET with Body:
  • RFC 9110 states: "Although request bodies on GET are allowed by syntax, they have no defined semantics and implementations SHOULD reject or ignore them."
  • Production Reality: Intermediaries (AWS CloudFront, Akamai, Cloudflare, NGINX) frequently strip or drop request bodies on GET requests. Never design a REST API requiring a body in GET.
• DELETE with Body:
  • RFC allows a body in DELETE requests. However, some HTTP client libraries (e.g., older Android OkHttp or Angular $http) historically threw errors when sending payloads with DELETE. Use query parameters for DELETE wherever possible.

6. HTTP Status Codes in Payment Gateways: Precision & Nuance

6.1 2xx Success Semantics

• 200 OK: Standard synchronous success response containing representation.
• 201 Created: Resource successfully created. RFC Mandate: MUST return a Location header pointing to the newly minted resource.
  HTTP/1.1 201 Created Location: /v1/payments/pay_008129 Content-Type: application/json {"payment_id": "pay_008129", "status": "INITIALIZED"}
• 202 Accepted: The Asynchronous Pattern. The request has been accepted for processing, but processing is incomplete. Crucial for asynchronous bank rails (e.g., initiating a UPI Mandate or NEFT payout). Returns a polling URL or status token.
• 204 No Content: Action succeeded, but there is no entity body to return (e.g., DELETE /v1/tokens/tok_123 or acknowledging a webhook test ping).

6.2 3xx Redirections: 301/302 vs 307/308 (Why Payment POST Fails on 302)

Crucial Payment System Interview Question: "Why will an API gateway break payment processing if it redirects HTTP to HTTPS using status 302 or 301?"

[!CAUTION] If a merchant calls http://api.juspay.in/v1/payments via POST and the load balancer responds with 301 or 302 Moved, the client library will redirect to https://... via GET, dropping the JSON payment body! The gateway must use 308 Permanent Redirect or force HTTPS at the DNS/HSTS layer.

6.3 4xx Client Failures

• 400 Bad Request: Generic client syntax failure (malformed JSON syntax, invalid JSON tokens).
• 401 Unauthorized: Authentication is missing or invalid. The user has not established their identity. RFC 9110 requires the server to return a WWW-Authenticate header specifying the challenge scheme.
• 403 Forbidden: The client is authenticated, but does not possess permission to perform this action on this resource (e.g., Merchant A trying to fetch Merchant B's order, or using a sandbox API key against the production live cluster).
• 404 Not Found: Resource URI does not exist.
• 405 Method Not Allowed: E.g., sending DELETE /v1/orders. RFC requires returning the Allow header listing permitted methods (Allow: GET, POST).
• 409 Conflict: Request conflicts with current state of the server.
  • Payment Use-Case 1: Calling POST /v1/payments with an Idempotency-Key that is currently in-flight by a parallel thread.
  • Payment Use-Case 2: Attempting to refund an order that is already in state REFUNDED.
• 410 Gone: Resource previously existed, but has been deliberately deleted or expired permanently (e.g., an expired one-time payment link). Crawlers and clients must cease querying this URI.
• 412 Precondition Failed: Used in conjunction with If-Match ETags. Signals that the resource was modified by another actor before this request could be applied.
• 415 Unsupported Media Type: Client sent a body with Content-Type: text/xml when server expects application/json.
• 422 Unprocessable Entity (RFC 4918 / RFC 9110):
  • The JSON is syntactically valid (not a 400), but contains semantic business validation failures (e.g., "amount": -500, "currency": "XYZ", "expiry_month": 15).
• 429 Too Many Requests: Client exceeded rate limits. Must include Retry-After: <seconds> header.

6.4 5xx Upstream Bank Failures: 500 vs 502 vs 503 vs 504

When Juspay acts as a payment switch between a merchant and banking rails (HDFC, ICICI, NPCI), 5xx errors have strictly distinct meanings:

• 500 Internal Server Error: An unhandled exception or crash inside Juspay's own application code (e.g., NullPointerException, unhandled Rust panic).
• 502 Bad Gateway: Juspay's edge proxy (NGINX/Envoy) received an invalid, malformed, or TCP connection reset from our internal backend container or bank switch.
• 503 Service Unavailable: Juspay's service is temporarily overloaded, under maintenance, or the Circuit Breaker to the bank has tripped OPEN due to high bank error rates.
• 504 Gateway Timeout: Juspay sent the transaction request to the upstream bank (or payment network), but the bank did not return a response within the configured socket timeout (e.g., 15-30 seconds).
  • Critical Payment Semantics: A 504 NEVER means the payment failed! The customer's account may have been debited. The transaction status is PENDING until reconciled.

6.5 Standardized Error Payloads: RFC 7807 (Problem Details)

Professional payment APIs do not return arbitrary error JSON. They follow RFC 7807 (Problem Details for HTTP APIs) (Content-Type: application/problem+json):

7. High-Scale Idempotency Implementation in REST APIs

7.1 The Two-Generals & Network Timeout Dilemma

The merchant's client times out after 10 seconds. The merchant does not know:

1. Did the request fail before reaching Juspay?
2. Did Juspay fail before reaching the Bank?
3. Did the Bank succeed, but the response packet was dropped on the wire?

If the merchant retries blindly, they risk double-debiting the customer.

7.2 Architecture of an Idempotency Filter (Redis Lock + DB Store)

7.3 Payload Hash Mismatch Detection & Concurrency Edge Cases

Juspay Interview Question: "What if a merchant sends the same Idempotency-Key twice, but changes the amount from ₹100 to ₹500 in the second request?"

• Answer: The server must compute and store a SHA-256 digest of the request payload alongside the idempotency key.
• If incoming_key == stored_key AND incoming_payload_hash != stored_payload_hash:
  • The server must immediately reject the request with 422 Unprocessable Entity or 400 Bad Request with error code: IDEMPOTENCY_KEY_PAYLOAD_MISMATCH.
  • Replaying a different payload with an old key indicates a catastrophic client-side programming error or malicious tampering attempt.

8. Optimistic Concurrency Control via HTTP Headers (ETags)

8.1 The Lost Update Problem in Payment States

8.2 Strong vs Weak ETags (If-Match vs If-None-Match)

To eliminate the Lost Update problem without locking the database row during read operations:

1. The Server generates an ETag on read:
   HTTP/1.1 200 OK ETag: "v1_hash_9a8b7c6d" Content-Type: application/json {"order_id": "ord_1", "status": "PENDING", "version": 3}
2. The Client sends the conditional header on mutation:
   PATCH /v1/orders/ord_1 HTTP/1.1 If-Match: "v1_hash_9a8b7c6d" Content-Type: application/json {"status": "CANCELLED"}
3. Server atomic validation:
   • UPDATE orders SET status = 'CANCELLED', version = version + 1 WHERE id = 'ord_1' AND version = 3;
   • If row count == 0, the record was modified concurrently. Server returns 412 Precondition Failed.
   • Agent B's request fails cleanly, prompting them to re-fetch the latest state.

9. Pagination for High-Volume Ledgers: Keyset (Cursor) vs Offset

9.1 Why Offset Pagination Degrades to $O(N)$ and Duplicates Records

Offset pagination (GET /v1/transactions?limit=20&offset=100000) relies on:

1. Performance Catastrophe:
   • The database engine must scan and traverse all $100,020$ index entries and discard the first $100,000$ in memory. Query latency degrades linearly $O(N)$ as page numbers increase.
2. Data Inconsistency Under Concurrent Writes:
   • In payment systems, thousands of transactions are inserted every minute.
   • If a new transaction is inserted while the user is flipping from Page 1 to Page 2, all rows shift down by 1. Page 2 will return the last row of Page 1 again (Duplicate Record).
   • If a transaction is deleted/purged, a record will be permanently skipped.

9.2 Keyset / Cursor Pagination Mechanics & DB Index Seek

Used by Juspay, Stripe, and modern high-throughput ledgers.

Database Query Generated:

• Guarantees:
  • $O(1)$ constant execution time regardless of whether fetching page 1 or page 10,000.
  • Immune to concurrent inserts: tx_099 serves as an immutable navigational anchor.

10. Distributed Rate Limiting & Throttling

10.1 Rate Limiting Algorithms

10.2 Standard Headers (Retry-After, X-RateLimit-*)

When throttling a client, payment APIs return:

10.3 Distributed Redis Lua Atomic Rate Limiter

Why Lua is mandatory: Avoids race conditions (check-then-set) between distributed NGINX/API gateway nodes:

11. API Versioning Strategies in Production

11.1 The 4 Strategies Compared

11.2 The Stripe/Juspay Event Adapter & Backward Compatibility Pattern

How does a payment gateway support 100,000 merchants across 50 API versions without running 50 different microservices?

• The internal microservices always execute on the latest schema.
• A pipeline of deterministic transformation gates translates older request formats forward, and translates newer response payloads backward to the merchant's pinned API version.

12. Webhook Architecture: Reliable Asynchronous REST Callbacks

12.1 Webhooks vs Long Polling vs Server-Sent Events

• Long Polling: Inefficient for 100,000 concurrent transactions; exhausts merchant and gateway socket descriptors.
• Server-Sent Events / WebSockets: Excellent for frontend checkout UI timer updates, but unsuited for server-to-server transactional settlement.
• Webhooks (Reverse REST): The payment gateway makes an HTTP POST request to the merchant's registered HTTPS URL whenever a background payment event occurs (order.successful, refund.failed).

12.2 Security: HMAC-SHA256 Signatures & Anti-Replay Timestamps

Because merchant webhook endpoints are exposed to the public Internet, attackers can forge fake payment.success webhooks to obtain free merchandise.

How Juspay Protects Webhooks:

Verification Algorithm:

1. Merchant extracts timestamp $t$ (1726500120) and signature $v1$.
2. Replay Attack Defense: Merchant checks $|t_{current} - t| \le 300\text{ seconds}$. If older than 5 minutes, reject immediately!
3. Merchant computes: $$\text{Expected Signature} = \text{HMAC-SHA256}(\text{key} = \text{webhook_secret}, \text{data} = t + "." + \text{raw_payload})$$
4. Constant-Time Comparison:
   # Must use constant-time comparison to prevent Timing Attacks hmac.compare_digest(computed_signature, received_v1_signature)

12.3 Delivery Guarantees: Exponential Backoff, Jitter & Dead Letter Queues

Network failures are inevitable. If a merchant's server returns 500 or times out:

$$\text{Retry Delay} = \min(\text{MaxDelay}, \text{BaseDelay} \times 2^{\text{attempt}}) + \text{Full_Jitter}$$

• At-Least-Once Delivery: Merchants must design their webhook handlers to be idempotent (recording processed event_id in a database with unique constraint).

13. API Security, Authentication & Hardening

13.1 Publishable vs Secret API Keys (PCI-DSS Scoping)

Payment platforms segregate credentials into two strict privilege tiers:

13.2 Mutual TLS (mTLS) in Core Banking Interfacing

While public merchant APIs use one-way TLS (client verifies server certificate), communication between Juspay and Banking Switches (NPCI, Visa, HDFC) enforces mTLS (Mutual TLS):

• The Bank presents its certificate to Juspay.
• Juspay presents its cryptographically signed client certificate to the Bank.
• Both parties cryptographically authenticate each other at Layer 4 before any Layer 7 REST/HTTP request is transmitted.

14. Juspay-Specific Interview Scenarios & Verbatim Defense

Question 1: "Design the REST API for a high-volume UPI Collect Payment flow. Walk through request, response, and eventual settlement."

Verbatim Architecture Answer: "I would design this using an Asynchronous REST + Webhook Event Pattern:

1. Initiation: The client issues: POST /v1/payments Headers: Idempotency-Key: <UUIDv4>, Authorization: Bearer <sk_key> Body: {"order_id": "ord_101", "vpa": "customer@okhdfcbank", "amount": 150000, "currency": "INR"}
2. Immediate Gateway Response: Because NPCI and the user's UPI PSP app take up to 60-90 seconds to authorize, the gateway cannot hold an open HTTP socket. The gateway writes an IN_PROGRESS record, dispatches a message to the Kafka UPI Switch queue, and responds immediately with: HTTP/1.1 202 Accepted Body: {"payment_id": "pay_001", "status": "PENDING", "poll_url": "/v1/payments/pay_001"}
3. Settlement Notification: Once the customer enters their UPI PIN and NPCI delivers the bank callback, Juspay dispatches an asynchronous Webhook to the merchant's endpoint: POST https://merchant.com/webhook with X-Juspay-Signature: t=...,v1=... Body: {"event": "payment.succeeded", "data": {"payment_id": "pay_001", "amount": 150000}} The merchant validates the HMAC-SHA256 signature, records payment_id idempotently, and responds with 200 OK."

Question 2: "An upstream bank times out after 15 seconds. Should our REST API return 500, 502, 504, or 200 with status=PENDING? Why?"

Verbatim Architecture Answer: "Returning 500 Internal Server Error is wrong because our server didn't crash. Returning 502 Bad Gateway is wrong because the upstream bank didn't send an invalid response; it sent no response at all.

The correct architectural choice depends on whether the endpoint is synchronous or asynchronous:

• If an asynchronous payment resource model is used (recommended), the server should return 200 OK (or 202 Accepted) with "status": "PENDING", because a payment timeout is not a terminal failure—the money may still be debited from the user's account. Our background reconciliation cron or NPCI reversal job will determine the final state.
• If the API contract strictly requires standard HTTP status reporting for a transport gateway failure, it must return 504 Gateway Timeout, accompanied by an explicit RFC 7807 payload stating "status": "PENDING", "retryable": false. The client must be instructed never to treat a 504 as a failed payment without running a status inquiry (GET /v1/payments/{id})."

Question 3: "Why is PUT idempotent, while PATCH is non-idempotent? Give a concrete code example."

Verbatim Architecture Answer: "PUT represents a complete replacement of the target resource. By definition of set equality: If I execute PUT /v1/customers/1 with payload {"name": "Aditi", "age": 28}, executing this 1 time or 100 times results in the exact same resource state: name = 'Aditi', age = 28.

PATCH represents a partial delta modification. RFC 5789 does not guarantee idempotency because a patch payload can describe relative mutation operations rather than absolute values.

Concrete Example: Under RFC 6902 (JSON Patch), a client can issue:

[ { "op": "add", "path": "/failed_attempts", "value": 1 } ]

Every time the server executes this PATCH request, failed_attempts is incremented by 1. Executing it 5 times results in failed_attempts = 5. Thus, $f(f(x)) \ne f(x)$, violating idempotency."

Question 4: "Explain how to implement distributed rate limiting without race conditions in a cluster of 50 API gateway nodes."

Verbatim Architecture Answer: "In a distributed cluster, a naive implementation of:

1. GET count
2. If count < limit then SET count = count + 1 creates a classic Time-of-Check to Time-of-Use (TOCTOU) race condition where concurrent requests from multiple gateway nodes interleave, allowing traffic to blow past the rate limit.

To solve this, we must execute the rate limiting check atomically in an in-memory datastore like Redis. We use a Redis Lua script or Redis INCR + EXPIRE pipeline. Because Redis is single-threaded in its command execution engine, the Lua script executes from start to finish without interleaving. If the incremented counter exceeds the threshold, the gateway rejects the request immediately with 429 Too Many Requests, including Retry-After: <seconds> and X-RateLimit-* headers, preventing the request from ever consuming backend worker threads or database connections."