💳 Juspay Interview Preparation: Distributed Webhook Delivery Engine

Role Focus: SDE / Backend / Distributed Systems (Juspay)
Project Theme: Mission-Critical Payment Notification & Event Dispatch Infrastructure
Core Tech Stack: TypeScript, Node.js, Express.js, BullMQ, Redis, PostgreSQL, Docker

📌 1. The 30-Second Elevator Pitch

"Use this when the interviewer asks: 'Tell me about yourself' or 'Briefly summarize your favorite project'."

"I built a Distributed Webhook & Event Dispatch Engine, which is a high-availability system designed for reliable event delivery. For example, in payment processing, when a transaction completes, an event needs to be delivered to the merchant's server so they know the payment has succeeded. But if the merchant's server is down or errors out, that event can get lost—meaning the payment went through, but the merchant is never notified and the user doesn't see their order confirmed.

To solve this problem, I designed a Redis-backed State Machine (Circuit Breaker) with exponential backoff to isolate failing merchant endpoints and eliminate retry storms. I also implemented HMAC-SHA256 payload signing and an immutable PostgreSQL audit ledger so that merchants can verify payload authenticity and every delivery attempt is fully tracked."

🏗️ 2. The 2-to-3 Minute Deep Dive (STAR Method)

1. Situation & Payment Context

In payment gateways like Juspay, transaction execution must be strictly decoupled from downstream merchant webhook notifications:

• Downstream Unreliability: If a merchant's server experiences latency spikes or downtime (e.g., during a flash sale), synchronous HTTP calls block payment threads, exhaust server connection pools, and degrade the core gateway.
• Thundering Herd / Retry Storms: Retrying dead endpoints blindly saturates worker queues and starves healthy merchants (Head-of-Line / Noisy Neighbor blocking).
• Financial Compliance: Payment events require strict tamper-proofing and immutable auditability for dispute resolution and financial reconciliation.

2. Task

Build an enterprise-grade, high-throughput webhook dispatch system guaranteeing:

1. Asynchronous Non-blocking Ingestion: Sub-10ms response to upstream payment switches.
2. Fault Isolation: Automatic failure detection and traffic pausing for failing endpoints.
3. Cryptographic Authenticity: Zero-trust payload signing.
4. Auditability & Observability: End-to-end delivery history and live telemetry.

3. Architecture & Data Flow

Detailed Execution Steps:

1. Non-Blocking Ingestion (Express API):
   • Validates payload and endpoint status.
   • Persists event to PostgreSQL (events table).
   • Enqueues job to BullMQ and returns 200 Queued in < 10ms, freeing the upstream transaction pipeline immediately.
2. Asynchronous Pull (BullMQ Worker Pool):
   • Distributed worker processes pull jobs independently from Redis.
3. Finite State Machine (Redis Circuit Breaker):
   • Before firing an HTTP request, the worker checks the endpoint's state in Redis:
     • Closed: Normal dispatch. If 5 consecutive failures occur, the circuit flips to Open.
     • Open: Fails fast in < 1ms without sending an HTTP request, pausing traffic for a 60-second cooldown to let the merchant recover.
     • Half-Open: Allows a single probe request after 60s. A success heals the circuit to Closed; a failure resets the cooldown.
4. Intelligent Retry Engine:
   • On 5xx errors or network timeouts, BullMQ schedules retries with custom exponential backoff intervals (1s -> 5s -> 15s -> 30s -> 5m -> 1h) up to 6 attempts.
5. Security & Cryptography:
   • Signs payloads with HMAC-SHA256 using the merchant's secret key (x-webhook-signature).
6. Financial Audit Ledger & Live Observability:
   • Logs every attempt (latency, HTTP status, error traces, attempt number) in PostgreSQL.
   • Publishes delivery results over Redis Pub/Sub, broadcasted via WebSockets to a live dashboard.

4. Results & Metrics

• Throughput: 1,080 requests/second sustained under load.
• Latency: 126ms p95 latency across the end-to-end delivery pipeline.
• Reliability: 0% dropped messages; sub-5ms circuit tripping during downstream outages.

🧠 3. Juspay-Specific Technical Deep Dives (System Design Q&A)

Q1: How do you solve the Dual-Write Problem between PostgreSQL and Redis?

Answer: "In standard two-write architectures, if writing to PostgreSQL succeeds but enqueuing to Redis fails (e.g., network partition), the event is orphaned in the database.

To achieve 100% resilience in a financial system like Juspay, we implement the Transactional Outbox Pattern:

1. Store both the event and an outbox record inside the same ACID database transaction in PostgreSQL.
2. A Change Data Capture (CDC) worker (such as Debezium tailing Postgres Write-Ahead Logs / WAL) or an asynchronous poller reads committed outbox entries and guarantees publishing to Redis. This ensures strictly at-least-once delivery with zero state divergence."

Q2: How do you handle Out-of-Order Webhook Delivery in Payments?

(e.g., payment.refunded arriving before payment.captured due to network retries)

Answer: "Network retries naturally introduce the possibility of out-of-order delivery. We handle this on three tiers:

1. Monotonic Sequence / Versioning: Every webhook payload carries a monotonically increasing version or state timestamp (createdAt).
2. Consumer-Side State Machine (FSM): Merchants must validate state transitions against their internal FSM (e.g., rejecting a captured transition if the order is already in refunded terminal state).
3. Partitioned FIFO Queuing: In BullMQ, jobs can be partitioned by transaction ID (tx_id) into sequential queues so that lifecycle events for a single transaction are processed in FIFO order."

Q3: How do you prevent the "Noisy Neighbor" problem where one broken merchant starves others?

Answer: "If Merchant A's servers hang for 30 seconds on every request, worker sockets get tied up, delaying deliveries for Merchant B (Head-of-Line Blocking).

We mitigate this through:

1. Circuit Breaker Fast-Failing: Once Merchant A fails 5 consecutive times, the state flips to Open in Redis. Subsequent events fail fast in < 1ms without allocating TCP sockets.
2. Bulkheading (Queue Isolation): In production, we isolate traffic into separate queues (e.g., Tier-1 high-volume vs Tier-2 standard merchants) or enforce per-merchant BullMQ concurrency limits."

Q4: Why use Redis for the Circuit Breaker instead of in-memory or PostgreSQL?

Answer:

1. Why not In-Memory? Our worker pool is horizontally scaled across multiple Node.js processes and Docker containers. In-memory state in Worker 1 cannot protect Worker 2 from hammering the failing merchant. Redis provides a centralized, synchronized state machine.
2. Why not PostgreSQL? Circuit states are evaluated on every single job dispatch. Querying PostgreSQL on every attempt creates disk I/O bottlenecks and connection pool exhaustion. Redis performs HGETALL and HSET in sub-millisecond in-memory time."

Q5: How does HMAC-SHA256 prevent Replay Attacks and Timing Attacks?

Answer:

1. Authenticity & Integrity: The worker computes HMAC_SHA256(signingSecret, payloadString) and passes it in x-webhook-signature. The merchant computes the same hash over the raw body.
2. Replay Attack Defense: Payloads contain a timestamp (createdAt). Merchants enforce a validity window (e.g., rejecting events older than 5 minutes).
3. Timing-Attack Defense: Verification must use constant-time string comparison (crypto.timingSafeEqual) so attackers cannot infer valid signature bytes via microsecond comparison differences."

Q6: What happens if a merchant server is down for a long time (e.g., several hours)?

Answer: "In my current implementation:

1. Exponential Retries: BullMQ retries the event up to 6 times with increasing delays (1s, 5s, 15s, 30s, 5m, 1h) over approximately 1 hour and 6 minutes.
2. Audit Logging: Every failed attempt (along with its HTTP status, latency, and error message) is inserted into PostgreSQL delivery_attempts.
3. Terminal Failure: After the 6th failed attempt, BullMQ marks the job in the failed state in Redis and stops retrying. The event and all attempt logs remain permanently in PostgreSQL for auditing.

Production Extension: In a large payment gateway like Juspay, after exhausting retries, the event would be moved to a permanent Dead Letter Queue (DLQ) or an admin dashboard where merchants can trigger a manual replay once their servers recover."

📊 4. Architecture Trade-Offs Summary Table

🚀 5. Key Terminology Checklist (Juspay Buzzwords)

• Finite State Machine (FSM) (Closed, Open, Half-Open states)
• Transactional Outbox Pattern & At-Least-Once Delivery
• Idempotency Keys & Deduplication
• Bulkheading & Noisy-Neighbor Isolation
• Head-of-Line (HoL) Blocking Prevention
• Non-blocking Event Loop & I/O Multiplexing
• Cryptographic Tamper-Proofing (HMAC-SHA256 & Timing-Safe Equivalence)
• p95 / p99 Tail Latency Optimization
• Dead Letter Queue (DLQ) & Reconciliation APIs
• Immutable Financial Audit Ledger