Backend Engineering

Flashsale System

A backend for flash sales thousands of buyers, limited stock,one second. Built twice: synchronously, then queue-based.

Focus: Performance · Concurrency · ResilienceIn collaboration with: Bayan Abd El Bary (bayan2002)

RoleBackend Engineer • System Architect

ComplexityDistributed state • Atomic stock • Controlled concurrency

ImpactStable API under load

01 / Engineering challenge

Keeping stock correct under load.

Two implementations of the same flash-sale backend: one completes the purchase inside the request, the other reserves stock and queues the work. I compared them after auditing 21 defects in the original implementation and report.

Stock reservationIdempotencyBackpressure

Peak-load benchmark / HTTP errors

0.00%

HTTP error rate under peak load

300 virtual users · 108,844 requests

HTTP error rate dropped from 4.65% in the synchronous run to 0.00% in the queue-based run under the same load profile.

02 / Controlled A/B

Acknowledgement and completion.

The queue returned an earlier response and recorded fewer request errors. It also took longer to fulfill orders and completed fewer orders per second in this closed-load test.

300 peak VUs
Same hardware
Same payment test service
Identical thresholds

Post-audit results: synchronous implementation, Approach 1, compared with the queue-based implementation, Approach 2.
Metric	SyncA1	QueuedA2
HTTP response latency
Median (ms)	1,090	6
p95 (ms)	5,135	11
p99 (ms)	11,342	20
Worst case (ms)	24,513	66
Errors and inventory
HTTP error rate	4.65%	0.00%
5xx responses	62	0
Stranded units	3	0
Completed orders
Order success rate	91.08%	91.00%
Fulfillment p95 (ms)	4,998	8,052
Throughput (orders/s)	72.9	32.5

Different response milestones. The synchronous response arrives after purchase processing. The queue returns 202 Accepted after reservation and enqueueing. The 467× ratio describes that HTTP-response difference.

The queued run recorded 108,844 HTTP requests with a 0.00% HTTP error rate. Order success remained around 91% in both runs. These results describe the reported 300-VU test; the fulfillment and throughput costs remain part of the comparison.

Architecture / request lifecycle

03 / Architecture

Two execution paths.

01 / Synchronous02 / Queue-based

Synchronous project topology. A client connects to one API node on port 3000. The API connects to Redis for cache and stock counting and to Redis Pub/Sub for stock updates. The Redis components exchange updates; PostgreSQL stores products, orders and users. — Approach 1 / Synchronous request path

Reply after payment.

A request performs validation, stock decrement, the payment call, and database writes before answering. The response includes the time spent completing that work.

Validate the purchase and check stock.
Apply the stock update and call the payment service.
Finish database writes, then return the result.

The important correction

The audit traced the original failure to a row lock held across a read, compute, write, and commit cycle. An atomic conditional update shortened that critical section while keeping the synchronous architecture.

Queue-based project topology. NGINX routes the client to two API nodes and a separate SSE endpoint. API nodes pass through Redis queue-limit middleware to Redis for cache, queueing and stock reservations. Order and payment workers process the work and write to PostgreSQL. Redis Pub/Sub sends stock updates through SSE. — Approach 2 / Queued execution, separate workers, and SSE updates

Reply after reservation.

The API validates the request, reserves stock, and hands the order to a queue. Workers perform the slower processing while the client receives status updates through SSE.

Route through NGINX and the API's queue-limit middleware.
Claim the client-supplied Idempotency-Key with atomic SET NX; reserve stock with DECR.
Enqueue the order and return 202 Accepted.
Process the order with a conditional Postgres update, then call the payment service.
Publish stock and status updates through Redis Pub/Sub and SSE.

What acceptance means

The order has been admitted to the processing path. A successful payment, a decline, and an unknown payment outcome still require different state transitions.

04 / Stock ownership

Separate reservation from confirmation.

The admission counter and the confirmed-order record answer different questions. Keeping their responsibilities separate is what protects pending reservations.

Redis / Can another order enter?: The reservation counter uses atomic DECR and INCR. A reserved unit stays accounted for while its order is pending.
Postgres / What actually sold?: Order records and conditional stock updates track the database side of the purchase. Confirming payment must leave the Redis reservation counter intact.

Payment succeeds
Confirm the order. Leave the reservation counter unchanged.
Card is declined
Release the reservation with INCR.
Payment times out or remains unknown
Hold the reservation and mark the order needs_reconciliation for human review.

Stock reconciliation / checked after each run

redis_stock+ pending_orders+ confirmed_orders= initial_stock

05 / Code audit

21 defects. Six findings worth examining.

An earlier report claimed a 42× latency gain under “100,000 concurrent users.” Auditing it against the implementation uncovered 21 defects. This page uses the post-audit measurements above; the earlier claim is not its benchmark.

Idempotency cached the wrong outcome.
Critical

The middleware deleted the cached response on success and cached it on failure. A retry after success consumed another unit of stock; a transient failure could block retries for the rest of the five-minute sale.
Confirmation erased pending reservations.
Critical

The payment worker overwrote Redis's reservation counter with a value from Postgres. Pending reservations disappeared from the admission calculation, allowing approximately 146 excess orders against unavailable stock during testing.
The worker's stock check had only moved.
High

The report said the stock re-check had been removed. The code had moved it from Redis to a Postgres SELECT … FOR UPDATE. Payment workers still serialized behind one row lock, which was also preventing actual overselling.
The lock window caused the collapse.
High

The original synchronous path held a row lock across multiple round-trips. Replacing the read, compute, and write sequence with UPDATE … WHERE stock > 0 RETURNING shortened the critical section without changing the architecture.
Earlier stock-correctness retest: the faulty run sold 184 of 1,000 units; the corrected path sold 1,000 of 1,000. These counts are separate from the controlled A/B results above.
Thrown errors leaked reservations.
Medium

Two operations could throw between reserving stock and handing the order to the queue. Either failure could strand the reservation. An explicit ownership flag made the release path depend on whether the request still owned that reservation.
Schema changes raced at startup.
Medium

sequelize.sync({ alter: true }) ran in four clustered API workers at once. Concurrent schema changes dropped constraints another worker expected, triggered crashes, and invalidated earlier benchmark runs.

Keeping stock correct under load.

HTTP error rate under peak load

Acknowledgement and completion.

Two execution paths.

Reply after payment.

Reply after reservation.

Separate reservation from confirmation.

Payment succeeds

Card is declined

Payment times out or remains unknown

21 defects. Six findings worth examining.

Idempotency cached the wrong outcome.

Confirmation erased pending reservations.

The worker's stock check had only moved.

The lock window caused the collapse.

Thrown errors leaked reservations.

Schema changes raced at startup.