CS Fundamentals Trainer · Guides
Design a payment system: idempotency, ledgers and reconciliation
A case study in correctness over speed: the payment flow with a payment service provider, idempotency keys, a double-entry ledger, state machines for a payment, handling timeouts and unknown outcomes, webhooks, reconciliation against the provider's reports, and why you never store card numbers yourself.
5 min read
The problem. A shop lets a customer pay by card, and the money must reach the seller exactly once. In most systems an occasional failure is an inconvenience; here a duplicate charge, a lost payment or a wrong balance is a financial and legal problem. This design optimises for correctness, auditability and recoverability before latency.
Requirements and numbers
Pay-in (charge a customer), refunds, payouts to sellers, a history for customers and finance, and fraud checks. Volume: 1 million orders a day is 1,000,000 / 86,400 ≈ 12 payments per second on average, perhaps 100 per second at peak. That is small for a database, which is a hint: the hard part is not throughput, it is exactly-once effects across systems you do not control.
Do not handle card data
Card numbers fall under PCI DSS. The standard design avoids touching them: the browser sends the card details directly to the payment service provider (PSP) (Stripe, Adyen, …) using the provider's hosted fields or SDK, and your server receives a token. You then ask the PSP to charge the token. Your scope shrinks to almost nothing.
The flow
Client --> Order service --> Payment service --(charge token, idempotency key)--> PSP --> card network
| |
payment record (state) webhook: succeeded / failed
| |
Ledger <------------ Payment service <---------------+
- The order service creates a payment record in state
createdwith a unique id (and anIdempotency-Keysent by the client for the checkout). - The payment service calls the PSP with the same idempotency key and moves to
pending. - The PSP answers synchronously or later; the final result also arrives as a webhook. State moves to
succeededorfailed. - On success the ledger is written, and the order is confirmed.
Idempotency
Networks time out and users double-click. If a request is repeated, it must have the effect of one. The client generates an idempotency key per checkout attempt; the server stores (key -> result) with a unique constraint, and a repeated request returns the stored result rather than charging again. Pass the key to the PSP as well, so a retry after a timeout cannot create a second charge there. The same applies to webhooks, which are delivered at least once: process each event id only once, and make the handler safe to repeat.
State machine
A payment moves through explicit states, for example created -> pending -> succeeded | failed, then refunded or partially_refunded. Allowed transitions are enforced in code and in the database (UPDATE … WHERE state = 'pending' so that two racing updates cannot both win). Anything that is not in the table of valid transitions is rejected and alerted on.
The unknown outcome
The nastiest case: you call the PSP, the connection drops, and you do not know whether the charge happened. Do not assume failure and do not blindly retry with a new key. Mark it pending, and resolve it by retrying the same idempotent call or by querying the PSP for the payment by your reference, and by the webhook. Design every step so that "unknown" is a first-class state with a resolver, not an error path.
The ledger
Balances must be derived from an append-only double-entry ledger: every movement is two entries that sum to zero (debit the customer's receivable, credit the merchant's payable), and entries are never updated or deleted. A mistake is corrected by a new, reversing entry. This gives an audit trail, makes totals checkable (all entries sum to zero) and lets you rebuild any balance. Money is stored as integers in minor units (cents) with a currency code, never as floating point.
entry(id, transaction_id, account, amount_minor, currency, created_at)
-- a 25.00 sale: (tx1, customer_receivable, +2500, EUR), (tx1, merchant_payable, -2500, EUR)
Write the payment state change and the ledger entries in one database transaction; publish events to other systems through a transactional outbox so that "committed" and "announced" cannot diverge.
Reconciliation
Even with all this, things drift: a webhook is lost, the PSP and you disagree. A nightly reconciliation job compares your records with the PSP's settlement reports, line by line, and with the bank deposits. Differences become tickets, and the job is the safety net behind everything else. Many incidents are found here and not in monitoring.
Reliability and security
- A relational database with strong consistency for payments and ledger; replicate and back up, and rehearse restores.
- Retries with backoff for calls to the PSP; timeouts on every call; a circuit breaker; an alert on payments stuck in
pending. - Fraud checks (velocity rules, risk scores, 3-D Secure) before and after authorisation.
- Least privilege and audit logs; encrypt sensitive data; keep personal data out of logs.
- Reduce risk with authorise now, capture later for goods that ship after a delay.
Trade-offs
Strong consistency and a single writer per account limit horizontal scale, which is fine at this volume; if you must scale writes, shard by merchant or account. Synchronous confirmation is a better user experience; asynchronous confirmation via webhook is more robust, so most systems show "processing" and update on the event.
Sources
- Alex Xu, System Design Interview Vol. 2, chapters "Payment System" and "Digital Wallet".
- Stripe: Designing robust and predictable APIs with idempotency.
- PCI Security Standards Council: PCI DSS.
- Pat Helland, Immutability Changes Everything — why append-only records suit financial data.
Practise this topic: System Design System Design › Case studies