Testing out-of-order webhooks: when completed arrives before pending
A payment provider sends payment.pending, then payment.completed.
Your handler sets the status from each event. In tests the events arrive
in that order, and everything works. In production, one day, completed
arrives first, the late pending follows, and a customer who paid is
back to pending with no access.
Why order breaks
Webhook delivery is a distributed system, and order is the first thing it gives up:
- Retries. If your server answered
503or timed out on the first event, the provider retries it later, often after it has already sent the next one. Backoff delays are per event, so two failed events can come back in either order. - Parallel delivery. Providers send events through many workers. Two events created a second apart can leave on different workers and arrive in either order.
- Your own side. A load balancer in front of several instances, or a queue between the endpoint and the worker, can reorder events again.
Large providers say this plainly: Stripe’s webhook documentation, for example, states that event delivery order is not guaranteed. Design for it rather than hope.
The bug
It usually looks harmless:
switch ev.Type {
case "payment.pending":
a.Status = "pending"
case "payment.completed":
a.Status = "active"
a.Credits += ev.Data.Amount
}
Each branch is correct on its own. Together they assume an order nobody
promised. Run completed, then pending, and the last write wins.
Three ways to fix it
1. Let the status only move forward. Give each status a rank and ignore events that would move it back:
var rank = map[string]int{"none": 0, "pending": 1, "active": 2, "revoked": 3}
func advance(a *Account, to string) {
if rank[to] > rank[a.Status] {
a.Status = to
}
}
This is the simplest fix and covers most payment flows. It needs care where a flow legitimately goes back, for example a subscription that is reactivated after a chargeback: then the event for the new payment, not the old one, must drive the change.
2. Ignore events that are older than what you already applied. Keep the creation time, or a version number if the provider sends one, of the last event applied to the object, and skip anything older. Watch out for two events created in the same second; a version or sequence number is safer than a timestamp.
3. Treat the event as a hint and fetch the current state. On any event, call the provider’s API for the object and store what it says now. The order of webhooks stops mattering, at the cost of an API call per event and a dependency on the provider being reachable.
Whichever you choose, also match the event to the right object: check that the event’s payment id is the payment you are tracking. Otherwise a late event for an old payment changes the current one, which is a different bug with the same symptom.
Reproducing the order on demand
The fix is only as good as the test, and the test needs the provider to
send completed before pending on purpose. Provider sandboxes rarely
let you choose. TwinStub does: it
plays the provider from a YAML scenario, and both events can be queued
when the payment is created, so you pick the order:
processing:
enter:
- webhook:
target: shop
event: payment.pending
after: 30s
body: '{"type": "payment.pending", "data": {"reference": "{{ .Session.Key }}"}}'
- webhook:
target: shop
event: payment.completed
after: 2m
body: '{"type": "payment.completed", "data": {"reference": "{{ .Session.Key }}"}}'
goto: completed
With the engine running (twinstub serve, admin API on port 9090) and
your app pointed at it, a test starts a session for the customer, lets
the app create the payment, and then delivers the queued events in
reverse:
# start a session for customer u1, then let your app create the payment
SESSION_ID=$(curl -s -X POST localhost:9090/api/v1/sessions \
-d '{"scenario": "acme.payment", "key": "u1"}' | jq -r .session.id)
curl -s -X POST localhost:3000/subscribe -d '{"user": "u1", "plan": "basic"}'
# deliver the queued webhooks now, completed first
curl -s -X POST localhost:9090/api/v1/sessions/$SESSION_ID/deliver \
-d '{"events": ["payment.completed", "payment.pending"]}'
# assert your own state, not the 200s
curl -s localhost:3000/accounts/u1
Both deliveries answer 200. The last line is the test: with the bug,
the account says pending; with the fix, active.
The same works without the shell: a coding agent with the
TwinStub Claude Code plugin
calls twinstub_deliver with the order it wants. In
our test run the agent staged
this case on purpose, and then hit it a second time by accident, when
retries after a 503 came back in the wrong order.
A checklist for your handler
- Deliver
completedbeforependingand check the customer still has access. - Deliver the same event twice and check it settled once.
- Make the handler fail once, let the provider retry, and check the result does not depend on which retry came first.
- Deliver an event for an old payment and check the current one is untouched.
Each of these is a row in a table of “events in this order, expected
state”, written before you run anything. The 200s will all look the
same. The state will not.