Testing out-of-order webhooks: when completed arrives before pending

November 16, 2026 · 4 min read

A payment provider sends payment.pending, then payment.completed. Your handler sets the status from each event. In tests the events arrive in that order, and everything works. In production, one day, completed arrives first, the late pending follows, and a customer who paid is back to pending with no access.

Why order breaks

Webhook delivery is a distributed system, and order is the first thing it gives up:

  • Retries. If your server answered 503 or timed out on the first event, the provider retries it later, often after it has already sent the next one. Backoff delays are per event, so two failed events can come back in either order.
  • Parallel delivery. Providers send events through many workers. Two events created a second apart can leave on different workers and arrive in either order.
  • Your own side. A load balancer in front of several instances, or a queue between the endpoint and the worker, can reorder events again.

Large providers say this plainly: Stripe’s webhook documentation, for example, states that event delivery order is not guaranteed. Design for it rather than hope.

The bug

It usually looks harmless:

switch ev.Type {
case "payment.pending":
	a.Status = "pending"
case "payment.completed":
	a.Status = "active"
	a.Credits += ev.Data.Amount
}

Each branch is correct on its own. Together they assume an order nobody promised. Run completed, then pending, and the last write wins.

Three ways to fix it

1. Let the status only move forward. Give each status a rank and ignore events that would move it back:

var rank = map[string]int{"none": 0, "pending": 1, "active": 2, "revoked": 3}

func advance(a *Account, to string) {
	if rank[to] > rank[a.Status] {
		a.Status = to
	}
}

This is the simplest fix and covers most payment flows. It needs care where a flow legitimately goes back, for example a subscription that is reactivated after a chargeback: then the event for the new payment, not the old one, must drive the change.

2. Ignore events that are older than what you already applied. Keep the creation time, or a version number if the provider sends one, of the last event applied to the object, and skip anything older. Watch out for two events created in the same second; a version or sequence number is safer than a timestamp.

3. Treat the event as a hint and fetch the current state. On any event, call the provider’s API for the object and store what it says now. The order of webhooks stops mattering, at the cost of an API call per event and a dependency on the provider being reachable.

Whichever you choose, also match the event to the right object: check that the event’s payment id is the payment you are tracking. Otherwise a late event for an old payment changes the current one, which is a different bug with the same symptom.

Reproducing the order on demand

The fix is only as good as the test, and the test needs the provider to send completed before pending on purpose. Provider sandboxes rarely let you choose. TwinStub does: it plays the provider from a YAML scenario, and both events can be queued when the payment is created, so you pick the order:

processing:
  enter:
    - webhook:
        target: shop
        event: payment.pending
        after: 30s
        body: '{"type": "payment.pending", "data": {"reference": "{{ .Session.Key }}"}}'
    - webhook:
        target: shop
        event: payment.completed
        after: 2m
        body: '{"type": "payment.completed", "data": {"reference": "{{ .Session.Key }}"}}'
      goto: completed

With the engine running (twinstub serve, admin API on port 9090) and your app pointed at it, a test starts a session for the customer, lets the app create the payment, and then delivers the queued events in reverse:

# start a session for customer u1, then let your app create the payment
SESSION_ID=$(curl -s -X POST localhost:9090/api/v1/sessions \
  -d '{"scenario": "acme.payment", "key": "u1"}' | jq -r .session.id)
curl -s -X POST localhost:3000/subscribe -d '{"user": "u1", "plan": "basic"}'

# deliver the queued webhooks now, completed first
curl -s -X POST localhost:9090/api/v1/sessions/$SESSION_ID/deliver \
  -d '{"events": ["payment.completed", "payment.pending"]}'

# assert your own state, not the 200s
curl -s localhost:3000/accounts/u1

Both deliveries answer 200. The last line is the test: with the bug, the account says pending; with the fix, active.

The same works without the shell: a coding agent with the TwinStub Claude Code plugin calls twinstub_deliver with the order it wants. In our test run the agent staged this case on purpose, and then hit it a second time by accident, when retries after a 503 came back in the wrong order.

A checklist for your handler

  • Deliver completed before pending and check the customer still has access.
  • Deliver the same event twice and check it settled once.
  • Make the handler fail once, let the provider retry, and check the result does not depend on which retry came first.
  • Deliver an event for an old payment and check the current one is untouched.

Each of these is a row in a table of “events in this order, expected state”, written before you run anything. The 200s will all look the same. The state will not.