<CP/>
← Writing
Engineering

Idempotency: Why Retrying a Request Shouldn't Duplicate It

Idempotency: Why Retrying a Request Shouldn't Duplicate It

I asked an AI coding agent for an endpoint that records a payment when a webhook arrives from a payment gateway. Receive the event, verify the signature, insert the order as paid. It worked on the first test. It worked on the second, third, and fourth test too.

It broke on the fifth, because I finally asked the question nobody asks by default: what happens if that same webhook arrives twice?

This is post #06 in an ongoing series I've been writing for developers who lean heavily on AI to write code — "vibecoders," as the term goes — and who sometimes skip the fundamentals an AI won't volunteer on its own. Today's fundamental is idempotency, and it's one of the ones that doesn't show up until production, usually as a support ticket.

The setup: a webhook that trusts its own delivery

Here's roughly what the AI-generated handler looked like the first time around:

app.post("/webhooks/payment", async (req, res) => {
  const event = verifySignature(req.body, req.headers["x-signature"]);

  await db.orders.insert({
    externalId: event.paymentId,
    status: "paid",
    amount: event.amount,
  });

  await sendConfirmationEmail(event.customerEmail);
  await decrementInventory(event.items);

  res.status(200).send("ok");
});

Clean. Readable. It does exactly what I asked for: on webhook, insert the paid order, send the email, adjust stock. I ran it against the gateway's test suite and every event created exactly one order. I moved on.

What I didn't test, because it never occurred to me to test it, is what happens when the same event shows up more than once. And in the world of payment webhooks, that's not an edge case. It's the default behavior of every gateway I've worked with.

Why the same event arrives twice (or three times)

Webhook delivery is a "best effort, with retries" contract, not a "exactly once" guarantee. A gateway will resend the same event if:

  • Your server takes too long to respond and the gateway's timeout fires before your 200 OK gets back.
  • There's a transient network blip between the gateway and your infrastructure.
  • The gateway proactively resends during a traffic spike, on the assumption that a dropped delivery is more likely than a duplicate one.

From the gateway's point of view, this is the safe default: better to deliver twice than to silently lose an event. From your endpoint's point of view, if it wasn't built with that in mind, it's the opposite of safe.

My naive handler had no concept of "I've already seen this." Every incoming request was, as far as the code was concerned, the first request. So a retried webhook created a second row for the same payment, sent a second confirmation email for a purchase the customer made once, and decremented inventory a second time for stock that was never actually sold twice.

None of this shows up as an error. No exception, no failed request, no red line in the logs. Your dashboard just quietly says you sold one more unit than you did, and a customer gets two "your order is confirmed" emails and wonders if they got charged twice.

What I tried first, and why it wasn't enough

The obvious fix: check if you've already processed this event before you process it again.

app.post("/webhooks/payment", async (req, res) => {
  const event = verifySignature(req.body, req.headers["x-signature"]);

  const existing = await db.orders.findOne({ externalId: event.paymentId });
  if (existing) {
    return res.status(200).send("already processed");
  }

  await db.orders.insert({
    externalId: event.paymentId,
    status: "paid",
    amount: event.amount,
  });

  await sendConfirmationEmail(event.customerEmail);
  await decrementInventory(event.items);

  res.status(200).send("ok");
});

This felt like the fix. It is, most of the time. It's also where I learned the difference between "usually works" and "actually correct."

Here's the failure mode: a gateway retries a webhook because it didn't get your response fast enough, not because the first request necessarily failed. That means two requests for the same event can arrive close enough together that both hit your server before either one finishes. Request A checks the database, finds nothing, and starts inserting. Before A's insert commits, Request B runs the exact same check, also finds nothing (A hasn't committed yet), and also starts inserting. Both requests believe they're the first to see this event, because at the moment each one checked, that was true.

This has a name: a check-then-act race, sometimes called TOCTOU (time-of-check to time-of-use). The gap between "I looked" and "I acted" is where duplicates sneak back in, and under enough concurrent load, that gap is not a hypothetical. I reproduced it locally by firing the same webhook payload twice with a few milliseconds of delay between them, and got two rows in the database more often than I expected. "Check first, then insert" reads as correct. It just isn't, once you have more than one request able to run that check at the same time.

What actually worked: push the guarantee into the database

The fix isn't a smarter check in application code. It's removing the gap between checking and acting entirely, by making the database enforce uniqueness instead of asking it politely.

First, a unique constraint on the column that identifies the event:

ALTER TABLE orders
ADD CONSTRAINT orders_external_id_unique UNIQUE (external_id);

Then, instead of "check, then insert," the insert itself becomes the check, atomically:

app.post("/webhooks/payment", async (req, res) => {
  const event = verifySignature(req.body, req.headers["x-signature"]);

  const result = await db.orders.upsert(
    { externalId: event.paymentId },
    { externalId: event.paymentId, status: "paid", amount: event.amount },
    { onConflict: "externalId", doNothing: true }
  );

  if (result.wasInserted) {
    await sendConfirmationEmail(event.customerEmail);
    await decrementInventory(event.items);
  }

  res.status(200).send("ok");
});

In raw SQL, that's the difference between:

INSERT INTO orders (external_id, status, amount)
VALUES ($1, 'paid', $2)
ON CONFLICT (external_id) DO NOTHING
RETURNING id;

Two requests can arrive at the exact same millisecond now, and the database itself will only let one of them succeed as an insert. The other gets told "conflict, nothing happened," with no gap in between for a second request to sneak through. The side effects (the email, the inventory decrement) only run when the insert actually happened, which is the signal that this is genuinely the first time we've seen this event, not the second or third delivery of the same one.

That's idempotency, stated plainly: running an operation once or running it twenty times produces the same end state as running it exactly once. Not "similar." Not "close enough." The same.

Generalizing beyond payments: the idempotency key pattern

Payment webhooks are the classic case, but the pattern shows up anywhere a client might retry a write operation without knowing if the first attempt actually succeeded: submitting a form over a flaky connection, an AI agent calling a tool and not getting a clean response so it tries again, a background job that gets rescheduled after a crash mid-execution.

The general version of the fix is an idempotency key: the caller generates a unique identifier for the operation, not for each retry of it, and sends it along every time:

POST /api/orders
Idempotency-Key: 8f14e45f-ceea-467e-b4b6-5c2fdc935d19

The server keeps a table of keys it has already processed, along with the response it gave the first time:

CREATE TABLE idempotency_keys (
  key TEXT PRIMARY KEY,
  response_body JSONB NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now()
);

app.post("/api/orders", async (req, res) => {
  const key = req.headers["idempotency-key"];

  const cached = await db.idempotencyKeys.findOne({ key });
  if (cached) {
    return res.status(200).json(cached.responseBody);
  }

  const order = await createOrder(req.body);

  await db.idempotencyKeys.insertIgnoreConflict({
    key,
    responseBody: order,
  });

  res.status(201).json(order);
});

This still has the same race I described above if you don't back the key column with a unique constraint and lean on that for the atomicity, rather than trusting the findOne check on its own. The lesson from the webhook case applies here identically: the guarantee has to live in a constraint the database enforces, not in application logic that assumes it'll be the only request running at that moment.

The reason this matters more, not less, in an AI-assisted codebase is that agents retry tool calls constantly, and usually without telling you. If a tool call times out or returns something the agent's loop doesn't recognize as success, a well-built agent will just try again. That's good behavior on the agent's side. It's only safe on your side if the endpoint it's calling was built to expect it.

What the AI didn't build, because I didn't ask

None of this was in the code my agent generated for me, and I don't think that's a flaw specific to that one agent. I asked for "an endpoint that records the payment," and it built exactly that: something that records the payment, once, assuming it's asked to do so once. It never modeled the idea that "record the payment" might be requested more than once for the same real-world event, because nothing in my prompt raised that possibility.

That's consistent with what I've found writing the rest of this series. The models are very good at building what's explicitly described. They're not going to volunteer a protection against a failure mode you didn't mention, no matter how predictable that failure mode is to anyone who's run a webhook handler in production before.

So the fix isn't a smarter prompt that magically knows to ask for idempotency. It's knowing to ask the second question yourself: if this gets called twice with the same input, what happens? If the honest answer is "something happens twice that should only happen once," that's the gap. Closing it usually isn't more code. It's usually one unique constraint and one ON CONFLICT clause, in the right place, doing the job a manual check-then-act never quite can.