← All posts
#Concurrency#Software Engineering Fundamentals#Production Systems

Why Your Code Works But Your System Doesn't

A race condition caused production to fail even though every test passed. Here's why it happens and how to prevent it.

·8 min read

A developer ships a feature. Every test passes. Three days later, it costs the company real money — and nobody touched the code.

Priya is building a coupon system for an online store checkout. The rule is simple: the code "WELCOME50" gives 50% off, but only once per user.

She writes the function. She writes tests. Every test passes:

  • New user applies the coupon → discount applied ✓
  • Same user tries again → rejected ✓
  • Invalid coupon code → rejected ✓

Code review approves it. It ships on a Friday. Everyone goes home happy.

Monday morning, finance flags something strange. Over the weekend, 340 users managed to use "WELCOME50" twice. Some got it three times. The company just lost real money.

Priya opens the function. She reads it top to bottom. It's exactly what she tested. It's correct. There's no bug in the logic.

So what actually happened?

The bug that isn't in any line of code

Here's Priya's function:

javascript

async function applyCoupon(userId, couponCode) {
// Step 1: check — has this user already used this coupon?
const existingUsage = await db.query(
'SELECT * FROM coupon_usages WHERE user_id = $1 AND coupon_code = $2',
[userId, couponCode]
);

if (existingUsage.rows.length > 0) {
throw new Error('Coupon already used');
}

// Step 2: act — apply the discount and record the usage
const discount = calculateDiscount(couponCode);

await db.query(
'INSERT INTO coupon_usages (user_id, coupon_code, used_at) VALUES ($1, $2, NOW())',
[userId, couponCode]
);

return { discount };
}

Read it slowly. Nothing here is wrong. It checks, then it acts. That's exactly what it's supposed to do.

Here's what happened on Friday night: a flash sale hit the site, and thousands of people opened checkout at nearly the same moment. For some of those users, two requests arrived close enough together that both requests finished the "check" step before either one finished the "act" step. Both checks came back "not used yet" — because at that exact instant, technically, neither one had been used yet. So both got approved.

There was no broken line. The bug lives in the gap between two correct lines — a gap that's invisible when you read the code slowly, and only becomes real when two requests are racing through it at the same time.

This kind of bug has a name: a race condition. But the name matters less than the shape of the idea, because that shape shows up everywhere, not just in coupons.

Why the tests didn't catch it

Here's the test that gave this code a green checkmark:

javascript

test('rejects a coupon that was already used', async () => {
await applyCoupon('user_1', 'WELCOME50'); // first call: succeeds
await expect(applyCoupon('user_1', 'WELCOME50')) // second call: should fail
.rejects.toThrow('Coupon already used');
});

This test is written correctly — and it still can't catch the bug. Why? Because await forces the first call to completely finish before the second one starts. There's no gap for anything to race through. The test accidentally checks a world where things only ever happen one at a time. Priya's real traffic doesn't live in that world.

This is the uncomfortable truth at the center of this whole problem: your tests don't just check your code — they also, silently, decide which situations get tested and which don't. Nobody sits down and thinks "let me test what happens if two requests arrive 4 milliseconds apart." That's not how people naturally think. It's how systems actually behave.

Here's a test that reflects reality instead:

javascript

test('only one request wins when two arrive at the same instant', async () => {
// Fire both requests at once — don't wait for one to finish before starting the other
const results = await Promise.allSettled([
applyCoupon('user_1', 'WELCOME50'),
applyCoupon('user_1', 'WELCOME50'),
]);

const succeeded = results.filter(r => r.status === 'fulfilled');
const failed = results.filter(r => r.status === 'rejected');

expect(succeeded).toHaveLength(1); // exactly one should win
expect(failed).toHaveLength(1); // exactly one should be rejected
});

Run this against the original code, and it fails — both requests succeed. One small change, firing both calls together instead of one after another, is the difference between a test that checks your code and a test that checks your system.

The habit worth building: whenever two operations could plausibly happen in parallel in production, write at least one test where they actually do.

The fix: don't shrink the gap, close it

Priya's first instinct is the same as most people's: "I'll just make the check faster." That doesn't work. It doesn't matter how small the gap is — even a gap of a few microseconds is enough for two requests to land inside it under real traffic. Shrinking the gap doesn't remove it. It just makes the bug rarer, which is arguably worse: a bug that shows up once a month is harder to catch than one that shows up constantly.

The real fix is to make the check and the act happen as one indivisible step, so there's no gap left for anything to slip into.

sql

-- Migration: make the database itself refuse to allow a duplicate
ALTER TABLE coupon_usages
ADD CONSTRAINT unique_user_coupon UNIQUE (user_id, coupon_code);

javascript

async function applyCoupon(userId, couponCode) {
const discount = calculateDiscount(couponCode);

try {
// The check and the act are now the same operation.
// There is no gap left for a second request to slip into.
await db.query(
'INSERT INTO coupon_usages (user_id, coupon_code, used_at) VALUES ($1, $2, NOW())',
[userId, couponCode]
);
} catch (err) {
if (err.code === '23505') { // Postgres: unique_violation
throw new Error('Coupon already used');
}
throw err;
}

return { discount };
}

Nothing here is more "careful" than before. It's not extra error handling bolted on top. The check and the act are simply the same step now, enforced by the database itself — not by Priya's code being fast enough or lucky enough.

Run the concurrency test from before against this version, and it passes. Every time. Not "most of the time." That's the actual point: the bug isn't rare now — it's impossible. Those two things can look identical in a demo, but they are not the same thing at all.

There are other valid ways to close this same gap — locking the row before touching it, or using a database operation that guarantees the check-and-update happens atomically. All of them share the same idea: stop giving the system room to misbehave, instead of trying to outrun the problem.

This isn't a coupon problem

This exact shape — check something, then act on what you found, with a gap in between where the world can change — is not specific to coupons, or even to databases. Once you can see it, you'll start noticing it everywhere:

  • Two people editing the same document, both reading "current version: 3" before either saves — whoever saves second silently overwrites the first person's work.
  • A flight booking system checking "is there a seat left?" and then booking it — two people can both see one open seat and both get it.
  • A background job checking "has this file been processed?" before processing it — and running twice, because two workers checked at nearly the same moment.

Different industries, different code, the same exact gap. Check, then act. Once you know to look for it, you'll find it in code you've already written.

The bigger idea

When you write a function, you're not really making the claim "this code is correct." You're making a narrower claim, whether you realize it or not: "this code is correct in the situations I imagined while writing it."

Production doesn't run your imagined situations. It runs all of them — including the ones that only exist when timing lines up a certain way, when two users click at the same moment, when a network call is slow on exactly the wrong millisecond. Most of those situations never occurred to you, not because you weren't careful, but because they're genuinely hard to picture. That's not a flaw in you. It's a limit that testing, by its nature, doesn't share with production.

Not "be more careful" — that's not a real strategy, and it doesn't scale. The actual habit is this: whenever your code reads something and then, based on what it read, changes something — stop and ask, "what happens if this exact sequence runs twice, at the same time, right now?" Most of the time, nothing bad happens. But when it matters — money, inventory, anything meant to happen only once — that single question is the difference between code that works, and code that works everywhere it's actually going to run.

Carry these forward

  • Correct code and a correct system are two different claims. Code is judged against the situations you imagined. Systems get judged against every situation that can actually occur.
  • A bug doesn't need a wrong line. It can live entirely in the gap between two right lines.
  • "Check, then act" is a pattern, not a one-off mistake. Once you can name it, you'll see it everywhere.
  • The fix is never "go faster." It's making the bad outcome structurally impossible, not just statistically rare.

Priya's code was never broken. It just never had to survive being run twice at once — until the day it did.