Everything I Know About Distributed Systems I Learned at Costco

I came for paper towels. I left with a kayak and a working mental model of egress validation.

I am going to be vulnerable for a second. I have a PhD. I have taught distributed systems. I have drawn the load-balancer-on-the-left, services-in-the-middle, database-on-the-right diagram so many times I could do it in my sleep. I have sat through enough architecture reviews to know that a lot of people in that room are nodding at diagrams they only partially understand.

And I was sometimes one of those people.

The boxes and arrows are technically correct. They are also just... boxes. Nobody looks at a Mermaid diagram of a rate limiter and feels what it means to be the one getting throttled.

But when I was standing in a Costco checkout line in a suburb of Orlando, with my totally needed items (48-pack of paper towels, oversized rotisserie chicken, a "normal" amount of energy drinks, and a kayak paddle). Something clicked.

I had been walking through a distributed systems tutorial every time I went shopping.

Authentication, ingress, rate limiting, observability, egress validation, vendor abstraction, SLA commitments, dead letter queues, DNS, capacity planning, disaster recovery, breaking API changes, microservice sprawl. All of it. Right here. Next to the free samples.

And now, let me walk you through it.

Authentication vs. Authorization

You cannot get into Costco without a membership card. It does not matter if you have been coming for 15 years, if you just need one thing, or if your friend is somewhere in the store. No card, no entry.

That is authentication and that is like your API key.

Here is the part that matters: the card does not mean Costco trusts you. They are going to watch you the entire time you are in the building and check your receipt on the way out. The membership card is not a character reference. It is proof that you are in the system, that someone can bill you, and that you agreed to the terms.

This is where a lot of teams get confused in practice. Authentication and authorization are two separate concepts that get collapsed into one in a lot of implementations. Usually because it feels faster to just check the API key and move on.

• Authentication: we know who you are. You have been issued credentials. There is a record.
• Authorization: here is what you are allowed to do. This is a policy decision, evaluated separately, often at a different layer entirely.

The membership card is authentication. What you are allowed to put in your cart, whether you can access the executive member early hours, whether your account is in good standing: all of that is authorization. It happens further in, not at the door.

In system terms: the API gateway does the authentication check (is this key valid?), and the downstream service handles authorization (is this key allowed to call this endpoint with these parameters?). Collapsing those two into a single "check the key, allow the request" pattern is exactly what causes privilege escalation bugs. The door greeter is not doing a full background check (that is not their job.)

And notice how the entrance itself enforces ingress control: one-way flow, defined entry path, no cutting through the loading dock. The client does not get to call the database directly. There is a door. You use the door.

Rate Limiting (or: the Sample Person Is Not Being Rude)

Saturday at Costco, you walk past the free sample stations. One sample per customer. Maybe two if you look enthusiastic and the person is feeling generous. Come back in 20 minutes and there might be a fresh batch and they might not recognize you. That is your retry window.

The sample person is not making a moral judgment about my ability to handle a full portion of the mini quiche (because I know I can handle it). They are a rate limiter. They are protecting finite supply for the entire population of people who will want this thing throughout the day. The whole point is: a service that says yes to every request at any rate is a service that eventually says yes to nothing.

Rate limiting tends to get implemented reactively. Something falls over under load, so then someone adds a limit. But thinking about it like the sample station helps reframe the purpose. The limit is not punitive, it is load shedding. It is how you keep the system available for everyone.

There is a nuance here worth naming. The sample person has no persistent tracking. They do not have system (other than their memory). If you approach from a different angle in 15 minutes, you might get another sample. Some rate limiting works exactly like this: session-based, not identity-based. A sliding window tied to an IP or session token rather than a user identity. It is good enough to prevent thundering herd behavior without requiring the overhead of cross-service identity resolution. It is not supposed to be perfect. The throttle accomplishes the goal without requiring surveillance.

The implication for implementation: know which type of rate limiting you actually need. Token bucket vs. fixed window vs. sliding window are not interchangeable. They have different behaviors under burst traffic, and your retry strategy on the client side needs to account for whichever model is running on the server side. Exponential backoff with jitter exists because the naive "wait and retry" strategy makes thundering herds worse, not better.

Observability (The Building Has No Windows)

Costco has no windows, no clocks on the walls, no natural light. You walk in at noon and walk out in the dark having been in there for hours without noticing. There is no external reference point. The system has given you zero information about time, context, or what is happening outside.

That is what an unobservable system feels like from the inside.

Observability is not the same as monitoring. Monitoring is: I defined a set of things I expected to go wrong and I am alerting on those things. Observability is: I can ask arbitrary questions about what my system is doing from the outside, using the outputs it emits, without having to go in and look. Logs, metrics, traces: the three pillars. They exist so you can understand system state without being inside it.

The unobservable system is the one where you find out something is wrong because a user called. Not because an alert fired or a dashboard went red. A user had to call. You were in a building with no clock and you walked out surprised it was dark.

If you have shipped something where the first indication of a production issue was a customer ticket, you know exactly what this feels like.

The practical failure mode is instrumentation that was added reactively rather than designed in. You get alerts on the things you thought to measure and fly blind on everything else. Good observability design means instrumenting the state transitions and failure boundaries of your system, not just the happy path latencies, so you can reconstruct what happened after the fact. The parking lot camera is not just for catching theft. It is so someone can go back and figure out what happened when the cart return got blocked at 2 p.m.

The Bottleneck Service

Every Costco has a gas station. The line is always about 15 cars long.

The gas is cheaper than anywhere nearby, and everyone and their mother knows this. The math always works in the line's favor and therefore the line never goes away.

This is your perpetually saturated service: the one that is never broken, always at capacity, and absolutely load-bearing for everything downstream. Every distributed system has one. The database that is always a little slow but is too central to refactor. The third-party API that six different teams route through and nobody owns. The internal shared service that everyone depends on and for which nobody has gotten budget to scale.

The line is not a bug. It is the predictable result of a pricing and capacity relationship that has never been resolved. Understanding this matters because teams keep building things that depend on the gas station. You need to know before you build, not after you are already in the queue, which service in your ecosystem is structurally saturated.

From an architecture standpoint, the response options are: async decoupling (stop waiting in line, get a notification when it is your turn), caching (buy your gas in bulk, hold it somewhere closer), circuit breaking (stop queueing when the line exceeds a threshold, fail fast), or actually scaling the capacity. Most teams do none of these until something breaks.

Egress Validation (The Receipt Checker)

You have been in the store for potentially hours. Someone scanned every item. You paid. You have a receipt. And before you walk out, someone with a yellow marker (or a pen, never know whats going on there) checks your receipt against your cart.

They are not verifying payment. They watched you pay. They are checking that what is leaving the building matches what the system registered as leaving the building.

That is egress validation, and it is the check distributed systems teams most commonly skip.

The reasoning usually goes: the data came in validated, it went through the process, it must be fine on the output side. The input was checked, the logic ran, the tests passed. What could have changed?

A lot, actually. The scanner misses things. Items end up in carts they should not be in. A transformation step silently drops a field. A serialization bug introduces nulls. The response schema drifts from what the contract specifies. A downstream join returns more rows than expected and nobody notices because the test fixtures were small.

Egress validation at the API layer means: does the response I am about to send actually conform to the contract I committed to? Not just "did the handler return 200" but does the shape, type, and content of the response match what the consumer agreed to receive. Consumer-driven contract tests (Pact is the common implementation) exist specifically for this: the downstream service defines what it expects, and you verify your output against that expectation before it ships.

The receipt checker does not prevent all theft. But it catches the systematic errors: the things that fell into the cart by accident, the mismatches between the process and the output. That is the job.

Side Effects After Commit (The Food Court)

The Costco food court is at the exit (by the registers), not the entrance.

You do not walk in past the pizza. The pizza is at the end. You already paid. The primary transaction is complete. And right as you are walking out, the smell finds your family, and now someone is ordering pizza slices that are physically larger than a human head. You did not choose this. You were already done shopping.

The main transaction committed. A downstream process triggered autonomously.

This is post-commit side effects, and it causes more subtle production bugs than almost anything else. The pattern looks like: primary operation completes, all validation passes, commit goes through, and then something else fires: a notification, a billing event, an audit log entry, a cache invalidation, a webhook. It was designed to be logically separate from the primary transaction. It is not actually separate. If that downstream thing fails, you now have a committed transaction with an incomplete side effect and no automatic rollback.

The question you need to answer at design time: what is the consistency requirement between the primary transaction and each side effect? Do they need to be atomic (saga pattern, two-phase commit, outbox pattern)? Or is eventual consistency acceptable, where the side effect will eventually succeed, the system will eventually be consistent, and the window of inconsistency is tolerable? The food court is the side effect. Grandma is already at the counter. The correct answer depends on whether it matters if the pizza order gets lost.

If you have not explicitly thought about this for every side effect your service triggers, you have implicit consistency assumptions baked in. They are going to matter at 2 a.m. when the primary committed and the audit log did not.

Vendor Abstraction (Kirkland)

Most Kirkland products are made by the same manufacturers that make the name brand versions. Same factory, same product, different label. You are not buying Duracell. You are buying the abstraction over Duracell. The Kirkland label is the interface. The implementation is somebody else's factory.

If Costco switches battery suppliers, you do not find out. Same SKU, same price, the implementation changed and the interface did not.

That is vendor abstraction, and it is worth being deliberate about.

The failure mode is coupling your application code directly to a specific vendor's SDK. You write to the AWS S3 SDK throughout your codebase, not to a storage interface that the S3 adapter implements. Then the business decides to move to GCP, or the vendor has an outage and you want to switch to a backup, or pricing changes enough that it is worth evaluating alternatives. The cost of switching is now enormous because you are not switching a label. You are rewriting everything that assumed the factory.

The abstraction layer does not have to be complex. A thin interface that your application code talks to, with concrete adapters behind it for each vendor implementation, is often enough. The interface is the Kirkland label, the adapters are the factories. And your application does not need to know which factory is running.

Abstraction is especially worth thinking about for: storage, queuing, email delivery, payment processing, and feature flagging. These are the places where vendor lock-in accumulates quietly and expensively, so it's better to do the work now versus later.

SLA Commitments (The $1.50 Hot Dog)

For those who know, the hot dog is not a menu item anymore. It stopped being a purely business decision sometime in the 1990s. It is now part of Costco's brand identity, the membership value proposition, and the customer relationship. The cost of changing it exceeds any revenue the change would generate. The hot dog became infrastructure.

This is what happens to public API contracts, guaranteed price tiers, and integration points that customers build on. The longer they exist, the more expensive they become to change. Not because the technical change is hard, but because the cost is distributed across every downstream system that built assumptions on top of your commitment.

Semantic versioning is the mitigation. The version number is a signal about the cost of upgrading. A major version bump says: assumptions have changed, budget for migration. Deprecation periods exist because you do not move the paper towels with no forwarding address (more on that in a moment). You give people a migration path. The longer the integration point has been stable, the longer the deprecation window needs to be.

The thing to internalize before you publish: understand what it will cost to stay committed to this interface indefinitely. And understand that cost goes up over time, not down. Every new consumer that integrates with your v1 API makes it harder to change. The hot dog was a business decision once. Now it is an infrastructure decision. Know which one you are making.

Dead Letter Queues (The Returns Counter)

Ok, I will admit it, this is not a perfect analogy (I'm stretch a bit, but bare with me). Costco has one of the most permissive return policies of any major retailer. They will take back food, a mattress you slept on for four years, a television after the Super Bowl, a Christmas tree in January. The policy is real, it is honored, and the cost is baked into their margins.

The returns counter is your dead letter queue plus retry logic.

A DLQ holds messages that could not be processed: events that failed, arrived out of order, or arrived in a format the consumer did not expect. Rather than being discarded, they are held somewhere while the system decides what to do with them. Some get reprocessed once the issue is resolved. Some get routed to a different handler. Some get manually inspected and written off. The important property is the message is not lost.

The reason the analogy is slightly imperfect is because in a real DLQ, messages are routed there automatically when processing fails. Nobody carries them back. The returns counter requires a human to initiate it. The customer has to come back. But once the item is in the queue, the behavior maps correctly. The system holds the thing, a human or automated process makes a routing decision, and the outcome is one of: back on the shelf (reprocessed), returned to vendor (escalated to another system), or written off (acknowledged failure, record kept).

The operational failure mode with DLQs is treating them as a set-it-and-forget-it feature. You set up the DLQ, messages start landing in it, and nobody looks at it for three months. By then you have a backlog of failed events, no clear understanding of why they failed, and a reprocessing problem that is now genuinely hard. The DLQ is not a solution. It is a holding area that buys you time to have a solution. The returns counter requires staff, policy, and a decision process. Same thing.

DNS (The Parking Lot)

The Costco parking lot has lanes, arrows, and rules painted clearly on the ground. Someone designed this with a coherent theory of how traffic would flow.

On Saturday morning, none of it applies. Someone is going the wrong way. Someone abandoned a cart in the middle of the flow path. Someone drove diagonally across four lanes to take a spot on the other side of the lot. The theory is clean. The practice is complete chaos.

That is DNS.

The domain name system has a hierarchy and rules. You ask for a name, a resolver checks its cache, makes a recursive query up the authority chain if needed, gets an IP address, and routes you there. In theory it is orderly. In practice: stale cache entries with TTLs that were set too high in 2018, records pointing at infrastructure that was decommissioned but not cleaned up, split-horizon configurations that behave differently inside and outside the VPN, secondary nameservers that are technically still authoritative but have not been touched since a contractor left, and propagation delays that mean different users are getting different answers depending on which resolver they hit.

The parking lot was designed correctly. The theory is sound. The Saturday experience is a different thing entirely.

DNS issues are particularly painful because they often manifest as intermittent failures that are hard to reproduce, they sit below the layer that most application observability covers, and the fix (flush a cache, update a record, wait for propagation) is often simple but the diagnosis takes hours. The lesson is less "here is how to fix DNS" and more: when something is behaving inconsistently across environments or users, check DNS before you spend two hours reading application logs.

Disaster Recovery (The Tire Center)

Costco has a tire center. It has staff, equipment, and a fully operational automotive service operation. I have been a member for years. Well, my mother is a member and I piggyback. I have never used the tire center. I assume it works. I have no recent evidence for this assumption. I have never seen it tested.

That is your disaster recovery runbook.

Your DR runbook exists. Someone wrote it. It is in a Confluence page that was shared in a Slack channel roughly two years ago and has not been updated since. It has steps. Those steps describe how to restore a service in the event of a major incident. They were accurate at the moment they were written by someone who understood the system as it existed then.

The system has changed since then. It always has.

The question is not "does the runbook exist." It is "when did someone last run it?" Not review it. Not update it in theory. Actually execute it from step one to the end, in a simulated or real scenario, and verify that the outcome matched the expected outcome.

Game days exist for this reason. Chaos engineering exists for this reason. The tire center works, probably. The DR runbook works, maybe. The operational difference between those two answers is whether you find out you were wrong during a scheduled drill or at 2 a.m. on a holiday weekend with real customer impact.

The tire center is not a liability. Having automotive service capacity is genuinely valuable on the specific day you need it. Untested disaster recovery is not a safety net. It is a confidence problem dressed up as documentation.

Breaking API Changes (Something Moved)

You know how you just "know" the layout of a store? Well I know the layout of Costco. Correction I knew. But then they move stuff. For example, I knew where the paper towels were at my Costco, but one day they were gone. No announcement. No sign saying "paper towels have moved." No forwarding address. I wandered around for 15 minutes being furious in a way that I fully recognized was disproportionate, but I think you understand.

That is a breaking change with no migration path.

My downstream process, finding paper towels, had been built on stable assumptions about where the resource lived. Those assumptions had not changed in years. The endpoint moved with no deprecation notice, no version bump, no grace period.

This is the failure mode that semantic versioning is designed to prevent. Major version increments exist to communicate to consumers: the contract has changed in a way that may break your integration, plan accordingly. The deprecation period exists to give consumers time to migrate before the old behavior disappears. Consumer-driven contract tests exist to catch cases where the producer changes something that breaks a consumer before it ships to production.

A sign in aisle 4 saying "paper towels are now in aisle 12" is a migration path. It is not complicated. It costs almost nothing. The absence of it costs your consumers time and trust, and it costs you the support tickets and the angry 15 minutes in the aisle.

Microservice Sprawl (Everything Else Costco Sells)

Costco started as a warehouse store. Bulk goods, low prices, that is the model. Costco now has a pharmacy, an optometrist, a travel agency, insurance products, a gas station, a food court, a tire center, a hearing aid center (my grandma goes there), and (I want to be clear that I looked this up) a casket. An actual casket. Delivered to your door. From Costco.

That is microservice sprawl.

The pattern is always the same. You start with a service that does one thing well. A business case emerges for adjacent functionality. The infra team says it would only take a sprint. Someone points out that you already have the data to support a third thing. Six sprints later, you have a service with eleven responsibilities, ownership that is unclear, and a deployment that is genuinely terrifying because nobody is sure what else might be affected.

Each individual decision made sense at the time. That is the defining feature of sprawl. It is not the result of bad decisions; it is the accumulation of locally reasonable decisions that were never evaluated globally.

The corrective is not always "tear it down and rebuild it clean." Costco is extremely successful. The sprawl works, more or less. But at some point you need a service map. You need to know what you actually have, who owns it, what depends on it, and what the failure domain looks like if any given piece of it goes down. A history of reasonable decisions is not the same thing as an architecture. An architecture is intentional. A history just is...

What This Is Actually About

Distributed systems concepts are not inventions of the tech industry. They are descriptions of coordination problems that arise any time many agents need to share limited resources reliably. Rate limiting, egress validation, bottleneck management, failure handling, contract versioning: these problems predate computers by decades. They showed up in warehouses and supermarkets and parking lots before anyone had a name for them.

The textbooks put boxes and arrows around them, and the boxes and arrows are correct. But they are bloodless. They do not carry the weight of standing in a returns line behind someone who has had that mattress since 2019. They do not have the texture of navigating a parking lot where the theory and the practice have completely diverged.

The concepts are not abstract. They are just dressed differently.

Next time you are sitting in a system design review arguing about whether you actually need a dead letter queue, or reviewing an incident where no alert caught the issue, or trying to explain to a product manager why you cannot just "move the endpoint." You have a better mental model now.

The people trying to get a giant rotisserie chicken on a Saturday afternoon understand distributed systems. They just do not know it yet.

This article was inspired by an episode of Chaotic Commits, a podcast by me: engineer, AWS Community Builder, professor, and someone who came home from Costco with a kayak I did not plan to buy. Listen wherever you get your podcasts.