Approaching Architecture
#systemsWhat is architecture?
Architecture is the stuff that’s expensive to undo, and that every later decision has to live with: datastore, sync vs async, how you split processes, the wire protocol. Everything else is design, and you can fix it next week.
There’s no hard line. The more code a decision touches and the harder it is to change in small steps, the more architectural it is.
Every system has one. If nobody picked it, you got the accidental one: whatever got imported decided your dependency graph.
Quick smell test for a bad one:
- How many files do you open to trace one request?
- Can you test a part without the database running?
- Does a library upgrade force changes in your own types?
Bad example: the standard library gives you a stream you can read in chunks. You wrap it in a Document type whose load() returns the whole file. Now every caller holds the full file in memory, nobody can pass a stream, and you have to edit the wrapper each time the library adds something you want. The wrapper made the code worse than calling the library directly.
The usual culprit is too much indirection. Every hop is another place to look when something breaks, so you get more code, harder debugging, behaviour nobody expects, and more attack surface.
What problem are we solving?
Start from the problem, not the tech. “We need a queue” is a solution. “Orders can’t be lost if the process dies halfway” is a problem.
Then find the cause. A slow page is a symptom, an N+1 query is the cause. Throw architecture at a symptom and you usually just move it.
Requirements
A useful requirement is measurable and flips a decision. Say which decision and at what number. “p99 under 200ms” is useless if every option hits it. “Lose a machine, lose no data” kills the single box. If no number would change your choice, you have a wish, not a requirement.
Invariants
Things that must hold in every reachable state: an order is never charged twice, balance never goes negative, one writer owns this record.
An invariant is not a requirement with a higher priority. It’s a different category:
- Miss a requirement and you’re degraded. Break an invariant and you have corrupt data.
- Requirements get load tested. Invariants get enforced by structure: constraints, a single owner, transactions.
- Requirements set the shape of the system. Invariants set the boundaries, because whoever guarantees one needs a single place to do it. “Never charged twice” means one owner of payment state.
Constraints
Team size, hardware, deadline, budget, regulation. You don’t negotiate with these, you design inside them.
Scale
Don’t ask “will it scale?”. Ask what runs out first: CPU, memory, disk, network, database throughput, or people shipping changes.
Measure what one machine does before you decide it’s not enough. Kubernetes on a workload that fits on one box is a scheduler with nothing to schedule.
Example: an internal API doing indexed reads against one Postgres on one box. Load test it first. A single process on modest hardware usually serves thousands of requests per second, and most internal tools never come close. The test takes an hour, the cluster takes weeks. Migrating later has a cost, but it’s paid once and only if you outgrow the box. Going distributed now means paying the tax every day.
Every step up costs:
- One process to many: shared state, locks.
- One machine to many: network failure, partial failure, coordination, consistency, discovery, deploys.
Go up a rung only when a measurement pushes you. Distributed is an order of magnitude harder because a call can fail or hang after the work was already done, and you can’t tell which. The reason to enter should be a number, not a guess about next year.
Reliability
Adding servers doesn’t give you reliability. Two instances raise questions: who takes the request, how do you notice one died, and what if both think they own the same job?
It’s worth paying for when failure costs more than the machinery that prevents it. A blog and an emergency control system can both be HTTP apps. They shouldn’t share an architecture.
Ask which invariants survive a partial failure. Redundancy that breaks “one writer” is worse than none.
Timeouts, circuit breakers and bulkheads are the exception to “wait until you need it”. They’re cheap now, and you only learn one is missing during an outage. Production will hit every failure path eventually (Release It!).
Maintainability
You want to change one thing without touching five others. That’s the requirement. Layers, repositories and dependency injection are just ways to get there, and only worth it where parts really change independently.
What holds up:
- Put a boundary around whatever must hold an invariant atomically, and around things that change at different rates.
- Dependencies point one way. A needs B, B never needs A. Cycles are coupling.
- Deep modules: lots of behaviour behind a small, stable interface (
open,read,write,close). Shallow modules make you learn their internals.
Test a boundary: does it remove coupling, or just add a file to open? If just a file, delete it.
Watch the “in case we change X” abstraction. You pay for it now and probably never use it. A wrapper that only forwards calls to a library you’ll never swap adds nothing the library didn’t have, and it lags behind the library’s API.
The usual excuse is “we might change the database”. Changing it changes the semantics too (transactions, consistency, query shape), and the wrapper hides exactly the part that hurts. A Repository.find(id) over Postgres tells you nothing about what breaks on a store with no joins and no multi-row transactions.
Boundaries that are justified today: the ones around invariants and around things that change at different rates.
Criticality
At some point failure isn’t acceptable. The question stops being “does it usually work?” and becomes “can we show it works under the conditions we care about?”
Then the architecture has to produce evidence: explicit failure modes, isolation, deterministic behaviour, audit trail, controlled rollout, observability, recovery you’ve actually practised.
That often means less code, not more. Critical parts skip the clever abstraction because they have to be easy to understand and verify.
Decide per component. Most of a system is ordinary. A small part holds the invariants that can’t break.
Data
The domain has nouns: Customer, Order, Invoice. The machine runs instructions over data. A model that helps you understand the problem isn’t automatically the right shape for computing it.
Ask:
- What data does this operation need?
- How is it laid out, and how often is it touched?
- Can it run sequentially, and does it fit in memory or cache?
A type for every concept ends up gluing your code to your own wrappers instead of the library or data underneath, and you pay with a conversion at every edge. OOP, DDD, functional, data-oriented: tools, not answers.
Databases are already data-oriented: tables, rows, columns, laid out for scanning. In-memory stores work the same way. The trouble starts when an ORM turns those rows into a graph of Order objects, each pointing at a Customer, and your code loops over the graph. You took a layout built for scanning and threw it away. Keep the domain model for explaining the business, and run the hot loop over the table (or a contiguous array) instead.
Complexity as a cost
Some complexity is the problem itself: the business, the hardware, the load, the rate of change, the cost of failure. No language removes it.
The rest is yours: what your solution adds. Architecture is mostly stopping that part from outgrowing the first.
Every abstraction costs concepts, indirection, code, failure modes and things to learn. It has to buy something: isolation, replaceability, scale, reliability, local reasoning. Ask:
- What complexity does it remove?
- Is that cheaper than the abstraction?
- When does the payoff arrive, and are we paying before then?
Solving a problem you have: justified. Solving one you might have: wait. If it’s cheap to add later (splitting a module), wait. If it’s expensive later (data model, wire protocol), think early.
“It depends” is a fine answer if the next question is “on what?”
A method
- What is the problem, and what causes it?
- What does solving it require? Measurable requirements that flip a decision, plus the invariants.
- What are the constraints?
- What fails first, and what happens when it does?
- What’s likely to change?
- What complexity does each option add?
- Is it justified by a problem you have now?
Then:
- Design enough to cover the main risks, then build. No design is debt, all design up front is waste.
- Throwaway proofs of concept for specific questions: is it feasible, does the library do what it claims, is the performance reachable.
- Write the options down against what matters: complexity, performance, ops cost, team skill, time. Nothing is free, you’re picking which costs to pay.
- Diagrams are for thinking. If the diagram is a mess, so is the design.
- Stuck? Walk away. A tired brain can’t tell good from bad.
Books worth reading:
- Design It! (Keeling)
- Release It! (Nygard)
- A Philosophy of Software Design (Ousterhout)
- Designing Data-Intensive Applications (Kleppmann)
- The Art of UNIX Programming (Raymond)
Videos: