What Actually Gets Hard When You Maintain Many Backend Services

Maintaining one backend service and maintaining many backend services are very different jobs.

When the number of services grows, the main difficulty stops being framework knowledge.

I still write endpoints, queries, validators, background jobs, and tests. But those are rarely the parts that consume the most reasoning time.

The harder questions are things like:

Which service owns this rule?
Where did this user context disappear?
Can I deploy this change independently?
Which service still depends on the old response shape?
Why did the notification fail when the source action succeeded?
Which database migration has to happen first?

That is where multi-service maintenance becomes real engineering work.

The first problem is ownership

A system with several backend services usually divides responsibilities by domain.

That sounds clean on a diagram.

In real work, requirements do not always respect those boundaries.

Suppose a document is approved and that should:

• update document state
• change workflow state
• create an audit record
• notify a user
• affect another downstream object

Which service owns the orchestration?

If every service starts calling every other service, the architecture slowly turns into a distributed monolith.

If everything is pushed into one "orchestrator," that component can become another monolith in the middle.

I learned to ask two separate questions:

Who owns the state?

and

Who owns the process?

Those are not always the same service.

Cross-service debugging is slower than local debugging

A normal local bug may have a path like this:

request
    ↓
handler
    ↓
query
    ↓
response

A multi-service issue can look like this:

frontend request
    ↓
service A
    ↓
internal call to service B
    ↓
background task
    ↓
service C
    ↓
notification service
    ↓
websocket
    ↓
frontend state update

Now imagine the user reports only:

I didn't get the notification.

The source action may have succeeded perfectly.

The failure could be:

• no event emitted
• wrong payload
• missing tenant context
• task not executed
• downstream API failure
• websocket disconnect
• frontend ignored the message

The problem is not that any one layer is unusually difficult.

The problem is locating the broken boundary.

Correlation matters more as the system grows

Once requests cross service boundaries, isolated logs become much less useful.

A line like this:

POST /notifications -> 500

is not enough when you need to answer:

Which user action caused this?

The more useful model is to carry a correlation or request identifier through the chain.

Conceptually:

request_id=abc123
 
Service A -> abc123
Service B -> abc123
Worker    -> abc123
Notify    -> abc123

Then logs become a trace of the operation rather than unrelated messages from different applications.

This is one of those things that feels optional when you have two services and essential when you have ten.

Authentication becomes infrastructure

In a single service, auth can feel like one dependency at the edge:

token -> validate -> user

Across many services, identity has to move safely between boundaries.

Now there are more questions:

Is this a user call or a service call?
Does the downstream service need the original user identity?
How is tenant context propagated?
Which claims are trusted?
Does every service verify the same issuer/audience rules?

If every team or service solves that differently, subtle security and integration bugs appear.

I started seeing authentication less as a feature of each API and more as shared platform behavior that every service must understand consistently.

Tenant context has the same problem

Multi-tenancy adds another piece of context that has to survive boundaries.

It is easy to correctly filter the first database query and still lose the tenant on an internal API call or background task.

The full path matters:

incoming token
    ↓
tenant resolution
    ↓
service call
    ↓
worker payload
    ↓
downstream query

The safest architecture is one where tenant context is difficult to forget accidentally.

For example, I prefer database/service helpers that require tenant scope instead of leaving every developer to remember to add it manually to every query.

Deployment order becomes part of the design

With one service, deployment is usually:

merge
build
deploy

With many services, a change may require sequencing.

For example:

1. database accepts new field
2. producer starts returning it
3. consumers start using it
4. old field is removed later

If you reverse the order, the code may be correct in isolation and still fail during rollout.

This is why backward compatibility matters so much in service architectures.

A good change is not only correct in the final state.

It should also be safe while the system is temporarily running mixed versions.

Database migrations become operational work

Schema changes are another area where scale changes the problem.

Adding a column is easy.

Adding it safely when several services or workers may read the same data path is different.

I try to separate migration steps like this:

expand
    ↓
deploy compatible code
    ↓
backfill
    ↓
move reads/writes
    ↓
contract

For example, instead of immediately changing a nullable field to required:

1. add it as nullable
2. deploy writers that populate it
3. backfill existing rows
4. verify
5. enforce NOT NULL

That pattern reduces deployment coupling considerably.

Shared libraries solve some problems and create others

When several services repeat auth, logging, schemas, or client code, the natural response is to create shared packages.

That can help.

But shared libraries can also synchronize services that were supposed to be independently deployable.

If changing one helper means every repository must immediately upgrade, the shared package has become another dependency surface.

I now distinguish between:

stable platform primitives

and

domain behavior

Stable primitives can often be shared.

Domain behavior usually belongs with the service that owns it.

Local development gets surprisingly difficult

Another problem nobody shows on the architecture diagram is simply running the system.

If feature A needs:

user service
entity service
document service
notification service
PostgreSQL
Redis
worker

then "run it locally" is no longer a trivial instruction.

This is where Docker Compose, sensible defaults, seed data, mocked dependencies, and good README files start paying for themselves.

The best local environment is not necessarily a perfect copy of production.

It is the smallest environment that lets a developer reproduce the important behavior reliably.

More services means more failure modes

In a single process, a function call normally either returns or throws.

Across the network, many more states exist:

request never arrived
request arrived but response was lost
timeout after side effect succeeded
partial downstream failure
retry duplicated the operation
consumer processed message twice

This changes how I think about important operations.

For anything that may be retried, I want to know whether it is idempotent.

For example, "create notification" may need a stable event identifier so a retry does not create duplicate notifications.

Network calls force you to think about uncertainty in a way local function calls often do not.

The hardest bugs are often boundary bugs

After maintaining many services, I noticed a pattern.

The most time-consuming issues often happen between components rather than inside them.

Examples include:

• schema mismatch
• auth context mismatch
• tenant propagation failure
• timeout expectation mismatch
• different interpretations of the same enum
• retry behavior
• deployment version mismatch

That changed where I spend review time.

A beautiful internal implementation is useful, but the boundary deserves at least as much attention.

What I optimize for now

When working across many backend services, I care about a few things much more than I did earlier in my career:

• explicit ownership
• stable service contracts
• backward-compatible rollouts
• correlation IDs and useful logs
• consistent auth and tenant propagation
• idempotent operations
• predictable error shapes
• migration sequencing
• realistic local development
• observability at service boundaries

None of these are specific to FastAPI, Spring Boot, Go, or any other framework.

That is the point.

Once the system becomes large enough, framework knowledge stops being the main constraint.

What I learned

Maintaining many services is not mainly about writing more code.

It is about preserving a shared understanding of the system while many independently deployable parts keep changing.

The question I ask more often now is not:

How should I implement this endpoint?

It is:

What assumptions will this change create for every component around it?

That question catches a surprising number of problems before they turn into cross-service debugging sessions.