Why? Because coordinating writes across multiple nodes can get ridiculously complicated. You have conflicting updates, network latency, consistency guarantees, and a whole lot of edge cases that can turn a seemingly simple architecture into an operational nightmare.

But here's the funny part.

Some of the same ideas that make distributed systems engineers nervous about coordination are helping researchers solve one of the biggest problems in AI training: how do you train massive models across geographically distributed hardware without forcing everything to communicate constantly?

I stumbled upon this connection while reading about database replication. I was deep in the weeds of multi-leader systems, trying to understand why an architecture that looks so intuitive on paper can become so painful in production.

And I kept thinking: surely there's a situation where letting independent nodes make progress and coordinating them less frequently is actually the right move?

That's when I stumbled upon a paper from Google DeepMind called DiLoCo: Distributed Low-Communication Training of Language Models.

And later, I found its 2026 follow-up, Decoupled DiLoCo.

Ngl, this was one of those moments where two completely different areas of computer science suddenly started making sense together.

So let's unpack it. No crazy AI prerequisites needed.

1. First, why is multi-leader replication so complicated?

Let's say you're building a globally distributed database.

You have one data center in India and another in Europe. Instead of routing every write to a single leader, you allow both data centers to accept writes locally.

Sounds great, right?

Users get lower write latency, and your application doesn't have to depend on a single geographical location to accept updates.

But now imagine both data centers update the same record at roughly the same time.

What happens?

You need a replication protocol, a conflict-resolution strategy, and clearly defined consistency guarantees. If the network goes down, the two locations might continue accepting writes without immediately seeing each other's changes.When connectivity returns, you need to reconcile those changes according to your system's rules.And if you try to synchronously coordinate every write across geographically distant nodes, network latency can become a serious bottleneck.

That's why multi-leader replication isn't a free performance upgrade. It introduces trade-offs that you need to understand before choosing it.

The important distinction is that multi-leader replication isn't inherently bad. It's simply a design with costs that might not be worth paying for your particular workload.

But here's the idea I couldn't stop thinking about:

What if independent components could make meaningful progress locally and only coordinate globally when they actually needed to?

Turns out, that's a pretty interesting question to ask about AI training.

2. Why does training an AI model require so much coordination?

Let's say you're training a neural network using thousands of GPUs.

One common approach is data parallelism.

Each GPU maintains a replica of the model and processes a different portion of the training data. During backpropagation, each worker calculates gradients that describe how the model parameters should change.

Those gradients need to be combined so the workers can make consistent updates according to the training algorithm.

In standard PyTorch DistributedDataParallel, this synchronization uses a collective operation called all_reduce.

At a high level, all-reduce allows participating workers to combine their values, commonly by summing them, and make the result available to every participant. In DDP, gradient averaging is a central part of this process.

The important thing is that the workers coordinate their updates rather than independently changing the model and hoping everything works out.

And for a tightly coupled training cluster, this works extremely well.

But imagine taking that same communication pattern and spreading the workers across continents.

Now you're dealing with WAN latency, limited bandwidth, network disruptions, and machines that might not all perform at the same speed.

The problem isn't just the amount of data being transferred.

It's how frequently the system needs to communicate before useful computation can continue.

If one worker becomes a straggler at a synchronization point, other workers may have to wait. If a worker fails, the distributed job might require recovery before it can proceed.

This is where the analogy to distributed systems becomes interesting.

In databases, coordination can become expensive when independent nodes need to reconcile changes frequently.

In distributed training, coordination can become expensive when independent workers need to exchange information frequently.

They're not the same problem, but they share an important engineering question:

How much coordination do we actually need?

3. Enter DiLoCo: what if we communicated less often?

In November 2023, Google DeepMind published a paper called DiLoCo, short for Distributed Low-Communication.

The basic idea is surprisingly intuitive.

Instead of requiring geographically distributed workers to synchronize their model updates after every local optimization step, let them train locally for many steps before communicating globally.

Think of each group of GPUs as an island.

Each island has enough local infrastructure to perform its training efficiently. Communication within an island can be relatively fast, while communication between islands might be slow or expensive.

Rather than constantly coordinating across the WAN, each island performs a series of local optimization steps and periodically contributes an update to the broader training process.

In the original DiLoCo algorithm, the inner optimizer is AdamW, while the outer optimizer uses Nesterov momentum.

The number of local steps between outer updates is configurable. The paper investigates settings with hundreds of local steps, including 500-step intervals.

Here's the clever part.

The system doesn't need to send every individual update generated during those local steps. Instead, it can communicate information about how the model parameters have changed over the local training phase.

Let's make that concrete.

Suppose the model parameters at the beginning of a local training phase are:

θ_start

After many local optimization steps, the learner has new parameters:

θ_local

The parameter displacement is:

Δθ = θ_local - θ_start

DiLoCo uses a pseudo-gradient based on this parameter difference, with the sign convention defined by the algorithm.

That pseudo-gradient summarizes the effect of multiple local optimization steps. The outer optimizer then uses the aggregated information to update the shared model state.

In other words, instead of constantly communicating the intermediate results of local training, the system can communicate a summary of the progress made over a much longer interval.

This is related to federated averaging, but DiLoCo adapts the idea for large-scale language-model training using its particular inner and outer optimization scheme.

And the results were pretty interesting.

On the C4 dataset, the researchers reported that DiLoCo with eight workers performed comparably to fully synchronous optimization while communicating 500 times less in their experiments.

Let that sink in.

A distributed training approach that communicates dramatically less can still achieve comparable model quality under the conditions tested.

That doesn't mean every AI model can be trained with 500 times less communication. It's a result from a particular experimental setup.

But it demonstrates something important:

Frequent global synchronization isn't always necessary for effective distributed training.

And if you're thinking about training models across distant data centers, that changes the economics of the problem.

Read the original DiLoCo paper.

4. But wait. What happens when the islands don't cooperate?

At this point, you might be thinking that DiLoCo solves the communication problem.

Well, it addresses a major part of it. But reducing communication frequency isn't the same as eliminating synchronization dependencies.

Imagine you have several training islands spread across different data centers.

One island has newer hardware and completes its local work quickly.

Another has older hardware and takes considerably longer.

A third experiences a hardware failure.

Even if the islands communicate less frequently, a training architecture that still requires all participants to reach the same synchronization point can suffer from stragglers and failures.

And this is where the 2026 follow-up becomes particularly interesting.

In April 2026, Google DeepMind published Decoupled DiLoCo, extending the approach to address these coordination and resilience problems.

The central idea is to decouple the learners from the global synchronization process.

Instead of treating the entire training job as one tightly synchronized unit, the system lets separate learner units perform local optimization and communicate asynchronously with a central synchronizer.

Let's break that down.

5. Decoupled DiLoCo: stop waiting for everyone

Imagine five training islands.

In a tightly synchronized system, a global synchronization point can force faster participants to wait for slower ones.

Now imagine that one island temporarily goes offline.

Depending on the training architecture and recovery mechanisms, the failure can disrupt the progress of the broader job.

Decoupled DiLoCo changes this coordination model.

Learners perform their local training and send parameter fragments to a central synchronizer asynchronously. The synchronizer is responsible for aggregating updates and producing new global model state.

The key difference is that the system doesn't require every learner to be ready at precisely the same moment for every global update.

The paper describes several mechanisms that make this possible.

Minimum quorum

The synchronizer can proceed when enough learner contributions are available, rather than requiring every learner to report back.

This helps prevent one slow or unavailable learner from blocking the entire process.

Adaptive grace window

The synchronizer can balance waiting for additional contributions against making progress with the updates already received.

Waiting longer might allow more learners to contribute. Waiting less can reduce the delay caused by stragglers.

Dynamic token-weighted merging

Learners can contribute different amounts of training work. The system accounts for differences in the number of tokens processed when merging updates.

This matters because treating every learner's contribution identically isn't necessarily appropriate when their workloads differ.

Together, these mechanisms make global coordination more flexible.

Notice what's happening here.

We're no longer talking only about reducing how often communication occurs. We're also changing the dependency between local computation and global coordination.

That's a much more substantial architectural change.

And it's the part I find particularly interesting as a software engineer.

We're separating the work that needs to happen locally from the work that needs to happen globally, then designing the coordination mechanism so that delays and failures in one part of the system don't automatically stall every other part.

Of course, asynchronous communication doesn't eliminate all synchronization. Learners still have local dependencies, and the synchronizer still needs to aggregate updates according to the algorithm.

The goal is to remove unnecessary global lock-step dependencies, not to pretend coordination no longer exists.

6. Okay, but how do you transfer a massive model over a WAN?

Let's do some napkin math.

Suppose a model has one trillion parameters, with each parameter represented using 16 bits.

That's two bytes per parameter.

So the raw parameter storage is:

1 Trillion parameters × 2 bytes = 2 Terabytes (TB)

That's 2 TB of raw parameter data, using decimal units.

Now imagine transferring that entire payload over a 2 Gbps network connection.

Since one byte is eight bits:

2 Terabytes × 8 bits = 16,000 Gigabits (Gb)

The ideal transfer time would be:

16,000 Gb ÷ 2 Gbps = 8,000 seconds

That's approximately 2 hours and 13 minutes.

And that's before accounting for protocol overhead, congestion, or other real-world limitations.

Obviously, you don't want your training architecture to depend on repeatedly transferring the entire model across continents at that rate.

But there's an important distinction here.

DiLoCo doesn't solve this problem merely by splitting the model into shards. It reduces how frequently distributed learners need to communicate globally.

Sharding can distribute model state across devices, but it doesn't automatically reduce the total number of bytes required for a given transfer.

The bigger architectural win comes from doing more computation locally between global updates.

Decoupled DiLoCo adds asynchronous communication and a synchronization mechanism that doesn't require every learner to participate in every update.

And we have an actual result to look at here, rather than relying only on hypothetical calculations.

In its April 2026 technical report, Google DeepMind described training a 12-billion-parameter model across four separate US regions using WAN connectivity of approximately 2–5 Gbps.

The researchers reported that this approach completed the training run more than 20 times faster than conventional synchronization methods in their comparison.

They also reported a bandwidth comparison across eight data centers:

• Conventional data-parallel training: 198 Gbps.
• Decoupled DiLoCo: 0.84 Gbps.

Those figures belong to the specific configurations and experiments described by the researchers; they're not universal bandwidth requirements.

Still, they demonstrate why this architecture is worth paying attention to.

The point isn't that network bandwidth has magically stopped mattering.

It's that changing the communication pattern can make a previously impractical network topology viable for training.

Source: Google DeepMind's Decoupled DiLoCo announcement and experimental results.

7. The other interesting consequence: heterogeneous hardware

There's another constraint worth discussing.

Large-scale training systems often benefit from homogeneous hardware because devices with different performance characteristics can complicate scheduling and synchronization.

If one group of devices is much faster than another, tightly synchronized execution can waste time waiting for slower participants.

Now consider the architecture we've been discussing.

Independent learner units can make local progress without requiring every other unit to complete the same amount of work at the same time.

The synchronizer can then account for differences in contributions when merging updates.

That creates an opportunity to combine hardware that might otherwise be difficult to use efficiently in a single tightly synchronized training job.

Google DeepMind reports experiments combining different hardware generations, including TPU v6e and TPU v5p, while maintaining comparable model performance to training on a single hardware type.

That's an important result.

It suggests that training infrastructure doesn't always need to be built around one perfectly uniform pool of accelerators.

Instead, a system can potentially make useful progress with resources distributed across different locations and hardware generations.

Of course, heterogeneous hardware introduces its own challenges: different compute speeds, memory capacities, communication characteristics, and scheduling requirements.

The architecture doesn't make those problems disappear.

It changes how much those differences constrain the entire training job.

And that distinction is huge.

8. So is this basically multi-leader replication for AI?

Not exactly.

And this is where I think the comparison becomes more interesting than the original analogy.

In a multi-leader database, multiple nodes accept application writes and replicate changes. Concurrent writes can create conflicts, and the replication protocol determines how those conflicts and consistency requirements are handled.

In DiLoCo, learners perform local optimization steps and periodically contribute parameter updates to a shared training process.

In Decoupled DiLoCo, the learners can communicate asynchronously with a synchronizer that aggregates their contributions.

These systems have different correctness requirements and different mechanisms.

A database might need to preserve a strict invariant for every committed transaction. A training algorithm can sometimes tolerate delayed or approximate updates, provided the resulting optimization remains effective.

So we shouldn't claim that DiLoCo is literally a multi-leader database architecture.

But the connection is still useful.

Both systems force us to think about the cost of coordination, the degree of independence we can allow between components, and what happens when parts of a distributed system become slow or unavailable.

And both show why the answer isn't always to synchronize everything as frequently as possible.

Sometimes the better approach is to identify which operations genuinely require coordination, let everything else progress independently, and design a robust mechanism for combining the results.

That is a distributed systems problem, regardless of whether the state being coordinated is a database record or a neural network's parameters.

9. What I took away from this

When I started reading about multi-leader replication, I was mostly focused on the things that could go wrong.

Conflicting writes. Replication lag. Consistency guarantees. Network partitions.

And that's fair. Those are real problems.

But looking at DiLoCo and Decoupled DiLoCo made me think about a different question.

Instead of asking how we can synchronize every component more efficiently, what if we first asked whether every component needs to synchronize that frequently at all?

That question leads to a different way of designing distributed systems.

In databases, the answer depends on the application's consistency requirements and workload.

In AI training, the answer depends on the optimization algorithm, communication costs, model quality, and failure model.

The solutions aren't interchangeable. But the underlying engineering instinct is surprisingly similar.

Don't pay the cost of global coordination unless the system actually needs it.

And if you can safely reduce that dependency, you might be able to build systems that are cheaper to operate, more resilient to failures, and capable of using resources that were previously too difficult to combine.

That's what makes these papers interesting to me.

Not that an infamous database anti-pattern somehow became the secret foundation of AI.

It's that a question I first encountered while reading about database replication led me to a different way of thinking about distributed computation.

And honestly, that's one of the best parts of learning distributed systems.

Sometimes the most useful insight isn't a new algorithm or a new technology.

It's recognizing the same fundamental trade-off appearing in two completely different places.

P.S. If you want to explore the research yourself, start here:

• DiLoCo: Distributed Low-Communication Training of Language Models (2023)
• Decoupled DiLoCo: Resilient Distributed Pre-training (2026)
• Google DeepMind's explanation of Decoupled DiLoCo and its experimental results

I'm considering writing a separate post on the optimization side of this, especially how AdamW, pseudo-gradients, and Nesterov momentum fit together. That's a rabbit hole of its own.

Why is this important? Consumer decisions are often driven by the quality of service, which is reflected in these ratings. From a technical perspective, this system must be highly available and low-latency. To talk numbers: let’s aim for a P99 of 50ms with peak traffic of 10k RPS (Requests Per Second).

To accommodate this, we need an efficient caching mechanism and pre-materialized views on a relational DB (meaning all necessary joins are performed beforehand).

Calculating the Scale Imagine we have 500 million reviews in total across all products. If each review is roughly 2KB, we are looking at 1TB of data. While 1TB isn't massive by modern standards, hitting a single-node PostgreSQL or MongoDB instance with 10k GET requests per second would likely lead to a system crash. To solve this, we shard the database. Imagine 5 master nodes, each with 2 replicas—giving us 5 write nodes and 10 read replicas. To ensure High Availability (HA), these replicas should be placed in different Availability Zones (AZ) so that an infra failure in one zone doesn't impact the business.

The Routing Problem How does the system know which DB node holds User A’s data? A naive approach is a simple hash function using a deterministic number to route requests. However, in large systems, cost and scalability are concerns. If your user base grows or shrinks by 40%, a fixed hash function is hard to scale because it requires massive data re-sharding.

This is where Consistent Hashing comes in. Avoiding the jargon: instead of a direct mapping, we use a "circular" range (the hash ring). Imagine a loop from 0 to 100. If you have two nodes, one might cover 0–50 and the other 50–100. If a request hash is 26, it goes to the first node. If you need to add a third node (e.g., at 75), you only need to move the data falling between 50 and 75. In the big picture, the amount of data moved is minimized, which is a huge win for system stability.

Even with efficient sharding, we have to consider "Hot Keys." Imagine a major product launch where a single item's reviews are fetched redundantly millions of times. Even with sufficient read replicas, every database operation adds latency and puts a load on your disk I/O.

Caching helps mitigate this issue. Using a CDN for static content or Redis for frequently accessed review data allows you to serve these requests from memory rather than hitting the database. This drastically reduces the load on your primary storage and ensures that a single popular product doesn't bring down your entire sharded architecture.

Conclusion A combination of consistent hashing, smart caching, and distributed data can solve the majority of scaling headaches. Follow for more—I am Aadinir.