Multiplayer AI is a distributed systems problem

Multiplayer AI is a distributed systems problem because people, agents, models, policies, and business tools coordinate across boundaries that can delay, duplicate, reorder, or reject work.

That sentence changes how I think about the product. A shared chat with several cursors may look multiplayer, but the difficult part begins when an agent can update a customer record, a finance approver can pause it, another agent can gather evidence, and a manager can redirect the work while all of those participants are seeing slightly different versions of reality.

The architecture has to make those differences explicit. It has to answer which state is authoritative, who currently owns the next action, whether a command has already happened, what can proceed during a network partition, and how the system recovers without asking the user to reconstruct the work.

This is the first article in Engineering Multiplayer AI, a technical companion to our product explanation of multiplayer AI. The product article explains shared presence, context, and control. This series examines what those promises cost to implement.

Where are the distributed boundaries?

A multiplayer AI run crosses more independent systems than its interface suggests. The browser is connected to an API. The API schedules an agent. The agent calls a model and several tools. Policy checks may occur in another service. A person may approve from email hours later. None of these components shares a clock, a transaction, or a failure mode.

Diagram 1: the real topology behind a shared work surface

The durable work service coordinates independently failing human, model, policy, runtime, and tool boundaries.

 Human participants
 ┌──────────┐  ┌──────────┐  ┌──────────┐
 │ Requester│  │ Approver │  │ Operator │
 └────┬─────┘  └────┬─────┘  └────┬─────┘
      └──────────────┼──────────────┘
                     ▼
          ┌─────────────────────┐
          │ Shared work service │◄──── durable source of truth
          └─────┬────────┬──────┘
                │        │ events
         command│        ▼
                │  ┌─────────────┐
                │  │ Agent queue │
                │  └──────┬──────┘
                ▼         ▼
          ┌──────────┐  ┌──────────────┐
          │ Policy   │  │ Agent runtime│
          │ decision │  └───┬────┬─────┘
          └──────────┘      │    │
                         model   tools
                            ▼    ▼
                       ┌─────┐ ┌──────────────┐
                       │ LLM │ │ CRM / ERP /  │
                       └─────┘ │ email / files│
                               └──────────────┘

Every arrow is a place where an apparently simple action can become uncertain. The request may have been accepted even though the client timed out. The model may finish after the user has cancelled the run. The CRM may create a record and then return a network error. An approval link may be opened after the underlying invoice changed.

If the UI hides that uncertainty, users experience the platform as erratic: a button appears stuck, the same email is sent twice, or work moves from “waiting” to “done” without explaining why. Good distributed architecture is therefore part of the interaction design.

Make the work durable, not the worker

An agent process is disposable. A model session is disposable. A browser connection is disposable. The work cannot be.

The durable object should contain at least:

• a stable work identifier and tenant boundary;
• the requested goal and normalized inputs;
• the current state and monotonically increasing version;
• the responsible human and agent identities;
• plan, policy, and connector versions;
• commands attempted and their idempotency keys;
• evidence, approvals, artifacts, and final outcome.

This object is the coordination point. Workers claim steps from it, perform bounded work, and propose state transitions. They never become the sole owners of business truth.

Diagram 2: work as an explicit state machine

User-visible states are authoritative domain states rather than labels inferred from process logs.

                  ┌───────────┐
            ┌────►│ CANCELLED │
            │     └───────────┘
            │
 ┌────────┐ │  ┌──────────┐  ┌─────────┐  ┌──────────────┐
 │ QUEUED ├─┼─►│ PLANNING ├─►│ RUNNING ├─►│ VERIFYING    │
 └────────┘ │  └────┬─────┘  └────┬────┘  └──────┬───────┘
            │       │             │              │
            │       ▼             ▼              ▼
            │  ┌──────────┐  ┌─────────────┐  ┌───────────┐
            └─►│ FAILED   │  │ WAITING_FOR │  │ COMPLETED │
               └──────────┘  │ _APPROVAL   │  └───────────┘
                             └──────┬──────┘
                                    │ approve / reject / expire
                                    └──────────────► RUNNING or FAILED

 Every transition records:
 actor + prior version + command + policy decision + timestamp

This is more than backend neatness. Explicit states let the product say “waiting for Priya to approve the vendor change” instead of showing a spinner. They let an operator distinguish a recoverable tool outage from a rejected policy decision. They let a person cancel work with a predictable result.

The rule I use is simple: if a state matters to a user, it must exist in the domain model rather than being inferred from logs.

Choose consistency according to consequence

Not every part of the system needs the same consistency model.

I would require an authoritative, serialized decision for:

• consuming an approval;
• assigning or transferring ownership;
• issuing a credential or widening permission;
• committing an irreversible external action;
• marking a work item complete.

I would accept eventual consistency for:

• presence avatars;
• progress summaries;
• search indexes;
• aggregate cost and throughput dashboards;
• derived notifications that can be rebuilt.

This split matters for experience. Strongly consistent presence would make the interface sluggish for little safety benefit. Eventually consistent approval consumption could allow two actors to advance the same work. The correct architecture is not “strong consistency everywhere”; it is “strong consistency exactly where the business invariant lives.”

The commit path should make that boundary explicit through four obligations.

First, it loads the authoritative work version and rejects a command prepared against older state. That prevents a stale interface or agent from silently overwriting a newer human decision. Second, it checks whether the command’s idempotency key already has a recorded result. A retry then returns the prior result instead of repeating the business action.

Third, the service evaluates the actor, policy, and requested state transition together. An agent cannot use a technically valid command to cross a business boundary that the current work state or delegation does not permit. Finally, the service records the domain event, idempotent result, and outgoing messages in one transaction. A transactional outbox can publish those messages later without allowing the durable work record and downstream queues to drift apart.

These are not implementation decorations. Together they define what the user can trust after a timeout, concurrent edit, policy change, or service restart: one accepted command, one authoritative state transition, and a continuation that can always be recovered.

Failure is a normal branch, not an exception

Consider an agent creating a supplier record. The external API accepts the request but the response is lost. The runtime cannot know whether retrying will recover the operation or create a duplicate.

Diagram 3: the ambiguous-success failure

The retry preserves the logical action identity, so a lost response does not create a duplicate supplier.

 Agent runtime        Connector        Supplier API       Work record
      │                   │                  │                  │
      │ create(key=K)     │                  │                  │
      ├──────────────────►│ POST + key K     │                  │
      │                   ├─────────────────►│                  │
      │                   │                  │ create supplier  │
      │                   │        response X│                  │
      │                   │◄ ─ ─ ─ lost ─ ─ ─┤                  │
      │ timeout           │                  │                  │
      │◄──────────────────┤                  │                  │
      │ retry(key=K)      │                  │                  │
      ├──────────────────►│ lookup key K     │                  │
      │                   ├─────────────────►│                  │
      │                   │ existing result  │                  │
      │                   │◄─────────────────┤                  │
      │ success(existing) │                  │ record outcome   │
      │◄──────────────────┤────────────────────────────────────►│

The pleasant user experience—one supplier, one completed task—depends on preserving identity across the retry. “We use retries” is not a reliability strategy unless the receiving boundary can recognize the same logical operation.

When a tool cannot accept an idempotency key, the connector needs a fallback: a deterministic external reference, a preflight lookup, or a reconciliation step. For irreversible actions with no reliable deduplication, the right product choice may be to require approval immediately before execution and surface ambiguous outcomes for human resolution.

Treat time as part of the data

Distributed systems also fail through time. Context becomes stale. Policies change. Credentials expire. The person who requested the work loses access. A plan created on Monday may no longer be valid when approved on Friday.

Every consequential decision should therefore carry:

• what was observed;
• when it was observed;
• which version was used;
• how long the decision remains valid;
• what must be rechecked before execution.

This produces a better approval experience. The interface can warn, “The purchase order changed after this recommendation was prepared,” and offer a one-click refresh. Without temporal metadata, the only choices are unsafe continuation or a mysterious restart.

The architectural test is whether users can recover

Infrastructure metrics are necessary, but the final test is human:

• Can a user tell what the system believes the current state is?
• Can they see who or what owns the next action?
• Can they safely retry without understanding idempotency?
• Can they cancel, redirect, or resume without losing completed work?
• When truth is uncertain, does the product say so?

Those questions connect architecture to trust. A reliable multiplayer AI product does not pretend failures disappear. It contains them, records them, and gives people a clear route forward.

The next article explains why approval gates are a systems primitive. The final part covers agent identity without human impersonation. For the product-level category definition, read multiplayer AI.

Build for recoverable work

The hardest part of multiplayer AI is not connecting several people to the same model response. It is preserving business truth while many independent participants act on it.

SoftworkerAI is being built around durable work, explicit state, scoped authority, approval boundaries, and evidence captured during execution. If you are designing or evaluating this architecture, join the early-access program and tell us which failure modes matter most in your workflows.