Skip to content
← Insights
Nov 15, 20265 min read

An agent restart should resume the job without repeating the side effect

A long-running agent needs durable progress and an explicit recovery rule. Event logs help, but safe recovery also depends on the systems that receive its writes.

A durable event-log spool connected to a paused workflow board, beside a restart arrow, a saved progress marker and a completed operation receipt.

An agent sends a request to create a document. The receiving service completes the write. Before the agent records the result, its worker crashes. On restart, the worker sees an unfinished step even though the document already exists.

That gap is enough to produce a duplicate if recovery simply starts the action again. Keeping a conversation transcript can help explain what the agent intended. The runtime also needs a rule for resolving what the receiving service actually did.

Save the facts needed for recovery

Give the job a durable identity. Record which input began the job, the actions already attempted, and the outcomes received. These are joint parts of its progress record. A replacement worker should be able to discover that state after the original process disappears.

The record needs to distinguish a planned action from an action whose result was confirmed. It also needs a representation for an uncertain outcome, such as the document request interrupted by the crash. Treating uncertainty as failure loses the information needed to reconcile the external result.

A log can provide an ordered history of these facts. Jay Kreps' foundational essay explains the log as a sequence that can support rebuilding system state. That idea applies to the recovery problem without choosing a particular vendor for the storage. The original essay develops the abstraction.

An HTTP service can persist its progress in a database before responding. A workflow engine can maintain an execution history. A streaming system can distribute events to several consumers. The right comparison is what each implementation commits durably and what recovery can reconstruct from it.

Replaying a result differs from invoking an action

Temporal offers a concrete example of durable execution. Its event history records workflow progress so a worker can reconstruct state after a failure. During replay, commands are matched against recorded events. A previously completed activity can supply its recorded result without executing the external work again. The event-history documentation explains that behavior.

A model request belongs on the side of the boundary that can produce a new result. Calling a model again may produce a different answer even when the input looks similar. Recovery can use a recorded answer when that is the intended semantics. A deliberate new attempt should be recorded as a new attempt.

The uncertain-completion window still needs attention. Temporal's activity documentation describes a case in which external work succeeds but its completion is not recorded before a failure. The activity can then be retried. The receiving operation must tolerate that retry or expose enough state to reconcile it. The activity reference describes the limitation.

For the document example, a stable operation identifier can let the receiving service recognize the same request. Another design can check for the existing result before another creation attempt. The appropriate method depends on the receiving API's contract. The workflow's own history cannot invent a guarantee that the other service lacks.

Order the work that shares a decision

Suppose two events concern one application. One records that approval was granted; another asks the agent to perform the approved action. Processing them in the wrong order can change the decision the worker makes. The system needs an ordering rule for events that affect that application.

Kafka provides ordering within a partition. Assigning events to partitions determines which events share that ordering. Its design documentation also explains why coordinating consumer progress with output written to an external system requires additional work. The Kafka design reference describes those boundaries.

Choose the grouping according to the state being changed. A conversation identifier can work when all decisions are confined to that conversation. If several conversations can modify one account, ordering only within each conversation leaves the account's shared state unresolved. That is a property of the application being built.

A large streaming deployment also adds operational work. A smaller application with one durable job owner may need a simpler implementation. Begin with the recovery and ordering requirements, then choose infrastructure that can satisfy them under the expected workload.

Retain enough history to support the promised recovery

Replay depends on keeping the events it needs. If records expire before a paused job resumes, recovery needs another source of state. A retention setting therefore changes what the system can promise about older work.

Compacting a stream can keep the latest value for an entity while discarding intermediate versions. That can serve a current-state view. An investigation into how that value was reached may require the intermediate history. Decide which purpose a retained dataset serves before removing records from it.

The record also needs to be usable without spreading private input into every diagnostic system. Store the data required for recovery under the application's access policy. A separate operational trace can use identifiers to connect the relevant records. Keeping the whole prompt in every event is a design choice with its own exposure.

Make the crash window part of the test

Stop the worker immediately after the receiving service completes a test write, before local completion is saved. Restart the worker from its durable state. Inspect whether the original result is found, a duplicate is created, or the job remains explicitly blocked for reconciliation. These are distinct observable outcomes.

Repeat the exercise at the other boundaries in the workflow. Keep the expected recovery behavior beside each boundary. The system is ready to resume that job when the replacement worker can reach the intended result without creating an extra side effect.

AI agentsDistributed systemsReliability