# Canonical Run and Execution Model
**Status:** Canonical implementation companion
**Date:** 2026-09-02
**Parent architecture:** [Canonical Platform Architecture](./CANONICAL_ARCHITECTURE.md)
**Scope:** Runs, execution, sandbox protocol, credentials, context, usage, and multi-agent semantics
## 1. Purpose
This document completes the execution design for the Cloudflare-centric coding and research agent platform. It is subordinate to, and deliberately compatible with, the four-plane architecture in `CANONICAL_ARCHITECTURE.md`.
It does not reopen the platform choices. It specifies how those choices cooperate safely under retries, Durable Object eviction, Workflow replay, provider failure, cancellation, budget pressure, context growth, and multi-agent concurrency.
The design establishes seven implementation contracts:
1. the logical run model and state machine;
2. provider-neutral execution contracts;
3. the Agent-to-execution-to-sandbox protocol;
4. the credential and egress broker;
5. context acquisition, compaction, memory, and provenance;
6. metering, reservations, cost, billing, and reconciliation;
7. multi-agent delegation, isolation, merging, and cancellation.
## 2. Required Outcomes
The runtime must provide all of the following:
- durable acceptance of a user-visible run;
- safe recovery after Worker, Agent, Fiber, Workflow, or sandbox interruption;
- resumable client event streams without making the client authoritative;
- effectively-once externally visible effects on top of at-least-once delivery;
- deterministic cancellation and terminal-state arbitration;
- no raw platform credential inside untrusted execution;
- deny-by-default execution policy that cannot be relaxed by a model or sandbox;
- auditable selection of model, tool, execution provider, context, and policy;
- bounded context, compute, time, output, cost, delegation depth, and fan-out;
- provider fallback only when the effective security policy remains enforceable;
- isolation of concurrent agent edits and an explicit integration owner;
- durable usage and cost evidence without treating telemetry as the billing ledger.
The runtime does **not** claim:
- exactly-once message delivery;
- exactly-once process execution;
- transparent live migration of a running sandbox;
- rollback of an external side effect that has already been accepted upstream;
- unrestricted open-web access beside private source code;
- automatic trust in model summaries, child-agent results, browser content, or tool output;
- provider billing data that is final before provider reconciliation.
## 3. Compatibility With the Four Planes
| Concern | Authoritative plane/component | Execution-design responsibility |
|---|---|---|
| API admission | Control / Hono | Authenticate, authorize, validate, deduplicate, reserve budget, create run |
| Live coordination | Control / Agent + Durable Object | Serialize the root run, sequence live events, coordinate Fibers and children |
| Agent-local durable work | Control / Agent Fiber | Checkpoint I/O-oriented loops, recover, inspect, cancel |
| Independent durable process | Control / Cloudflare Workflow | Own retryable multi-step graph, waits, approvals, and long-lived processes |
| Untrusted compute | Execution / Cloudflare Sandbox | Primary shell, Git, build, test, and sandbox-local browser execution |
| Ordinary fallback | Execution / E2B | Reconstruct from portable checkpoint when policy-equivalent fallback is possible |
| Heavy compute | Execution / Modal | Consume bounded input artifacts and return output artifacts |
| Business state | Data / Neon | Runs, versions, events, tasks, attempts, effects, policies, budgets, approvals |
| Live operational state | Data / Agent SQLite | Fiber snapshots, live stream replay, active handles, local budget allocations |
| Independent step state | Data / Workflows | Durable Workflow steps, retries, sleeps, events, status |
| Large immutable data | Data / R2 | Artifacts, log segments, context objects, native and portable workspace checkpoints |
| Model inference | Intelligence / AI Gateway | Enforce the approved capability route and normalized protocol adapter |
| Quality evidence | Intelligence / Langfuse | Traces, datasets, experiments, scores, and online evaluation |
| Credentialed egress | Control / outbound handler and broker | Resolve execution identity, enforce grant, attach secret, audit, sanitize |
No lower-trust component may directly mutate a higher-trust component's authoritative state. It submits a validated command or result to the owning service.
## 4. Runtime Vocabulary and Identity
### 4.1 Logical hierarchy
```text
Root Run
└── Run Task (root agent task)
├── Attempt
│ ├── Model Operation
│ ├── Tool Operation
│ ├── Execution Session
│ │ └── Process
│ └── External Side Effect
├── Delegated Child Task
│ └── its own Attempts and Operations
└── Workflow Instance
└── durable Workflow Steps
```
### 4.2 Definitions
| Term | Meaning |
|---|---|
| **Root run** | One user-visible, idempotently accepted objective with one global policy, deadline, budget envelope, event timeline, and terminal result |
| **Run task** | One bounded assignment in the root run graph, owned by one Agent or Workflow; the root task owns the final response |
| **Attempt** | A physical effort to execute a task after claim, retry, recovery, or provider replacement |
| **Operation** | One model, tool, broker, browser, artifact, or execution action with a stable logical identity |
| **Execution session** | A provider-backed isolated workspace associated with one attempt and one immutable `ExecutionPolicy` |
| **Process** | A container-generation-scoped command or PTY within an execution session |
| **Side effect** | An externally visible mutation, such as a Git push, PR, deployment, message, purchase, or remote write |
| **Workspace checkpoint** | A consistent recovery boundary containing native and/or portable workspace representations |
| **Delegation** | A parent-authorized edge from one task to a child task with scoped context, authority, budget, and result schema |
| **Context snapshot** | The immutable, provenance-bearing set of context items rendered for one model call |
| **Usage event** | An append-only measurement emitted by one operation and deduplicated by source identity |
### 4.3 Identifiers
Application identifiers are opaque and tenant-scoped. UUIDv7 is the default application ID format because it is sortable without encoding business meaning. External platform identifiers are stored separately.
```text
run_id logical root run
task_id node in the run graph
attempt_id physical task attempt
operation_id logical operation
execution_id logical execution session
provider_execution_id Cloudflare Sandbox / E2B / Modal handle
container_generation distinguishes replacement containers under one sandbox ID
process_id provider process handle; valid only in its container generation
workflow_id Cloudflare Workflow instance ID
fiber_id Agent Fiber ID
delegation_id parent-child edge
side_effect_id idempotency identity for an external mutation
context_snapshot_id model-call context manifest
usage_event_id append-only meter identity
trace_id cross-plane trace
```
Cloudflare Workflow instance IDs are derived from a run/task/purpose/generation tuple using a bounded, platform-valid encoding. A Workflow ID is never reused for a later logical invocation.
### 4.4 Version and hash set
Every run records the versions that affect reproducibility and recovery:
- API and contract schema version;
- runtime release and Agent class version;
- Workflow definition version;
- `ExecutionPolicy` version and canonical SHA-256 hash;
- model-routing policy and prompt versions;
- tool-registry version;
- execution image/toolchain version;
- context compiler version;
- price-book and billing-policy versions.
Canonical hashes use deterministic JSON serialization. A hash mismatch is a hard failure, not a warning.
## 5. The Root Run State Machine
### 5.1 States
| State | Meaning | Active work allowed? |
|---|---|---:|
| `accepted` | Hono durably created the run, idempotency receipt, initial policy, budget reservation, and dispatch outbox entry | No new work until coordinator acceptance |
| `queued` | The owning Agent or Workflow durably accepted dispatch; the run is waiting for its execution slot | No |
| `running` | One current coordinator epoch may admit model, tool, child, Workflow, or execution operations | Yes |
| `waiting` | The run is durably waiting for user input, approval, external event, retry time, child completion, or provider capacity | Only the awaited/reconciliation operation |
| `paused` | Work is deliberately suspended, active grants are revoked, and a safe checkpoint has been taken or accounted for | No |
| `cancelling` | Cancellation is durable; new operations are forbidden while active work and effects are reconciled | Cancellation/reconciliation only |
| `succeeded` | The root result and terminal usage summary are committed | No |
| `failed` | The run cannot meet its contract and has a classified terminal error | No |
| `cancelled` | Cancellation completed; remaining effects and consumed usage are accounted for | No |
| `expired` | A deadline, approval TTL, retention limit, or maximum wait elapsed | No |
`succeeded`, `failed`, `cancelled`, and `expired` are terminal and immutable.
`waiting` always has one structured reason:
```text
user_input
approval
external_event
retry_backoff
provider_capacity
child_tasks
workflow
reconciliation
```
### 5.2 State diagram
```mermaid
stateDiagram-v2
[*] --> accepted
accepted --> queued: coordinator durably accepts
accepted --> cancelling: cancel requested
accepted --> failed: dispatch permanently invalid
accepted --> expired: admission deadline
queued --> running: coordinator claims epoch
queued --> cancelling: cancel requested
queued --> failed: non-retryable startup failure
queued --> expired: queue deadline
running --> waiting: durable wait established
running --> paused: safe pause committed
running --> cancelling: cancel requested
running --> succeeded: result + usage commit
running --> failed: terminal classified error
waiting --> running: awaited event satisfied
waiting --> queued: capacity/retry wakeup
waiting --> paused: pause requested
waiting --> cancelling: cancel requested
waiting --> failed: awaited event rejects/fails
waiting --> expired: wait/deadline expires
paused --> queued: resume reauthorized
paused --> cancelling: cancel requested
paused --> expired: retention/deadline expires
cancelling --> cancelled: authority revoked; active work accounted for
succeeded --> [*]
failed --> [*]
cancelled --> [*]
expired --> [*]
```
Once `cancelling` is committed, the run cannot become `succeeded`. If success committed first, a later cancellation request returns the existing successful terminal state. This rule provides deterministic cancellation-versus-completion arbitration.
`cancelled` means the platform revoked further authority, stopped or detached its controlled work, and durably accounted for remaining handles. It does not claim that an upstream effect was reversed. Unknown or irreversible effects are retained explicitly in the terminal result and continue through operational reconciliation where possible.
### 5.3 Transition contract
Every transition is executed by the trusted `RunStateService` in a Neon transaction:
```ts
type RunTransitionCommand = {
schemaVersion: 1;
commandId: string;
runId: string;
expectedRunVersion: number;
expectedCoordinatorEpoch?: number;
from: RunStatus[];
to: RunStatus;
reason: RunReason;
source: {
kind: "api" | "agent" | "fiber" | "workflow" | "reconciler";
id: string;
sourceEventId: string;
};
data?: Record<string, unknown>;
occurredAt: string;
};
```
The transaction:
1. deduplicates `commandId` and `sourceEventId`;
2. locks or compare-and-swaps the run's current `version`;
3. validates the transition table and coordinator fence;
4. applies status, reason, timestamps, and projections;
5. increments `run.version`;
6. appends a sparse authoritative `run_event` with the next lifecycle sequence;
7. inserts any required `control_outbox` entries;
8. commits atomically.
The service returns the already committed result for an exact duplicate. A stale or contradictory command receives `VERSION_CONFLICT`, then reloads state rather than guessing.
### 5.4 Terminalization contract
Before committing a terminal root state, the coordinator must establish and record:
- no new operation, delegation, Workflow, execution, approval, or broker grant can be admitted;
- all required tasks satisfy the root success/failure policy;
- active internal handles are stopped, terminal, or explicitly registered for post-terminal reconciliation;
- every credential grant is revoked, expired, or exhausted;
- the final result and required artifacts are schema-valid, durable, and content-addressed;
- external effects are confirmed, rejected, compensated, or explicitly classified as unknown/irreversible;
- consumed usage known at that moment is committed, remaining reservations have release intent, and later provider reconciliation can append adjustments;
- the final user message, run summary, terminal lifecycle event, and projection outbox are committed consistently.
A run need not wait for a provider invoice to terminalize. Estimated or reported usage may reconcile later without changing the run's terminal state.
### 5.5 Sparse lifecycle events versus live stream events
There are two intentionally different event streams:
- **Neon `run_events`:** sparse, authoritative lifecycle, policy, approval, effect, and terminal events.
- **Agent SQLite `stream_events`:** high-frequency user-visible progress, model chunks, tool progress, child progress, and live replay.
The Agent assigns a monotonically increasing `stream_seq` per root run. Child, Workflow, model, and execution events arrive with stable `source_event_id`; the Agent deduplicates them, assigns the next root stream sequence, persists, and then broadcasts.
Final user and assistant messages are committed to Neon. Stream chunks are coalesced and do not become one Neon row per token. Large retained event payloads are stored in R2 and referenced by digest.
### 5.6 Public API contract
Hono remains the canonical boundary:
```text
POST /v1/runs
GET /v1/runs/{run_id}
GET /v1/runs/{run_id}/events?after={cursor}
POST /v1/runs/{run_id}/cancel
POST /v1/runs/{run_id}/pause
POST /v1/runs/{run_id}/resume
POST /v1/runs/{run_id}/input
POST /v1/runs/{run_id}/approvals/{approval_id}
GET /v1/runs/{run_id}/artifacts
```
Create, cancel, pause, resume, input, and approval requests require an idempotency key. The browser may use Agent WebSockets for live events, but user-authoritative commands still pass through the canonical API or an Agent callable method that invokes the same Hono/service contract.
`POST /v1/runs` returns `202 Accepted` with a stable receipt after the Neon admission transaction commits. Repeating the same key and canonical request fingerprint returns the same receipt. Reusing the key with a different fingerprint returns `409 IDEMPOTENCY_CONFLICT`.
The canonical fingerprint includes authenticated tenant, operation, semantic route, project/resource identity, objective, input and attachment digests, requested execution mode, and permitted policy hints. It excludes transport-only request IDs, tracing headers, and receipt timestamps.
Event cursors are opaque. If the requested Agent replay range has expired, the API returns a current durable snapshot and a new cursor rather than silently omitting history.
## 6. Admission, Coordination, and Fencing
### 6.1 Admission transaction
Run admission performs this sequence:
1. Better Auth resolves the session.
2. Hono resolves tenant/project/resource authorization.
3. Zod validates and canonicalizes the request.
4. The policy service compiles the initial immutable `ExecutionPolicy` envelope and tool/model constraints.
5. The budget service selects a price-book version and reserves the root run envelope.
6. Neon inserts the run, request fingerprint, initial task, policy/hash, budget reservation, `accepted` event, and dispatch outbox record in one transaction.
7. After commit, Hono calls the owning Agent through RPC and uses `waitUntil` only as delivery acceleration.
8. A scheduled `RunReconcilerWorkflow` scans undelivered outbox and stale `accepted` runs, retrying the same dispatch identity.
The Neon outbox is authoritative for delivery intent. Direct Agent RPC is an optimization; a lost response cannot create a second run.
### 6.2 Agent mapping and root serialization
The root coordinator is a Cloudflare Agent/Durable Object deterministically addressed by tenant plus durable conversation/agent identity. The default is one actively mutating root run per conversation Agent. Additional root runs queue unless product policy explicitly permits independent run lanes.
Parallelism occurs through child sub-agents, Workflows, model calls permitted by policy, and isolated execution tasks. This keeps conversation state and the final response under one writer without turning one global Durable Object into a bottleneck.
Agent schema initialization uses `blockConcurrencyWhile()` only for local SQLite migrations. External I/O never runs while all Agent concurrency is blocked.
### 6.3 Durable Agent acceptance
The Agent RPC `acceptRun()`:
1. validates run ownership, version, and current status against Neon;
2. deduplicates the dispatch source event;
3. stores a local live-run record;
4. calls `startFiber()` with an idempotency key derived from `run_id` and root task;
5. transitions the run from `accepted` to `queued` only after the Fiber is durably accepted;
6. publishes a live event and returns the retained Fiber receipt.
`startFiber()` is used instead of an untracked fire-and-forget promise because the root task requires durable acceptance, status inspection, idempotency, and cancellation.
The Fiber callback begins with an admission gate: it reloads Neon and cannot dispatch a model, tool, Workflow, child, or execution operation until the `queued` and subsequent `running` transitions commit. If callback start races the enclosing RPC response, both paths use the same transition command identity and converge idempotently.
### 6.4 Coordinator epoch and lease
Every active attempt has a monotonically increasing `coordinator_epoch` and bounded lease recorded in Neon:
```text
lease_owner
lease_expires_at
coordinator_epoch
attempt_id
run_version
```
Claiming or recovering a run increments the epoch. Every operation, broker grant, execution command, side-effect record, and task transition carries that epoch. A stale actor cannot commit results or use a broker grant after a newer epoch exists.
Rules:
- A lease is never held across an unbounded wait.
- Before and after external I/O, the coordinator revalidates cancellation, run version, and epoch before committing the result.
- Long external work is represented by a durable handle and a `waiting` state; a Workflow, Fiber checkpoint, or scheduled reconciliation wakes to inspect it.
- Lease expiry does not prove work stopped. Recovery first enters reconciliation and inspects known handles before replay.
- Brokered credential operations fail closed if the epoch is stale or Neon cannot verify it.
### 6.5 Optimistic concurrency
Durable Object single-threaded execution prevents concurrent JavaScript inside one activation, but it does not eliminate races with Workflow callbacks, API cancellation, retries, external I/O, or a recovered coordinator. Neon version checks and epochs remain mandatory.
Agent SQLite writes that must be atomic are executed synchronously without external awaits between related statements. In-memory state is only a cache after durable local state.
## 7. Attempts, Operations, and Replay Safety
### 7.1 Task lifecycle
Run tasks use:
```text
pending -> ready -> running -> waiting
| |
+----------+-> succeeded
+------------> failed
+------------> cancelled
pending/ready --------------------> skipped
```
`pending` means dependencies are incomplete; `ready` means dependencies, context, policy, and budget are satisfied. Terminal task states are immutable. `skipped` requires an explicit dependency, quorum, cancellation, or supersession reason.
The root run does not mechanically mirror one child state. The root Agent evaluates required tasks, declared failure policy, success criteria, artifacts, and effects before requesting a root terminal transition.
### 7.2 Attempt lifecycle
Task attempts use a separate state machine:
```text
created -> claiming -> running -> waiting -> reconciling
\-> succeeded
\-> failed
\-> cancelled
\-> lost
```
A new attempt may follow `failed` or `lost` only when task policy permits retry. Attempts are immutable after terminal state. The logical task aggregates attempts and remains one node in the run graph.
### 7.3 Operation lifecycle and replay classes
Operations use:
```text
planned -> reserved -> dispatched -> running -> succeeded
| -> failed
| -> cancelled
+------ -> unknown
```
`unknown` means dispatch may have reached an external boundary but no safe terminal outcome is known. It must be reconciled and cannot be treated as a retryable failure by default.
Every operation declares one replay class before dispatch:
| Replay class | Meaning | Recovery action |
|---|---|---|
| `pure` | No externally visible mutation; same inputs may be recomputed | Safe to repeat, while recording additional platform cost |
| `idempotent` | Mutation is protected by a stable upstream or application key | Inspect, then safely repeat with the same key |
| `reconcilable` | Mutation may have occurred; upstream status can be queried | Query by stable effect identity before retry |
| `non_replayable` | Mutation cannot be safely queried or deduplicated | Never repeat automatically; produce `SIDE_EFFECT_OUTCOME_UNKNOWN` and require resolution |
Model calls are normally `pure` from the business-state perspective even though retries consume cost. Repository reads and deterministic builds are `pure`. Artifact writes by content digest are `idempotent`. Git pushes, PR creation, deployments, and remote writes are `idempotent` or `reconcilable` only when the broker supplies a stable effect identity.
### 7.4 Side-effect ledger
Before an external mutation, Neon records:
```text
side_effect_id
root_run_id / task_id / operation_id
coordinator_epoch
provider and action
canonical request digest
upstream idempotency key
state: prepared | submitted | confirmed | rejected | unknown | compensated
upstream handle
approval_id
created_at / submitted_at / reconciled_at
```
The prepared record is committed before dispatch. The broker changes it to `submitted` in the same trusted operation that authorizes the upstream call where possible, then records the response. A retry first reads this ledger and queries upstream when needed.
The platform promises effectively-once **effects**, not exactly-once execution.
## 8. Recovery Semantics
### 8.1 Fiber recovery
Each Fiber checkpoint contains only JSON-serializable recovery data:
```ts
type RunFiberSnapshot = {
schemaVersion: 1;
runId: string;
taskId: string;
attemptId: string;
coordinatorEpoch: number;
phase: string;
pendingOperationId?: string;
workflowId?: string;
executionId?: string;
providerHandle?: string;
streamCursor?: string;
contextSnapshotId?: string;
lastLifecycleEventId: string;
};
```
`onFiberRecovered()` never blindly re-runs the lost closure. It:
1. reloads the authoritative run, task, attempt, cancellation, policy, and budget state from Neon;
2. stops if the run is terminal or cancelling;
3. validates snapshot schema/runtime compatibility;
4. inspects any Workflow, model, tool, broker, or provider handle;
5. reconciles an already completed operation;
6. retries only under the declared replay class;
7. creates a new attempt/epoch when ownership must change;
8. resolves the managed Fiber with a classified outcome.
Recovery hooks are versioned application code. The run stores a runtime version so deployments retain compatible recovery handlers during the maximum in-flight retention window.
### 8.2 Workflow replay
Workflow side effects occur inside named durable steps. Logic outside durable steps is treated as replayable control flow and must not cause a mutation.
Each Workflow step:
- has a stable versioned name and optional key for repeated logical items;
- performs one coherent external operation or one atomic database action;
- uses a side-effect or operation idempotency identity;
- opens and closes its own Hyperdrive/database connection within the step;
- returns bounded, serializable data or an R2 reference;
- reports progress outside the authoritative step result only as non-gating telemetry.
Workflow instance IDs are unique. Duplicate creation resolves the existing mapped instance rather than inventing a new logical process.
### 8.3 Failure recovery matrix
| Failure | Recovery rule |
|---|---|
| Worker request lost after run commit | Idempotent API retry returns same receipt; outbox reconciler redelivers dispatch |
| Agent/Durable Object evicted | Agent SQLite survives; managed Fiber recovery reloads Neon and inspects handles |
| WebSocket disconnect | Agent persists/coalesces stream events; client resumes from cursor |
| Fiber interrupted | `onFiberRecovered()` applies replay-class recovery; no automatic closure replay assumption |
| Workflow Worker restarts | Workflow engine resumes durable steps; side effects remain step-scoped and idempotent |
| Sandbox process exits | Persist exit, logs, output manifest, and usage; classify user-code versus platform failure |
| Sandbox container is replaced | Process IDs/cursors become invalid; restore checkpoint and create a new attempt/generation |
| Cloudflare Sandbox is unavailable | Fallback to E2B only from a portable checkpoint and only if policy-equivalent |
| Heavy task fails in Modal | Retry from immutable input artifacts according to replay policy |
| Neon unavailable | Do not admit new paid/effectful operations or broker calls; retain bounded local progress and retry |
| R2 unavailable | Do not claim artifact/checkpoint durability; wait/retry or fail without discarding local handle |
| AI Gateway/model response uncertain | Mark model operation uncertain, inspect available gateway/provider record, otherwise retry as a new cost-bearing attempt |
| Broker response lost | Reconcile by `side_effect_id`; never assume failure and repeat blindly |
### 8.4 Reconciler
A scheduled `RunReconcilerWorkflow` processes bounded batches of:
- `accepted` runs without coordinator receipt;
- expired active leases;
- `cancelling` runs with active handles;
- `waiting` runs whose wake time has elapsed;
- operations in `submitted` or `unknown` state;
- execution sessions with stale heartbeats;
- unfinalized usage reservations;
- outbox rows without delivery acknowledgement;
- terminal runs missing final projections or retention scheduling.
Reconciliation is idempotent and fenced. It repairs delivery and projections; it does not become a second state authority.
## 9. Pause, Approval, Cancellation, and Expiry
### 9.1 Pause
Pause is a controlled suspension, not cancellation.
1. Record pause intent.
2. Stop admitting new operations and child tasks.
3. Revoke active credential grants.
4. Ask running replay-safe processes to stop at a checkpoint.
5. Reconcile effectful operations already submitted.
6. Create a workspace checkpoint if safe and policy allows.
7. Commit `paused` only when no uncontrolled work remains.
Resume reauthenticates the actor, recompiles effective policy, rechecks deadline and budget, creates a new coordinator epoch, and restores from the latest compatible checkpoint. Historical grants never reactivate.
### 9.2 Approvals
An approval is bound to exact content:
```ts
type ApprovalRequest = {
approvalId: string;
runId: string;
taskId: string;
requestedByOperationId: string;
action: string;
resource: string;
requestDigest: string;
policyHash: string;
previewArtifactIds: string[];
risk: "medium" | "high" | "critical";
expiresAt: string;
};
```
Changing the action, resource, payload, artifact, policy, or relevant base revision invalidates the approval. The first valid approve/reject decision is immutable; an identical retry returns it and a conflicting retry is rejected.
Only the root Agent presents user approval requests. Child agents and Workflows submit approval needs to the root run. Workflows wait through a durable event and continue only after the Approval Service validates the decision and current policy.
### 9.3 Cancellation transaction
The cancellation API performs one authoritative Neon transaction:
1. compare-and-swap the run into `cancelling` unless already terminal;
2. record actor, reason, scope, timestamp, and idempotency receipt;
3. increment the coordinator epoch and revoke all active run and descendant credential grants;
4. make outstanding child/operation allocations unusable under the new run state and record reservation-release intent;
5. append the cancellation lifecycle event;
6. enqueue Agent, Workflow, child, and execution cancellation commands.
After commit:
- the Agent aborts new model/tool dispatch and calls `cancelFiber()` for managed Fibers;
- every registered Workflow is sent termination or a cancellation event according to its cleanup contract;
- child agents are cancelled transitively, then aborted if they do not stop;
- model HTTP streams are aborted;
- sandbox processes receive graceful termination, then force termination after the policy grace period;
- the execution adapter terminates the session when no further reconciliation is needed;
- the broker denies all calls for the revoked epoch/grants;
- actual consumed usage and irreversible effects are finalized;
- safe partial outputs may be retained as explicitly partial artifacts;
- `cancelled` is committed with `clean`, `partial_outputs`, or `partial_effects` outcome.
Cloudflare sub-agent abortion stops an active child but preserves its storage; deletion occurs only after reconciliation and retention. Child-started Workflows are not assumed to be automatically controlled by the parent: every Workflow ID must be registered in the root run graph for explicit cancellation.
### 9.4 Expiry
Expiry uses the cancellation machinery with an `expired` terminal result. The expiry reason is retained:
```text
run_deadline
queue_deadline
user_input_timeout
approval_timeout
paused_retention
workspace_retention
```
Expiry never silently approves an action. Approval timeout defaults to rejection/expiry unless a specific low-risk product rule defines otherwise.
## 10. Provider-Neutral Execution Contract
### 10.1 Boundary
Agents and Workflows call the trusted Execution Service. They never receive Cloudflare Sandbox, E2B, or Modal API credentials and never directly construct provider SDK clients.
```text
Agent/Fiber/Workflow
-> Execution Service contract
-> policy and budget validation
-> provider adapter
-> untrusted execution environment
```
### 10.2 Session contract
```ts
type CreateExecutionRequest = {
schemaVersion: 1;
commandId: string;
runId: string;
taskId: string;
attemptId: string;
coordinatorEpoch: number;
executionPolicyId: string;
executionPolicyHash: string;
workspace:
| { mode: "empty"; imageVersion: string }
| { mode: "restore"; checkpointId: string; imageVersion: string };
deadlineAt: string;
traceContext: TraceContext;
};
type ExecutionHandle = {
executionId: string;
provider: "cloudflare" | "e2b" | "modal";
providerExecutionId: string;
containerGeneration: string;
state: ExecutionState;
policyHash: string;
imageVersion: string;
createdAt: string;
};
type ExecutionState =
| "requested"
| "provisioning"
| "ready"
| "busy"
| "checkpointing"
| "stopping"
| "stopped"
| "lost"
| "failed";
```
### 10.3 Process contract
Shell-string interpolation is not the default. Commands use explicit argument vectors:
```ts
type ProcessSpec = {
schemaVersion: 1;
operationId: string;
executionId: string;
coordinatorEpoch: number;
argv: string[];
cwd: string;
env: Record<string, string>; // non-secret values only
stdin?:
| { kind: "inline"; value: string }
| { kind: "artifact"; artifactId: string };
terminal: "none" | "pty";
timeoutMs: number;
outputLimitBytes: number;
replayClass: "pure" | "idempotent" | "reconcilable" | "non_replayable";
sideEffectId?: string;
expectedOutputs?: Array<{
path: string;
mediaType?: string;
required: boolean;
}>;
};
type ProcessHandle = {
processId: string;
providerProcessId: string;
executionId: string;
containerGeneration: string;
launchSpecHash: string;
state: ProcessState;
startedAt?: string;
};
type ProcessState =
| "created"
| "starting"
| "running"
| "exited"
| "killed"
| "timed_out"
| "lost"
| "failed";
```
Rules:
- `cwd` must resolve inside the authorized workspace root.
- Environment variables contain configuration and operation IDs, never upstream credentials.
- A shell interpreter is invoked explicitly in `argv` only when policy permits shell syntax.
- PTY access is exceptional, time-bounded, audited, and not used for normal autonomous commands.
- An effectful process requires a prepared `side_effect_id` or is rejected.
- Observation timeout or client abort does not imply the process stopped; the adapter must explicitly signal/kill it.
- Provider process IDs and log cursors are valid only for the recorded container generation.
### 10.4 Normalized execution errors
```text
POLICY_UNSATISFIED provider cannot enforce effective policy
PROVIDER_CAPACITY eligible for bounded retry/fallback
PROVIDER_UNAVAILABLE eligible for bounded retry/fallback
PROVISION_FAILED session could not start
WORKSPACE_RESTORE_FAILED snapshot invalid, incompatible, or unavailable
PROCESS_USER_FAILURE command ran and returned a user/code failure
PROCESS_TIMEOUT provider deadline terminated process
PROCESS_LOST container/process disappeared
OUTPUT_LIMIT_EXCEEDED output collection stopped at policy limit
ARTIFACT_INVALID expected artifact missing or digest mismatch
SIDE_EFFECT_UNKNOWN effect may have occurred and needs reconciliation
CANCELLED operation stopped due to durable cancellation
INTERNAL trusted runtime invariant failure
```
Errors carry `retryable`, `replayClass`, `safeMessage`, `internalDetailsRef`, `provider`, `operationId`, and trace context. Raw provider diagnostics are redacted before reaching the model or user.
### 10.5 Provider routing and fallback
Provider selection occurs only after policy eligibility filtering:
1. Can the provider enforce network, credential, data-class, repository, region, resource, and retention constraints?
2. Can it restore the required checkpoint representation?
3. Does it provide the required image, filesystem, browser, CPU, memory, or GPU capability?
4. Is it healthy and within quota?
5. Does it meet latency and cost policy?
Cloudflare Sandbox is selected for eligible ordinary workloads. E2B is not a hot replica; it is a recovery provider. Fallback occurs only before execution starts or at a consistent portable checkpoint. Modal receives immutable input artifacts for specialized tasks and returns immutable outputs.
No provider fallback may widen egress, expose credentials, drop audit, weaken data handling, or change the approved operation.
## 11. Agent-to-Sandbox Protocol
### 11.1 Protocol shape
The logical protocol is between the Agent/Workflow and the trusted Execution Service. The provider adapter translates it to the actual Sandbox SDK or alternate-provider APIs.
Every command contains:
```text
schema_version
command_id
root_run_id
task_id
attempt_id
operation_id
coordinator_epoch
execution_id, when allocated
policy_id + policy_hash
deadline
trace_context
```
Every event contains:
```text
schema_version
source_event_id
command_id
root_run_id
task_id
attempt_id
operation_id
execution_id
container_generation
provider_sequence or cursor
event_type
payload or R2 payload reference
occurred_at
trace_context
```
### 11.2 Commands
| Command | Purpose | Idempotency behavior |
|---|---|---|
| `execution.create` | Allocate provider session with policy | Same command returns same logical handle or reconciles uncertain creation |
| `workspace.restore` | Restore a native or portable checkpoint | Same checkpoint/execution is safe to retry after inspection |
| `workspace.materialize` | Materialize authorized input artifacts/files | Content-digest idempotent |
| `process.start` | Start a supervised argv process | Requires operation record; duplicate inspects existing handle before launch |
| `process.signal` | Send graceful or forceful termination | Idempotent for same signal target |
| `process.stdin` | Write bounded input to an approved PTY/process | Sequence-numbered; not replayed after uncertain acknowledgement |
| `artifact.collect` | Hash, validate, and persist declared output | Content-digest idempotent |
| `workspace.checkpoint` | Create native and/or portable checkpoint | Stable checkpoint intent; content-addressed result |
| `preview.expose` | Expose an approved local port through authenticated preview gateway | TTL- and policy-bound |
| `preview.close` | Revoke preview | Idempotent |
| `execution.terminate` | Stop processes and release provider resources | Idempotent |
### 11.3 Events
```text
command.accepted
execution.provisioning
execution.ready
workspace.restored
workspace.materialized
process.started
process.stdout
process.stderr
process.exited
process.signalled
process.lost
artifact.detected
artifact.stored
workspace.checkpointed
preview.ready
preview.closed
limit.warning
policy.denied
execution.stopping
execution.stopped
operation.error
```
Process exit is not task success. The Agent/Workflow validates exit code, expected artifacts, result schema, policy, tests, and any requested acceptance criteria before transitioning the task.
### 11.4 Output streaming and backpressure
- Provider log streams are consumed with provider cursors where available.
- The Execution Service converts logs into UTF-8-safe, bounded chunks and maintains byte offsets per stream.
- The Agent coalesces chunks, persists a replay window in SQLite, assigns root `stream_seq`, and broadcasts over WebSocket.
- Full retained logs are segmented into R2 objects; Neon stores the manifest and digest.
- `maxOutputBytes` applies independently to live output, retained output, and a single event payload.
- When the live client is slow, durable collection continues within limits while nonessential UI chunks are coalesced.
- On output limit, collection truncates with an explicit marker; the process is stopped when policy says output overflow is fatal.
- A cursor from an old container generation is never applied to a new container.
### 11.5 Files and artifacts
The protocol transfers files through manifests and R2, not unbounded RPC bodies:
```ts
type FileManifestEntry = {
relativePath: string;
kind: "file" | "directory" | "symlink";
size: number;
sha256?: string;
mode?: number;
target?: string;
};
```
Paths are normalized and rejected if absolute, traversal-based, device-backed, or outside the workspace. Symlink targets are validated at collection time. Artifact collection re-reads and hashes the final bytes, applies content and malware policy where configured, uploads to R2, then commits metadata in Neon.
### 11.6 Browser and preview operations
- Deterministic Worker-native browser tasks use Browser Run and `@cloudflare/playwright` outside the sandbox protocol.
- Playwright CLI uses the sandbox protocol when browser and local workspace must coexist.
- Local application previews are exposed only through a trusted preview gateway with Better Auth, project/run authorization, TTL, port allowlist, and audit.
- Provider preview tokens or raw wildcard URLs are not treated as authorization and are not placed in model-visible output by default.
## 12. Workspace Checkpoint Model
### 12.1 Two checkpoint representations
Cloudflare native backup/restore is optimized for fast same-provider recovery but is not assumed portable to E2B or Modal. Therefore a logical checkpoint may have two representations:
1. **Native snapshot** — Cloudflare Sandbox directory backup stored in the dedicated R2 workspace bucket for fast copy-on-write restore.
2. **Portable checkpoint** — provider-neutral reconstruction manifest stored in R2.
The portable checkpoint contains:
```text
base repository URL/identity without credentials
base commit digest
submodule and LFS metadata where permitted
tracked patch or content-addressed changed-file manifest
authorized untracked-file artifacts
lockfiles and toolchain/image version
working-directory layout and executable modes
test/build metadata
excluded paths and reasons
checkpoint digest and parent checkpoint
```
It never contains secrets, credential-helper state, upstream cookies, SSH agents, provider tokens, or historical broker grants.
### 12.2 Checkpoint consistency
A checkpoint is created only after process quiescence or a declared application-consistent boundary. The adapter:
1. stops or accounts for mutating processes;
2. flushes files;
3. captures Git and file manifests;
4. produces native and/or portable representations;
5. verifies digests and size limits;
6. uploads immutable objects;
7. commits Neon metadata and lineage;
8. emits `workspace.checkpointed`.
A snapshot upload without committed metadata is garbage-collectable. Metadata is not marked usable until all required objects verify.
### 12.3 Restore
Restore validates tenant, project, policy, snapshot lineage, compatibility, data class, expiry, digest, and image/toolchain constraints. A fresh execution identity and coordinator epoch are always assigned.
Cross-provider restore favors correctness over cache preservation. Dependencies and build caches may be rebuilt rather than transferring opaque provider state.
## 13. Credential and Egress Broker
### 13.1 Grant model
The broker stores authority, not credentials, in the run model:
```ts
type CredentialGrant = {
schemaVersion: 1;
grantId: string;
runId: string;
taskId: string;
executionId: string;
coordinatorEpoch: number;
policyHash: string;
capability: string;
resource: string;
actions: string[];
hosts: string[];
methods: string[];
pathRules: string[];
maxCalls: number;
maxRequestBytes: number;
maxResponseBytes: number;
maxCostMicros?: string;
approvalId?: string;
notBefore: string;
expiresAt: string;
state: "pending" | "active" | "revoked" | "expired" | "exhausted";
};
```
The grant references a secret alias held only in trusted configuration. It never contains secret material.
### 13.2 Cloudflare Sandbox enforcement
Protected Sandbox classes start with internet disabled. `allowedHosts` opens only policy-approved hosts and virtual broker hosts. Outbound handlers run in trusted Worker code outside the sandbox.
For each request the handler:
1. obtains container/sandbox identity from trusted handler context;
2. maps it to `execution_id`, run, task, and current epoch;
3. reloads active run/grant state from Neon for credentialed operations;
4. denies on database uncertainty, stale epoch, cancellation, expiry, exhausted limits, or mismatch;
5. validates scheme, host, port, method, normalized path, query, redirect policy, content type, body digest, and byte limit;
6. resolves the specific secret alias in trusted code;
7. attaches the credential or calls the Cloudflare binding;
8. attaches upstream idempotency identity for mutations;
9. performs the request without permitting an unchecked redirect to a new destination;
10. strips authorization, cookies, signed URLs, internal IDs, and unsafe headers from the response;
11. applies response size/content policy;
12. records audit, side effect, and usage events;
13. returns only the bounded sanitized response.
### 13.3 Virtual capability hosts
Prefer narrow virtual services over general credential attachment:
```text
git-read.internal
git-write.internal
artifacts.internal
deploy.internal
project-api.internal
```
For example, `git-write.internal` exposes only the approved repository and ref actions. It does not turn a GitHub token into general `api.github.com` access.
### 13.4 Direct egress
Unauthenticated direct egress is allowed only for hosts in the active profile, such as approved package registries. Direct responses remain size- and destination-constrained. Direct egress cannot access link-local, private, metadata, internal broker, or alternate proxy destinations except explicit virtual hosts.
### 13.5 E2B and Modal
An alternate provider is eligible for credentialed execution only if it offers equivalent non-extractable workload identity and network enforcement. A bearer token copied into the sandbox is not equivalent.
When equivalent transparent interception is unavailable:
- credentialed actions run as explicit Control Plane tools outside the sandbox; or
- the workload remains on Cloudflare Sandbox; or
- the alternate provider is used only for credential-free artifact computation.
The system fails closed rather than weakening the grant model for fallback.
### 13.6 Revocation and outage
Cancellation atomically revokes grants with the run transition. Broker calls check current state, so no cache TTL can extend credential authority after revocation. If Neon or the policy service is unavailable, credentialed calls are denied. Already issued upstream operations are reconciled through the side-effect ledger.
## 14. Context-Management System
### 14.1 Objectives
Context management must maximize task-relevant evidence while preserving authority, provenance, privacy, and token headroom. It is not merely message truncation.
The context pipeline is:
```text
discover -> authorize -> normalize -> classify -> deduplicate
-> retrieve -> rank -> budget -> render -> hash -> trace
```
### 14.2 Context item
```ts
type ContextItem = {
contextItemId: string;
tenantId: string;
projectId?: string;
runId?: string;
kind:
| "system_policy"
| "developer_instruction"
| "user_message"
| "assistant_message"
| "run_plan"
| "task_result"
| "repository_file"
| "repository_diff"
| "test_result"
| "tool_result"
| "web_source"
| "artifact"
| "memory"
| "summary";
contentRef: { inline?: string; artifactId?: string };
contentSha256: string;
source: ContextSource;
trust: "trusted_control" | "user" | "project" | "external_untrusted";
sensitivity: "public" | "internal" | "confidential";
createdAt: string;
validAt: string;
expiresAt?: string;
supersedes?: string[];
tokenEstimateByTokenizer: Record<string, number>;
priority: number;
citations?: CitationRef[];
};
```
Every item has a source and digest. A summary without links to its covered sources is invalid.
### 14.3 Authority order
Context is rendered in explicit authority classes:
1. platform system policy;
2. application/developer instructions and immutable run policy;
3. current authenticated user objective and approved inputs;
4. run plan, state, and accepted decisions;
5. project/repository data;
6. tool, child-agent, browser, and web results.
Lower classes are data. They cannot redefine higher-class policy, grant tools, approve actions, change budgets, or instruct the runtime to expose secrets.
### 14.4 Context snapshot
Each model operation receives an immutable snapshot:
```ts
type ContextSnapshot = {
contextSnapshotId: string;
runId: string;
taskId: string;
modelOperationId: string;
compilerVersion: string;
modelRouteVersion: string;
tokenizer: string;
maxContextTokens: number;
reservedOutputTokens: number;
reservedToolTokens: number;
orderedItems: Array<{
contextItemId: string;
contentSha256: string;
renderedTokens: number;
transform?: string;
}>;
omittedItemIds: string[];
renderedPromptSha256: string;
createdAt: string;
};
```
The manifest is stored in Neon. Large rendered content or compaction artifacts are stored in R2 according to retention and sensitivity policy. Langfuse receives the trace and permitted prompt/output content; redacted deployments may send hashes and metadata instead.
### 14.5 Budgeting and packing
The context compiler reserves capacity before selecting optional material:
```text
model context limit
- protocol/tool schema overhead
- non-droppable system and policy
- current user instruction
- minimum completion reserve
- tool-result reserve
= retrievable context budget
```
Packing priorities:
1. non-droppable policy and current objective;
2. unresolved constraints, approvals, and current run/task state;
3. recent conversation needed for coherence;
4. exact code/diff/test evidence for the active task;
5. relevant retrieved sources and memories;
6. historical summaries and optional background.
Budgets are dynamic rather than fixed percentages. The compiler records why an item was included, summarized, or omitted.
### 14.6 Compaction
Compaction is append-only and hierarchical:
```text
raw events/messages
-> segment summary
-> task/run summary
-> verified project memory candidate
```
A compaction artifact includes:
- source item IDs and digests;
- covered event/message sequence range;
- summary prompt/model/compiler versions;
- decisions, commitments, constraints, unresolved questions, errors, artifact refs, and citations;
- fidelity checks and any known information loss;
- creation time and supersession links.
Raw source is not overwritten by a summary. Retention may later remove raw content according to policy, but the deletion is explicit and the summary retains provenance metadata.
Compaction never summarizes secrets into a less protected class and never promotes untrusted instructions into trusted memory.
### 14.7 Repository context
- Repository files are addressed by repository identity, commit/base digest, path, and content digest.
- A context item becomes stale when its base digest no longer matches the active workspace.
- The compiler prefers symbols, diffs, test failures, and targeted file ranges over whole repositories.
- Generated/vendor/binary content is excluded by default and referenced as artifacts when required.
- Child patches include a base digest so the integration owner can detect drift.
### 14.8 Web and research context
Web sources retain URL, retrieval time, content digest, extraction method, and citation spans. Search snippets are leads, not final evidence. Prompt-injected webpage instructions remain `external_untrusted`.
Research performed in a broad-web environment receives no private repository context. Cross-boundary handoff consists only of sanitized questions and evidence bundles.
### 14.9 Memory
Memory has explicit scope and lifecycle:
```text
scope: user | organization | project | agent | run
state: candidate | verified | rejected | superseded | expired
kind: preference | fact | decision | procedure | episodic_summary
```
Models propose candidate memories with source references, confidence, sensitivity, and expiry. High-impact facts and preferences require deterministic confirmation or user approval before becoming verified. Contradictions create a superseding record rather than mutating history.
Neon owns memory records. V1 uses Neon metadata/full-text and, where appropriate, pgvector-backed semantic retrieval behind a `ContextIndex` interface. A future Cloudflare Vectorize index may be added only as a rebuildable derived index, never as the sole memory authority.
### 14.10 Caching
- Normalized source/extraction caches are content-addressed by source digest, parser version, and policy.
- Context snapshots are immutable and reusable only when every input digest, permission, and policy hash still matches.
- Model-response caching is limited to safe deterministic/read-only operations whose cache policy permits it.
- Tool-bearing, user-specific sensitive, effectful, approval, and current-state model calls are not cached merely because prompts match.
## 15. Usage Accounting and Budgets
### 15.1 Separate measurement, cost, and customer billing
The platform keeps three separate ledgers:
1. **Meter ledger** — what resources were consumed.
2. **Cost ledger** — what the platform estimates or is billed by a provider.
3. **Billing ledger** — what the product charges or credits the customer under product policy.
Langfuse and AI Gateway are observability and cost-evidence sources. Neither is the authoritative customer billing ledger.
### 15.2 Usage event
```ts
type UsageEvent = {
schemaVersion: 1;
usageEventId: string;
source: "ai_gateway" | "langfuse" | "execution" | "browser" | "tool" | "storage" | "broker" | "reconciliation";
sourceEventId: string;
tenantId: string;
projectId: string;
runId: string;
taskId?: string;
attemptId?: string;
operationId?: string;
provider?: string;
capability: string;
meter: string;
unit: string;
quantity: string; // decimal string, never binary floating point
quality: "estimated" | "reported" | "reconciled";
priceBookVersion: string;
estimatedCostMicros?: string;
occurredAt: string;
receivedAt: string;
metadata?: Record<string, string>;
};
```
`(source, sourceEventId, meter)` is unique. Corrections append adjustment/reversal events; they never rewrite consumed history.
### 15.3 Meter classes
At minimum:
| Capability | Meter examples |
|---|---|
| Models | input tokens, cached input tokens, output tokens, reasoning units, image/audio units, requests |
| Sandbox/E2B/Modal | wall time, CPU class time, memory class time, GPU time, session start, storage, network bytes |
| Browser | browser time, session/concurrency allocation, requests |
| Web tools | searches, pages, crawl credits, extraction units |
| R2 | stored bytes/time, class-A/class-B operations, egress where applicable |
| Broker | credentialed API calls, upstream billable units, egress |
| Control plane | optional Worker/Workflow units for internal cost analytics |
Provider retries are always recorded in the cost ledger. Whether they are customer-billable is determined separately by billing policy.
### 15.4 Price books
Price books are immutable and effective-dated. Values use integer micros of the billing currency or fixed-precision decimal units. A run pins its admission price-book version; a long run may use a new version only under an explicit product rule recorded as a billing event.
Provider-reported cost, negotiated cost, estimated cost, and retail charge are distinct fields. AI Gateway custom costs and Langfuse model definitions may assist estimation but do not replace the price-book record.
### 15.5 Reservation hierarchy
```text
Tenant budget
-> root run reservation in Neon
-> task allocation
-> operation reservation
-> committed actual usage + released remainder
```
Admission atomically reserves the root run envelope in Neon. The root Agent acts as the live per-run Budget Coordinator inside that reserved envelope:
- allocations are persisted in Agent SQLite;
- child agents and Workflows request task/operation allocations through typed RPC;
- allocation is strongly serialized per root run;
- no child may create or transfer budget;
- the sum of active allocations cannot exceed the Neon-reserved run remainder;
- periodic and terminal aggregates are committed to Neon;
- if the Budget Coordinator cannot verify remaining capacity, new paid operations stop.
This avoids a Neon write for every streamed token while preventing parallel children from oversubscribing the run envelope.
### 15.6 Operation lifecycle
1. Estimate worst-case or configured maximum usage.
2. Reserve from the task/run allocation before dispatch.
3. Include the reservation ID in the operation.
4. Stream coarse progress/usage checkpoints where available.
5. Stop further work when the hard limit is reached; do not rely on a post-hoc warning.
6. On terminal response, commit reported actual usage and release the unused amount.
7. If terminal usage is missing, commit a provisional estimate.
8. Reconcile later against AI Gateway, provider records, tool invoices, or infrastructure usage.
9. Append adjustments without mutating the original event.
Cancellation bills/records actual consumed resources according to product policy and releases all unused reservations.
### 15.7 Hard and soft limits
- **Hard limit:** blocks a new operation or stops a cooperative stream/process at the boundary.
- **Soft limit:** emits warning and may route to a cheaper eligible model/provider.
- **Run envelope:** total cap for the root run including children.
- **Task allocation:** maximum a child/task can consume.
- **Operation maximum:** bounded exposure for one model/tool/execution action.
AI Gateway spend limits are defense in depth. Because gateway/accounting limits may be eventually consistent or use estimated prices, the application reservation ledger remains authoritative for product budgets.
### 15.8 Reconciliation
Reconciliation compares internal usage with:
- AI Gateway request logs/analytics;
- Langfuse generation usage;
- model-provider statements where applicable;
- Cloudflare Sandbox/Browser/Workers billing exports;
- E2B, Modal, Parallel, Firecrawl, Neon, and other provider usage;
- R2 storage and operation records.
Differences are classified as timing, estimation, missing event, duplicate, price-version mismatch, provider correction, or unknown. Material unknown variance opens an operational incident and may temporarily tighten budgets.
## 16. Multi-Agent Semantics
### 16.1 Topology
A root run is a rooted acyclic task graph, not an unrestricted peer mesh.
- The root Agent owns the user objective, global plan, final response, event sequence, and run budget.
- A parent may create child tasks only within its delegation grant.
- Children return results to their parent; siblings do not exchange unaudited messages directly.
- Nested delegation is permitted only when explicitly granted and bounded by depth, task count, concurrency, and budget.
- A task cannot become its own ancestor. The Delegation Service checks the ancestor path transactionally.
### 16.2 Cloudflare primitive mapping
| Need | Primitive |
|---|---|
| Model-driven child with retained progress, replay, drill-in, cancellation | Agents as tools |
| Deterministic typed call to a known child where parent owns forwarding/replay | `subAgent()` / typed RPC |
| Long independent multi-step child process | Cloudflare Workflow registered in run graph |
| Child durable local loop | Managed Fiber inside that child Agent |
| Child heavy or untrusted computation | Separate execution session under child task policy |
Every child has isolated Agent SQLite. Framework child registries aid runtime management; Neon remains authoritative for the root task/delegation graph and billable/auditable facts.
### 16.3 Delegation contract
```ts
type DelegationRequest = {
schemaVersion: 1;
delegationId: string;
idempotencyKey: string;
rootRunId: string;
parentTaskId: string;
childTaskId: string;
parentAgentId: string;
childRole: string;
objective: string;
successCriteria: string[];
resultSchemaId: string;
contextSnapshotId: string;
artifactGrants: ArtifactGrant[];
toolGrants: ToolGrant[];
executionPolicyPatch: RestrictiveExecutionPolicyPatch;
budgetAllocationId: string;
failurePolicy: "fail_fast" | "collect_partial" | "best_effort" | "quorum";
priority: number;
deadlineAt: string;
maxChildDepth: number;
createdAt: string;
};
```
The effective child policy is the intersection of root policy, parent policy, child role policy, and the restrictive patch. Empty intersection rejects delegation. A child cannot inherit authority that was not explicitly included.
### 16.4 Child context
Children receive a `DelegationBrief`, not the parent's entire prompt or transcript by default. It contains:
- objective and success criteria;
- required output schema;
- relevant decisions and constraints;
- explicitly selected context/artifact references;
- trust and sensitivity labels;
- base repository/checkpoint digest;
- allowed tools and execution policy;
- task budget and deadline;
- parent contact/result channel.
This reduces token waste, prompt-injection spread, accidental data disclosure, and conflicting interpretations.
### 16.5 Delegation result
```ts
type DelegationResult = {
schemaVersion: 1;
delegationId: string;
childTaskId: string;
status: "succeeded" | "failed" | "cancelled" | "partial";
summary: string;
claims: Array<{
statement: string;
evidenceRefs: string[];
confidence: number;
}>;
artifactIds: string[];
proposedPatchArtifactId?: string;
baseWorkspaceDigest?: string;
testArtifactIds: string[];
sideEffectIds: string[];
unresolved: string[];
contextDeltaCandidates: string[];
usageSummaryId: string;
completedAt: string;
};
```
Child output is untrusted until the parent validates its schema, evidence, base digest, tests, policy compliance, and merge conditions. The child never publishes the root final response directly.
### 16.6 Concurrency and workspace isolation
No two agents concurrently mutate the same live filesystem.
- Each coding child restores/forks from an immutable workspace checkpoint or Git base.
- The child writes only its isolated workspace and returns a patch/content manifest.
- The parent designates one integration task as the sole merge owner.
- The merge owner verifies the child's base digest, applies changes serially, resolves or reports conflicts, and runs required tests.
- A child with a stale base is rebased/re-run or its patch is rejected; it never overwrites newer work.
- Modal tasks consume immutable inputs and cannot mutate the main workspace.
Research results merge as structured evidence bundles with citation deduplication and contradiction detection, not by concatenating prose.
### 16.7 Budgets and limits
Delegation requires a reserved child allocation. Parent and child limits include:
```text
maximum depth
maximum total child tasks
maximum concurrent children
maximum model/tool/execution cost
maximum wall time
maximum context size
maximum artifact/output bytes
allowed child roles
allowed delegation types
```
Unused child allocation returns to the parent. A child cannot borrow from siblings or the root remainder without a new authorized allocation.
### 16.8 Cancellation and failure propagation
- Root cancellation cascades to every descendant task, registered Workflow, Fiber, broker grant, execution, and process.
- Parent-task cancellation cascades only through that task's descendants.
- Child cancellation does not automatically cancel parent or siblings.
- `fail_fast` cancels remaining siblings after one required child fails.
- `collect_partial` waits for all children and returns valid partial results.
- `best_effort` treats child failure as a warning unless root success criteria fail.
- `quorum` completes after the declared threshold and cancels unnecessary remaining children.
Runtime abort and durable cancellation are distinct. The system first commits cancellation, revokes authority, and registers cleanup; then it aborts the child instance. Child storage is retained through reconciliation and audit retention. Deletion is a later garbage-collection action.
### 16.9 Approvals and user interaction
Children cannot directly approve themselves or obtain privileged input from the user. They submit a structured approval/input request to the parent. The root Agent deduplicates, presents, and records the decision. Any UI drill-in is observational unless routed through the canonical approval/input API.
### 16.10 Cleanup
Terminal child runs, child Agent facets, Fibers, Workflow records, stream replay, sandboxes, and checkpoints have independent retention policies. Cleanup verifies that:
- the task is terminal;
- usage is reconciled or explicitly pending;
- external effects are known;
- required artifacts/checkpoints are durable;
- no active descendant or Workflow remains;
- audit retention is satisfied.
Only then may a child Agent be deleted and its SQLite wiped.
## 17. End-to-End Flows
### 17.1 Interactive coding run
1. User submits through Hono with idempotency key.
2. Hono authorizes, compiles policy, reserves budget, and commits `accepted`.
3. Root Agent durably accepts a managed Fiber and commits `queued`.
4. Agent claims an epoch, commits `running`, and builds a context snapshot.
5. Model Router calls AI Gateway using the approved capability route.
6. Agent creates one Cloudflare Sandbox execution with repository profile.
7. Sandbox restores the latest native checkpoint or reconstructs the portable checkpoint.
8. Agent issues bounded argv commands; logs stream through the Execution Service and Agent.
9. Credential-free Git/package reads follow allowlists; credentialed repository actions use the broker.
10. Agent validates changes and tests, creates native and portable checkpoints, and stores output artifacts.
11. Usage commits, final assistant message and run result persist to Neon, then the run becomes `succeeded`.
### 17.2 Long Workflow with approval
1. Root Agent creates a uniquely identified Workflow and registers it in the run graph.
2. Workflow performs each mutation inside an idempotent durable step.
3. Before a privileged action, it creates a content-bound approval request and reports `waiting/approval`.
4. Root Agent presents the request; user decision enters through Hono.
5. Approval Service validates actor, digest, expiry, and policy, then sends the Workflow event.
6. Workflow resumes with a new short-lived broker grant, performs the action, and reports completion.
7. Agent sequences progress and terminal events; Neon receives final projection and effect/usage records.
### 17.3 Research combined with private-code work
1. Root Agent creates a repository child with private workspace and restricted egress.
2. It separately creates a research child with broad-web policy and no private repository access.
3. The research child returns a cited evidence artifact.
4. The parent sanitizes and explicitly grants that artifact to the repository child.
5. No environment ever contains private code plus unrestricted internet plus credentials.
### 17.4 Cancellation during execution
1. User cancellation commits `cancelling` and revokes grants/budget allocations.
2. Agent stops dispatch, cancels the Fiber, and cascades to children/Workflows.
3. Execution adapter sends graceful process termination, then force termination if required.
4. Broker denies any later request from the stale epoch.
5. Reconciler checks submitted side effects and provider handles.
6. Actual usage and safe partial artifacts are finalized.
7. Run commits `cancelled` with explicit partial-output/effect status.
### 17.5 Provider loss and fallback
1. Cloudflare Sandbox process/container becomes lost.
2. Attempt moves to `reconciling`; old process handles and cursors are never reused.
3. Side effects are inspected and the latest usable checkpoint is selected.
4. If same-provider restore is healthy, use the native checkpoint.
5. Otherwise, policy eligibility is recomputed for E2B.
6. E2B reconstructs from the portable checkpoint with the same or stricter policy.
7. A new attempt and coordinator epoch start; consumed platform cost from the lost attempt remains recorded.
## 18. Persistence Model
### 18.1 Neon tables
| Table | Core purpose / invariant |
|---|---|
| `runs` | One row per root run; current version/status/epoch/policy/budget/terminal result |
| `run_idempotency` | Unique tenant + operation + key; canonical request digest and stable receipt |
| `run_events` | Sparse append-only authoritative lifecycle sequence |
| `conversations` | Durable conversation/Agent identity and project/tenant ownership |
| `messages` | Canonical finalized user/assistant messages and content/artifact references |
| `run_tasks` | Rooted DAG nodes, owner Agent/Workflow, state, parent, depth, policy and budget refs |
| `task_attempts` | Immutable physical attempts and coordinator epochs |
| `operations` | Model/tool/execution/broker operations, replay class, reservation, state, result ref |
| `side_effects` | Prepared/submitted/confirmed/unknown externally visible mutations |
| `control_outbox` | Durable delivery intent and acknowledgement for Agent/Workflow/reconciler commands |
| `workflow_instances` | Run/task to unique Workflow ID and definition version mapping |
| `execution_sessions` | Logical/provider handles, generation, image, policy hash, state |
| `execution_processes` | Launch hash, provider process ID, generation, state, exit and log refs |
| `workspace_checkpoints` | Native/portable refs, digests, base, lineage, compatibility, retention |
| `artifacts` | R2 metadata, ownership, digest, provenance, retention and security class |
| `credential_grants` | Non-secret authority, scope, epoch, limits, expiry, revocation |
| `approvals` | Content-bound requests and immutable decisions |
| `context_items` | Provenance-bearing context/memory objects and content references |
| `context_snapshots` | Immutable per-model-call ordered context manifests |
| `memories` | Candidate/verified/superseded scoped memory records |
| `usage_events` | Deduplicated append-only raw meter events |
| `usage_reservations` | Tenant/run/task/operation reservations and releases |
| `cost_entries` | Estimated/reported/reconciled provider cost with adjustments |
| `billing_entries` | Product charges, credits, reversals, and billing policy version |
| `price_books` | Immutable effective-dated unit prices and currency precision |
| `delegations` | Parent-child contract, policy intersection, budget, result, lifecycle |
### 18.2 Agent SQLite
Application-managed local tables include:
```text
live_runs current local coordination snapshot
stream_events bounded resumable user-visible event window
source_event_dedupe child/workflow/provider event IDs already sequenced
budget_allocations live task and operation allocations within run envelope
active_handles workflow/execution/process handles used for fast recovery
local_checkpoints compact recovery pointers
```
Framework-owned Fiber, scheduling, message, state, and sub-agent tables remain framework-managed. Agent message storage is the hot/live view; finalized canonical messages are committed to Neon. Application migrations use a dedicated migration table; no critical state exists only in class properties.
### 18.3 R2 layout
```text
artifact bucket:
artifacts/sha256/{digest}
logs/{run_id}/{operation_id}/{segment}
context/sha256/{digest}
portable-checkpoints/sha256/{digest}
workspace backup bucket:
native/{tenant_id}/{project_id}/{checkpoint_id}/{provider_object}
```
Keys do not grant access. Every read is authorized using Neon ownership metadata and current policy.
## 19. Contract and Deployment Versioning
- Every envelope includes `schemaVersion`.
- Readers accept current and explicitly supported older versions.
- Writers emit one current version per deployment.
- Additive fields are optional until every reader supports them.
- In-flight Fibers and Workflows retain compatible recovery code for their maximum lifetime.
- A deployment that cannot recover an older snapshot must ship a deterministic migration or fail it safely as `INCOMPATIBLE_RUNTIME`; it cannot guess.
- Provider adapters isolate preview/stable SDK differences from domain contracts.
- Execution image, toolchain, policy, prompt, route, and price-book versions are immutable references.
- Database rollout uses expand/migrate/contract sequencing and preserves the rollback window.
## 20. Observability and User Visibility
### 20.1 Trace structure
One root Langfuse/OTel trace corresponds to the root run. Child tasks, model calls, tools, Workflow steps, executions, processes, broker calls, approvals, and artifact operations are child observations/spans with the common identifiers from the architecture.
Sensitive prompt/output logging follows data policy. Even when payload logging is disabled, hashes, route/model, token usage, timing, result classification, and correlation IDs remain.
### 20.2 Required operational metrics
```text
run admission and dispatch latency
state-transition conflicts
accepted runs without coordinator receipt
lease expiry and recovery success
Fiber interruption/recovery outcome
Workflow retry/wait/termination
sandbox provision, restore, checkpoint, and lost-session rates
native versus portable restore latency
broker allow/deny/revoke and side-effect reconciliation
context tokens by source class and compaction rate
model/tool/provider selection and fallback
budget reservation denial and reconciliation variance
child depth, fan-out, cancellation, and merge-conflict rates
terminal run outcome and cancellation latency
```
### 20.3 User-visible progress
Public events use a stable taxonomy and safe payloads:
```text
run.accepted
run.queued
run.started
run.waiting
run.paused
run.cancelling
run.completed
run.failed
run.cancelled
task.started
task.progress
task.completed
approval.required
model.progress
tool.started
tool.progress
tool.completed
execution.started
execution.output
artifact.available
warning
```
Internal provider errors, secrets, raw policy details, and unredacted logs are never emitted to the public stream.
## 21. Verification and Acceptance Gates
The runtime is not production-ready until these tests pass:
### 21.1 State and idempotency
- Concurrent identical create requests produce one run and one budget reservation.
- Conflicting payload under one idempotency key returns conflict.
- Duplicate Agent dispatch produces one managed root Fiber.
- Duplicate Workflow creation resolves one registered instance.
- Replayed lifecycle commands cannot skip the transition table or stale version.
- A stale coordinator epoch cannot commit a result or call the broker.
### 21.2 Recovery
- Agent eviction at every Fiber checkpoint resumes or safely classifies the operation.
- Workflow restart duplicates no side effect.
- Lost sandbox process never reuses an invalid cursor/handle.
- Native Cloudflare restore reproduces the checkpoint digest.
- Portable restore into E2B reproduces the defined source/patch/file manifest.
- Recovery from `submitted` broker state reconciles before retry.
### 21.3 Cancellation and approval
- Cancellation committed before success always prevents `succeeded`.
- Success committed before cancellation returns success unchanged.
- Cancellation revokes broker grants before any later credentialed request.
- Root cancellation reaches every registered descendant Workflow/execution.
- Aborted child storage survives until reconciliation.
- Approval cannot be reused after request digest, policy, base, actor, or expiry changes.
### 21.4 Security
- Environment, filesystem, process list, logs, and crash output reveal no upstream credential.
- Private repository profile cannot reach arbitrary internet destinations.
- Research profile cannot access private repository artifacts.
- Alternate providers are rejected when policy-equivalent egress cannot be proven.
- Redirect, DNS, IP-literal, proxy, tunnel, path-normalization, symlink, and oversized-payload attacks fail closed.
- Cross-tenant run, Agent, artifact, checkpoint, context, memory, grant, and delegation references are denied.
### 21.5 Context
- Every model call has a reproducible context manifest and prompt digest.
- No untrusted context item changes policy authority.
- Compaction preserves referenced decisions, unresolved obligations, artifacts, and citations under golden tests.
- Stale repository context is detected after workspace base changes.
- Children receive only explicitly granted context.
### 21.6 Usage
- Duplicate provider/Langfuse/AI Gateway events create one meter entry.
- Parallel child allocations cannot exceed the root reservation.
- Cancellation releases unused reservation and retains consumed cost.
- Missing terminal model usage becomes provisional and later reconciles by append-only adjustment.
- Cost ledger and customer billing ledger can differ without corrupting either.
### 21.7 Multi-agent
- Cycles, depth overflow, fan-out overflow, and unbudgeted delegation are rejected.
- Concurrent coding children cannot share a writable workspace.
- Only the integration owner changes the parent workspace.
- Stale-base child patches are rejected or explicitly rebased.
- Failure policies cancel or collect siblings exactly as declared.
- Child-created Workflows remain discoverable and cancellable through the root registry.
## 22. Implementation Sequence
Build in this order so later layers rest on stable contracts:
1. Shared IDs, Zod envelopes, error taxonomy, canonical hashing, and version helpers.
2. Neon run/task/attempt/event/idempotency/outbox schemas and `RunStateService`.
3. Admission, budget envelope reservation, Agent mapping, and durable `acceptRun()` Fiber.
4. Agent live event sequencer, WebSocket resume, and run reconciler.
5. Provider-neutral Execution Service and Cloudflare Sandbox adapter.
6. Deny-by-default profiles, outbound-handler broker, grant/side-effect ledger, and cancellation.
7. Artifact collection plus native and portable workspace checkpoints.
8. Context items, snapshot compiler, compaction, memory, and AI Gateway model adapters.
9. Usage meter/cost/billing ledgers, live Budget Coordinator, and reconciliation.
10. Workflow integration, approvals, pause/resume, and long external operations.
11. Multi-agent delegation graph, Agents-as-tools integration, isolated workspaces, and merge owner.
12. E2B policy-equivalent portable fallback and Modal artifact-compute adapter.
13. Fault injection, security conformance, load tests, evaluation gates, and recovery drills.
The first vertical slice should be one authenticated interactive coding run on Cloudflare Sandbox with restricted egress, a single model route through AI Gateway, resumable events, cancellation, one workspace checkpoint, and complete usage correlation. Add multi-agent and alternate providers only after that slice passes failure and security tests.
## 23. Final Execution Invariants
1. Neon is the sole authority for root lifecycle, idempotency, policy, approval, effect, and billing facts.
2. The Agent is the sole sequencer of the root live event stream and final response assembly.
3. A Fiber coordinates recoverable Agent-local I/O; it does not become a compute engine.
4. A Workflow owns its durable independent step graph; Neon stores only the business projection and mapping.
5. Every active coordinator is fenced by a monotonically increasing epoch.
6. Recovery inspects before replay and obeys an operation's declared replay class.
7. Every external mutation has a prepared, stable `side_effect_id`.
8. Delivery may repeat; state transitions deduplicate, and effects retry only under an idempotent or reconcilable contract.
9. Cancellation commits and revokes authority before runtime interruption.
10. No sandbox possesses an upstream credential.
11. Credentialed broker calls fail closed on stale epoch, cancellation, policy uncertainty, or control-data outage.
12. Provider fallback occurs only from a consistent portable checkpoint and never weakens policy.
13. Cloudflare native snapshots optimize primary recovery; portable checkpoints preserve provider fallback.
14. Every model call has an immutable, provenance-bearing context snapshot.
15. Context authority is explicit; web, repository, child, model, and tool content cannot grant privileges.
16. Meter, provider cost, and customer billing are separate append-only ledgers.
17. A run reserves its outer budget before work; children consume explicit sub-allocations.
18. Multi-agent execution is a bounded rooted DAG with one root response owner.
19. Concurrent coding agents use isolated workspaces and one serial integration owner.
20. Child Agents, Workflows, Fibers, broker grants, executions, processes, artifacts, and usage remain registered until terminal reconciliation.
## 24. Current Platform Reference Anchors
- [Cloudflare Agents: durable execution with Fibers](https://developers.cloudflare.com/agents/runtime/execution/durable-execution/)
- [Cloudflare Agents: long-running agents](https://developers.cloudflare.com/agents/concepts/agentic-patterns/long-running-agents/)
- [Cloudflare Agents: sub-agents](https://developers.cloudflare.com/agents/runtime/execution/sub-agents/)
- [Cloudflare Agents: Agents as tools](https://developers.cloudflare.com/agents/runtime/execution/agent-tools/)
- [Cloudflare Agents: WebSockets and hibernation](https://developers.cloudflare.com/agents/runtime/communication/websockets/)
- [Cloudflare Workflows: rules](https://developers.cloudflare.com/workflows/build/rules-of-workflows/)
- [Cloudflare Workflows: events and parameters](https://developers.cloudflare.com/workflows/build/events-and-parameters/)
- [Cloudflare Sandbox: process execution](https://developers.cloudflare.com/sandbox/1-0-preview/processes/)
- [Cloudflare Sandbox: command/process control](https://developers.cloudflare.com/sandbox/api/commands/)
- [Cloudflare Sandbox: outbound traffic](https://developers.cloudflare.com/sandbox/guides/outbound-traffic/)
- [Cloudflare Sandbox: backup and restore](https://developers.cloudflare.com/sandbox/guides/backup-restore/)
- [Cloudflare AI Gateway: REST API](https://developers.cloudflare.com/ai-gateway/usage/rest-api/)
- [Cloudflare AI Gateway: spend limits](https://developers.cloudflare.com/ai-gateway/features/spend-limits/)
- [Langfuse: token and cost tracking](https://langfuse.com/docs/observability/features/token-and-cost-tracking)
- [Langfuse: evaluation concepts](https://langfuse.com/docs/evaluation/core-concepts)
## 25. Final Run and Execution Diagram
```mermaid
flowchart TB
User[User / API client]
subgraph Control[Trusted Control Plane]
Hono[Hono canonical API]
State[RunStateService<br/>CAS transitions + outbox]
Root[Root Agent / Durable Object<br/>live sequencer + Budget Coordinator]
Fiber[Managed Agent Fiber<br/>I/O coordination + recovery]
Workflow[Cloudflare Workflow<br/>independent durable steps]
Policy[Policy + Approval Services]
ExecRouter[Execution Router]
ExecService[Execution Service<br/>provider-neutral protocol]
Broker[Credential / Egress Broker]
Reconciler[Run Reconciler Workflow]
end
subgraph Execution[Untrusted / Heavy Execution Plane]
CFS[Cloudflare Sandbox<br/>primary]
E2B[E2B<br/>portable-checkpoint fallback]
Modal[Modal<br/>artifact-based heavy compute]
Proc[Supervised process / Playwright CLI]
end
subgraph Data[Data Plane]
Neon[(Neon<br/>runs, tasks, effects, policy,<br/>usage and billing authority)]
SQLite[(Agent SQLite<br/>Fibers, live replay,<br/>handles and allocations)]
R2N[(R2 native workspace backups)]
R2P[(R2 portable checkpoints)]
R2A[(R2 content-addressed artifacts)]
WFState[(Workflow step state)]
end
subgraph Intelligence[Intelligence + Tooling Plane]
Context[Context Compiler<br/>provenance + budgets]
Gateway[Cloudflare AI Gateway<br/>versioned capability route]
Models[Workers AI + approved external models]
Tools[Tool Router<br/>Parallel / Firecrawl / Browser Run]
Langfuse[Langfuse<br/>traces + evaluations]
end
subgraph Children[Bounded Multi-Agent DAG]
ChildA[Child Agent task A<br/>isolated SQLite/workspace]
ChildB[Child Agent task B<br/>isolated SQLite/workspace]
Integrator[Integration task<br/>sole merge owner]
end
User -->|idempotent command| Hono
Hono --> Policy
Hono -->|admission transaction| State
State <--> Neon
State -->|durable dispatch intent| Root
Root <--> SQLite
Root --> Fiber
Root --> Workflow
Workflow <--> WFState
Reconciler <--> Neon
Reconciler --> Root
Fiber --> Context
Context --> Gateway
Gateway --> Models
Gateway --> Langfuse
Fiber --> Tools
Fiber --> ExecRouter
Workflow --> ExecRouter
ExecRouter --> ExecService
ExecService --> CFS
ExecService -.policy-equivalent fallback.-> E2B
ExecService -.heavy artifact job.-> Modal
CFS --> Proc
CFS <--> R2N
ExecService <--> R2P
ExecService <--> R2A
CFS -->|credential-free virtual request| Broker
E2B -.explicit Control Plane tool only when needed.-> Broker
Modal -.explicit Control Plane tool only when needed.-> Broker
Broker -->|verify epoch, grant, effect| Neon
Broker --> External[Approved external service]
Root --> ChildA
Root --> ChildB
ChildA -->|structured result + patch| Integrator
ChildB -->|structured result + evidence| Integrator
Integrator --> Root
Policy -.immutable policy hash.-> ExecService
State -.cancellation first revokes authority.-> Broker
State -.fenced cancellation.-> Root
Root -.cascade.-> Workflow
Root -.cascade.-> ChildA
Root -.cascade.-> ChildB
ExecService -.usage events.-> Neon
Gateway -.usage evidence.-> Neon
Langfuse -.quality feedback.-> Gateway
```