Runtime And Deployment

OpenEve separates authoring from execution:

agent folder -> compiler -> .openeve artifact -> runtime host -> durable runs

The compiler validates a folder and emits a deterministic manifest. The runtime loads the compiled artifact, executes runs, records events, stores context and artifacts, manages approvals and human input pauses, and delivers final responses idempotently.

CLI Commands

Inside this repo, build first:

pnpm build

Then use the local CLI:

pnpm openeve <command> <agent-root>

Commands:

Command	Purpose
init	Scaffold a minimal agent folder.
add	Install an adapter or channel provider package and wire it into gateway.ts, channels/, or agent.ts (--role <role> disambiguates multi-role kinds like docker; --no-install prints the install command instead of running it; --pm overrides the lockfile-detected package manager).
validate	Check required files, exports, schemas, routes, schedules, connections, subagents, skills, and gateway bindings.
manifest	Print or write the compiled manifest.
build	Emit the .openeve/ runtime artifact.
dev	Validate, build, and run a local dev check. With --watch, serve locally and rebuild + restart on the same port when the agent changes.
run	Execute one local run from a message and optional tool/input.
serve	Start the compiled Node runtime locally.
channels wire	Print or apply channel provider ingress URLs after deploy; Telegram webhooks are set by API when credentials are present.
models	List provider/model specs from the local model catalog (--provider <name> filters).
deploy --dry-run	Build a deployment plan and report missing setup without publishing.
deploy	Publish or prepare the target declared by gateway.ts or --target.
help [command]	Show usage for all commands or one command (--help also works per command).

Common flags:

Flag	Purpose
--root <path>	Agent root override.
--out <dir>	Build artifact directory.
--message <text>	Message for dev or run.
--tool <name>	Force a local tool call.
--input <json>	Tool input JSON.
--approve	Allow or resume approval-gated tool execution.
--dry-run	Print deploy plan without publishing.
--target <name>	Override gateway deploy target.
--once	Boot-check long-lived commands once, then exit.
--port <number>	Port for serve, dev --watch, or local deploy.
--watch	Keep dev serving and rebuild + restart on agent changes.
--no-model-check	Skip the validate model catalog check.
--migration-command <bin>	Run artifact migrations before hosted deploy.

Run openeve help <command> for the full per-command flag list.

Build Artifact

openeve build emits:

.openeve/
  manifest.json
  agent-revision.json
  Dockerfile
  package.json
  preflight.json
  route-table.json
  schedules.json
  server/
    boot.json
    boot.js
    sources.json
    source-metadata.json
    source-map.json
  assets/
  migrations/

Important files:

• manifest.json - complete compiled agent contract.
• agent-revision.json - deterministic source/config revision.
• preflight.json - required env and provider setup.
• route-table.json - HTTP channel routes.
• schedules.json - static schedule registration metadata, including trigger handler references.
• server/boot.js - production Node runtime boot entrypoint.
• deployment.json - created by openeve deploy after a local or provider publish/prepare operation, not by plain build.

The artifact is generated output and should not be committed.

Build artifacts support two package modes:

• Local mode is the default inside this repo. It writes file:./vendor dependencies for OpenEve workspace packages, copies those packages once under vendor/, links them into node_modules, and copies the runtime dependency closure needed for no-install local artifact smoke tests.
• Release mode is for published package deployments. Pass packageMode: "release" to buildAgent() or set OPENEVE_ARTIFACT_PACKAGE_MODE=release; the artifact package.json references published @openeve/* package versions and omits local vendor/ and node_modules/ copies. The generated Dockerfile then installs dependencies through normal package-manager semantics.

Runtime Lifecycle

For each run, the runtime:

1. Persists the run before execution starts.
2. Builds and stores a context bundle.
3. Starts the model loop or forced tool call.
4. Records model events, tool requests, approvals, tool results, sandbox events, usage, and checkpoints.
5. Creates delivery obligations for final responses.
6. Sends deliveries idempotently through the channel adapter.
7. Marks the run completed, failed, waiting for input, or waiting for approval.

Failed final deliveries are durable. When a channel send keeps failing retryably, the obligation is deferred onto a durable delivery queue (delivery.deferred, status pending with backoff) instead of going terminal, and a delivery worker drains it later — including after a crash or in another replica (Postgres leases use for update skip locked). Exhausted or non-retryable deliveries end as delivery.failed with failedAt.

Recovery is staleness-guarded and conservative. Executing runs heartbeat updatedAt (default 30s, OPENEVE_RUN_HEARTBEAT_MS); only runs stuck in created/running past max(5min, 4x heartbeat) are swept, and each candidate is claimed through an idempotency key so concurrent replicas never double-recover. The sweep completes already-delivered runs without re-sending, cancels tool calls before side effects start, attempts one in-place resume when a harness continuation checkpoint exists, enqueues a real pending delivery for runs that reached a model response but not delivery, and marks everything else failed (with run.failed) instead of pretending it completed. listenNodeRuntime runs recoverIncompleteRuns() once at boot and startBackgroundWorkers() keeps the delivery worker, sandbox-sync worker, and periodic sweep running until the server closes.

Node Runtime HTTP API

The Node host exposes:

Endpoint	Purpose
GET /health	Liveness plus agent revision.
GET /healthz	Liveness plus agent revision.
GET /readyz	Readiness: 200 normally, 503 while the runtime is draining during graceful shutdown.
GET /manifest	Compiled manifest.
GET /routes	Compiled route table.
POST /runs	Start a local/API run. Pass "stream": true to receive the whole run as server-sent events (lifecycle events, token deltas, then a final run_result).
GET /runs	Query run summaries.
GET /runs/:id	Inspect one run timeline.
GET /runs/:id/events	Inspect raw run events.
GET /runs/:id/timeline	Inspect grouped timeline.
GET /runs/:id/stream	Attach to a run's live SSE stream: replays the durable event log (SSE id: is the event sequence, so Last-Event-ID reconnects resume where they left off), then tails live events including ephemeral model.response_delta tokens, and ends with stream_end after a terminal event. Works for runs started by any channel, schedule, or client.
POST /runs/:id/approve	Resume a run paused on tool approval (waiting_for_approval): runs the gated tool and re-enters the harness. Returns the post-resume run summary. 404 if the run is unknown, 409 if it is not waiting for approval or a resume is already in flight.
POST /runs/:id/answer	Resume a run paused on ask_question (waiting_for_input) with {"answer": "..."}: splices the answer as the tool result and continues. 400 without an answer, 404/409 as above.
GET/POST /openeve/scheduler/tick	Trigger due static and dynamic schedules from a gateway or cloud scheduler.
GET/POST /openeve/connections/callback	Complete connection authorization callbacks.

Compiled channel routes, such as /slack/events or /message, are also mounted from the manifest route table. In production, first-party provider helper routes remain public so the provider can call them, but their normalizers must verify signatures or shared secrets before accepting a turn. Generic defineChannel() HTTP routes require a host auth policy or Authorization: Bearer <OPENEVE_ADMIN_TOKEN>; the raw message fallback is dev-only unless the host has authenticated the request.

In dev mode, local inspection and API-run endpoints are open for quick iteration. In production, the Node host fails boot unless control-plane auth is configured with a host-provided auth policy or OPENEVE_ADMIN_TOKEN. /manifest, /routes, /runs, /runs/:id, /runs/:id/events, /runs/:id/timeline, and /runs/:id/stream require Authorization: Bearer <OPENEVE_ADMIN_TOKEN> when the built-in token policy is used. POST /runs is disabled in production by default; set OPENEVE_ENABLE_API_RUNS=true only for deployments that intentionally expose authenticated API-triggered runs. The HITL resume endpoints (POST /runs/:id/approve and /runs/:id/answer) use the same admin auth but are not gated by OPENEVE_ENABLE_API_RUNS — a paused run must be resolvable in production even when API-triggered run creation is off. They are safe to replay: each resume claims the run's per-pause idempotency key, so a double-submit executes the gated tool at most once, even across replicas.

Request bodies are capped before parsing. The default limit is 10 MiB; set OPENEVE_HTTP_MAX_BODY_BYTES or NodeRuntimeServerOptions.maxRequestBodyBytes only when a deployment intentionally accepts larger webhook payloads.

Provider channel routes remain public HTTP routes because providers such as Slack, Telegram, Teams, and Discord interactions call them directly. Those routes must rely on their channel adapter's signature or token verification before accepting a turn. Long-lived provider ingress such as Discord Gateway is started by the Node host through startIngress() and feeds the same idempotent accepted-turn path as HTTP channels.

/openeve/scheduler/tick accepts optional now and lookbackMs values in the query string or JSON body. In production, set OPENEVE_SCHEDULER_SECRET and send it as Authorization: Bearer <secret> or x-openeve-scheduler-secret. Without a configured secret, the endpoint only accepts dev-mode runtime requests.

Scheduler startup comes from gateway.scheduler:

• adapter("local") starts an in-process loop with the Node host.
• adapter("gateway") registers schedules but relies on the endpoint or host code calling runDueSchedules().
• adapter("postgres") starts the polling loop and coordinates duplicate workers through Postgres-backed state/idempotency. Pair it with state: adapter("postgres").

State And Blob Storage

Local development uses file-backed state and local blob storage. Production state should use Postgres:

import { defineGateway } from "@openeve/core";
import { neonPostgres } from "@openeve/postgres";
import { r2Blob } from "@openeve/s3";

export default defineGateway({
  state: neonPostgres(),
  blob: r2Blob()
});

Postgres stores runs, events, messages, conversations, tool calls, approvals, delivery obligations, schedules, schedule run lifecycle status, memory indexes, file catalog records, sandbox leases, usage records, run queues, idempotency keys, learned skills, dynamic schedules including trigger metadata, and dynamic connections.

Connection grants and OAuth authorization sessions use the durable state adapter when it implements those stores. The Postgres adapter does, so Postgres-backed production deployments do not need a separate file encryption secret for connection credentials.

When a production Node deployment uses file-backed connection credential stores, set OPENEVE_CONNECTION_STORE_SECRET or OPENEVE_SECRET to a stable secret of at least 32 characters. The built-in local development fallback is accepted only with devMode: true; production boot rejects missing, short, or known development secrets.

Blob storage stores context bundles, attachments, extracted text, generated artifacts, and sandbox sync bundles. Blob records are private by default. S3/R2 *_PUBLIC_BASE_URL is used only when code explicitly writes a blob with visibility: "public"; context bundles, memory files, and inbound attachments should remain private.

Sandbox Sync

Sandbox-backed file tools write through provider workspaces, but durable production persistence remains OpenEve state and blob sync. When a run finishes with dirty sandbox files and async sync is enabled, the runtime retains the dirty sandbox instead of deleting it, queues a sync job, and lets final delivery complete. The sync worker first calls adapter connect() and then wake() so paused, stopped, detached, or otherwise retained warm sessions can still be synced. After a successful sync the session is marked clean and disposed through the adapter's clean lifecycle path.

Hosted sandbox adapters do not use provider snapshots as the normal persistence path. Snapshots are created only when the configured sandbox snapshot policy requests them and the provider SDK exposes snapshot creation.

Durability Workers

listenNodeRuntime runs recoverIncompleteRuns() once at boot, then calls runtime.startBackgroundWorkers() to keep the delivery worker, the sandbox-sync worker, and the periodic orphan sweep running; all three stop when the server closes. Environment knobs:

Variable	Default	Purpose
OPENEVE_DELIVERY_WORKER	true	Kill-switch for the delivery queue worker.
OPENEVE_SANDBOX_SYNC_WORKER	true	Kill-switch for the sandbox-sync worker.
OPENEVE_RUN_RECOVERY	true	Kill-switch for boot recovery and the periodic orphan sweep.
OPENEVE_DELIVERY_QUEUE_LEASE_MS	60000	Lease duration for a sending delivery before it is requeued.
OPENEVE_DELIVERY_QUEUE_BATCH_SIZE	10	Deliveries leased per worker tick.
OPENEVE_DELIVERY_QUEUE_MAX_ATTEMPTS	5	Total send attempts before a delivery goes terminally failed.
OPENEVE_DELIVERY_QUEUE_INTERVAL_MS	15000	Delivery worker tick interval.
OPENEVE_DELIVERY_RETRY_ATTEMPTS	2	In-process send attempts before deferring to the queue.
OPENEVE_RUN_HEARTBEAT_MS	30000	How often executing runs bump updatedAt to stay out of the orphan sweep.
OPENEVE_RUN_RECOVERY_INTERVAL_MS	60000	Orphan sweep interval. Runs are only candidates after max(5min, 4x heartbeat) of staleness.

Concurrency And Rate Limiting

Brand-new runs pass through a bounded semaphore before any state is written. When the limit is reached the runtime rejects the work with RunCapacityError before consuming the event's idempotency key, so provider retries and webhook redeliveries succeed once capacity frees up. The Node host maps capacity rejections to HTTP 429 with a Retry-After header (whole seconds) on POST /runs and provider channel routes. Resumes (approvals, human input, connection callbacks, orphan-recovery continuations, and scheduler continuations of an existing run) never queue behind the limit — queueing a resume behind the run it unblocks would deadlock — but still count toward the drain performed by graceful shutdown.

Ingress rate limiting is a token bucket applied after auth on provider-channel, run-create, and scheduler routes (health and admin routes are exempt). It is off by default and enabled either through NodeRuntimeServerOptions.rateLimit (see the customization guide) or env:

Variable	Default	Purpose
OPENEVE_MAX_CONCURRENT_RUNS	unlimited (16 via createProductionRuntimeOptions outside dev)	Max simultaneously executing brand-new runs.
OPENEVE_MAX_QUEUED_RUNS	0	Brand-new runs allowed to wait for a slot before being rejected with 429.
OPENEVE_INGRESS_RATE_LIMIT	off	Provider-channel token bucket as capacity/refillPerSecond (e.g. 60/10).
OPENEVE_RUNS_RATE_LIMIT	off	POST /runs token bucket in the same capacity/refillPerSecond form.

Rate-limited requests receive 429 { "error": "Rate limited." } with a Retry-After header.

Graceful Shutdown

listenNodeRuntime returns a NodeRuntimeHandle with an idempotent shutdown({ timeoutMs? }) and a closed promise. The shutdown sequence: mark draining (/readyz starts answering 503 so load balancers stop routing new traffic; /health//healthz stay 200 for liveness) -> close the HTTP listener and stop channel ingress -> stop the scheduler -> stop background workers -> wait for in-flight runs up to the timeout -> flush the telemetry sink -> close the state adapter (Postgres ends its pool when it created it). Runs still executing at the timeout are abandoned safely: orphan recovery repairs them on the next boot.

openeve serve and openeve deploy --serve install SIGTERM/SIGINT handlers (installSignalHandlers from @openeve/node): the first signal drains gracefully and exits 0; a second signal exits 1 immediately.

Variable	Default	Purpose
OPENEVE_SHUTDOWN_TIMEOUT_MS	30000	Max wait for in-flight runs during graceful shutdown.

Preflight

Use dry-run deploys before publishing:

pnpm openeve deploy examples/minimal-agent/agent --dry-run

Preflight requirements are inferred from:

• gateway.ts adapters.
• Static connection files.
• Static channel files.
• The model provider prefix in agent.ts.

Model provider env keys:

Prefix	Env
anthropic/	ANTHROPIC_API_KEY
cerebras/	CEREBRAS_API_KEY
deepseek/	DEEPSEEK_API_KEY
fireworks/	FIREWORKS_API_KEY
google/	GOOGLE_API_KEY
groq/	GROQ_API_KEY
mistral/	MISTRAL_API_KEY
openai/	OPENAI_API_KEY
openrouter/	OPENROUTER_API_KEY
together/	TOGETHER_API_KEY
xai/	XAI_API_KEY

LiveKit voice-call tools and subagent harnesses use LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET. Outbound phone-call defaults can also use LIVEKIT_OUTBOUND_TRUNK_ID and LIVEKIT_VOICE_AGENT_NAME. Subagent harness and defineLiveKitConnection() usage contributes the required LiveKit env to preflight.

Provider setup details are in Adapters.

Deploy Targets

Local

Use local deploy for developer machines or long-lived VMs:

pnpm openeve deploy agent --target local --serve --port 3000

Railway

adapter("railway") or railwayDeploy() publishes the built artifact through the @openeve/railway deploy publisher and Railway CLI.

Required:

• RAILWAY_TOKEN or authenticated Railway CLI
• Linked project/service, or --railway-project and --railway-service

pnpm openeve deploy agent \
  --target railway \
  --railway-project prj_x \
  --railway-service svc_y

Syncing secrets to the target

By default deploy sets nothing on the remote service — you configure variables in the provider dashboard. Pass --sync-secrets to push your local secrets as part of the deploy: the CLI reads the project .env (or --secrets-from <path>) and calls the publisher's syncSecrets before publishing, so the first deploy boots with them. Only key names are logged, never values; empty keys are skipped. Supported on railway (railway variables --set) and fly (flyctl secrets set); targets without a remote service (docker, local) report that sync is unsupported and leave secrets to you.

pnpm openeve deploy agent --target railway --sync-secrets

The deploy receipt (written to .openeve/deployment.json) separates deploymentUrl — the reachable service URL when the provider CLI reports one, otherwise null — from dashboardUrl, the provider's management console, so the two are never conflated.

Docker Preview

adapter("docker") builds the compiled artifact as a Docker image through the @openeve/docker deploy publisher and can run it locally. This deploy path is preview until live smoke coverage exists.

pnpm openeve deploy agent \
  --target docker \
  --docker-image openeve/my-agent \
  --serve \
  --port 3000

Fly Preview

adapter("fly") writes a minimal fly.toml and deploys the artifact with flyctl through the @openeve/fly deploy publisher. This deploy path is preview until live smoke coverage exists.

Required:

• FLY_API_TOKEN
• --fly-app or FLY_APP_NAME

pnpm openeve deploy agent \
  --target fly \
  --fly-app my-agent \
  --fly-region iad

Migrations

If the build artifact contains migration files under .openeve/migrations, hosted deploys require a migration runner:

pnpm openeve deploy agent \
  --migration-command ./scripts/run-openeve-migrations

The migration process receives:

• OPENEVE_ARTIFACT_ROOT
• OPENEVE_AGENT_REVISION
• OPENEVE_DEPLOY_ENV
• OPENEVE_MIGRATION_FILES

The Postgres adapter records schema migrations with id, checksum, description, package version, and applied time.

Production Checklist

• Use a stable agent.id for agents with learned skills or durable state.
• Use Postgres for hosted durable state.
• Set OPENEVE_CONNECTION_STORE_SECRET or OPENEVE_SECRET if production uses file-backed connection credential stores instead of Postgres-backed stores.
• Use S3-compatible blob storage or R2 for attachments and artifacts.
• Use Docker or a hosted sandbox for untrusted code and shell work. The production Node helper refuses the local sandbox unless OPENEVE_ALLOW_LOCAL_SANDBOX_IN_PRODUCTION=true.
• Use deploy --dry-run and resolve all required preflight items.
• Keep model provider, channel, database, blob, sandbox, and deploy credentials out of the agent folder.
• Set OPENEVE_ADMIN_TOKEN or provide a host auth policy before production boot.
• Set OPENEVE_ENABLE_API_RUNS=true only when authenticated API-created runs are intended.
• Set OPENEVE_BASH_TOOL_MODE=approval or disabled for agents that should not get direct shell access. The default is enabled in every runtime mode.
• Set TELEGRAM_WEBHOOK_SECRET for Telegram channels and either PHOTON_WEBHOOK_SIGNING_SECRET/PHOTON_SIGNING_SECRET or PHOTON_INGRESS_TOKEN/PHOTON_WEBHOOK_BEARER_TOKEN for Photon channels before production boot.
• Bound run concurrency (OPENEVE_MAX_CONCURRENT_RUNS; createProductionRuntimeOptions defaults to 16 outside dev) and enable ingress rate limiting (OPENEVE_INGRESS_RATE_LIMIT, OPENEVE_RUNS_RATE_LIMIT) on internet-facing hosts.
• Point load-balancer readiness at GET /readyz (drains to 503 during shutdown) and liveness at /health; deliver SIGTERM for deploys so in-flight runs drain within OPENEVE_SHUTDOWN_TIMEOUT_MS.
• Verify /health, /readyz, authenticated /manifest, channel routes, and authenticated /runs after deploy.
• Make side-effect tools idempotent and approval-gated where appropriate.