Autoresearch gives AI agents GPU nodes and batch GPU jobs on demand, over MCP. Your agent gets a container on an H100 with a persistent encrypted disk. You pay per minute. When you stop a node, billing stops and the files stay on the disk. The next command wakes the node.
Add the MCP server to your agent. This example uses Claude Code. Other clients are below.
claude mcp add --transport http givemeanode https://mcp.autoresearch.sfcompute.comYour client opens a browser window one time to sign you in.
A new account starts with $0 credit, and a node needs credit to start. Ask your agent to add some:
autor billing topup 25 # with no card on file, it prints a payment linkbuy_credits(amount_usd: 25)
→ {charged: false, payment_url: "https://checkout.stripe.com/...", note: "..."}With no card on file, buy_credits charges nothing and answers with a payment link. Open the link and pay by card. That payment adds the credit and saves your card. After that, buy_credits charges the card directly and automatic top-up works. You can also pay on /billing with Pay on a card page. See Credit and automatic top-up.
Then ask your agent for a GPU:
Get me a GPU node, clone github.com/you/yourrepo on it, and run the benchmark.
Every example in these docs shows one action in three ways. The English tab is what you say to your agent. The CLI tab is the autor command. The MCP tab is the tool call your agent sends. The quick start again, in all three ways:
autor node create --chip h100 --wait # polls to running; prints the locked rate
autor run tulip -- nvidia-smi
autor node stop tulip # parked: files intact, billing stoppedcreate_node(chip: "h100") → {name: "tulip", status: "provisioning", rate: "$/min"}
run_command("tulip", "nvidia-smi") → {exit_code: 0, stdout: "..."}
stop_node("tulip") → parked: files intact, billing stoppedBilling starts when the node is ready. Billing stops when you stop the node. The rate is in the create_node response. The next run_command on a stopped node wakes it. The disk is exactly as you left it.
Every client uses the same server. It is streamable HTTP with OAuth 2.1 at this address:
https://mcp.autoresearch.sfcompute.comYour client finds the sign-in flow by itself. A person approves it one time in the browser. There are no API keys.
On a remote or cloud machine, the browser approval does not work. Start at Remote machines instead.
claude mcp add --transport http givemeanode https://mcp.autoresearch.sfcompute.comApprove the connection when Claude first uses it. /mcp shows the status.
On a machine that is rebuilt for every task, a code approval on every rebuild is not practical. Make a service token and pass it as a header. autor mcp install writes this form for you when GMN_TOKEN is set.
claude mcp add --transport http givemeanode https://mcp.autoresearch.sfcompute.com \
--header "Authorization: Bearer $GMN_TOKEN"The token is its own identity. Your cloud agent stays inside the workspaces you give the token. When you revoke the token, your own login is not affected.
Go to Settings, Connectors, Add custom connector. Enter the remote MCP server URL:
https://mcp.autoresearch.sfcompute.comcodex mcp add givemeanode --url https://mcp.autoresearch.sfcompute.comOr add this to ~/.codex/config.toml:
[mcp_servers.givemeanode]
url = "https://mcp.autoresearch.sfcompute.com"If Codex does not ask you to sign in on first use, run codex mcp login givemeanode. Give Codex the host only, with no path after it. Codex reports a 404 from any other path.
If Codex says the client timed out after 30 seconds and names startup_timeout_sec, Codex stopped waiting before the connection was complete. Thirty seconds is its default. A first connection can need more. Raise the budget and start Codex again.
# ~/.codex/config.toml
[mcp_servers.givemeanode]
url = "https://mcp.autoresearch.sfcompute.com"
startup_timeout_sec = 60If it still times out, sign out and sign in again. Run codex mcp logout givemeanode, then codex mcp login givemeanode. This replaces the saved login. If the timeout returns, use a service token instead. A token does not use a saved login.
On a machine with no browser, autor mcp install configures Codex for you through codex mcp add. If codex is not on your PATH, it prints this block with the correct path for your machine:
# ~/.codex/config.toml
[mcp_servers.givemeanode]
command = "/home/you/.local/bin/autor"
args = ["mcp", "serve"]If the browser approval cannot complete, for example in CI, skip OAuth. Make a service token and give it to Codex as a bearer token from the environment:
# ~/.codex/config.toml
[mcp_servers.givemeanode]
url = "https://mcp.autoresearch.sfcompute.com"
bearer_token_env_var = "GMN_TOKEN"Set GMN_TOKEN to the gmnt_ value in the environment where Codex runs. No browser opens. When you revoke the token, your own login is not affected.
Add this to opencode.json (project or global). On a remote machine, autor mcp install writes the local-command form for you instead.
{
"mcp": {
"givemeanode": {
"type": "remote",
"url": "https://mcp.autoresearch.sfcompute.com",
"enabled": true
}
}
}On a dev box, a Codespace, or any machine you reach over SSH, the browser approval redirects to localhost. That is a different machine from the one your agent runs on, so the approval never arrives. Install the CLI on that machine and run one command:
curl -fsSL https://autoresearch.sfcompute.com/cli.sh | bash
autor mcp installYou sign in one time with a code that you approve on any other device. The command finds every coding agent on the machine and points each one at autor mcp serve. That is a local command that reaches Autoresearch for the agent. No browser opens on either machine.
Claude Code, Codex, Cursor, opencode, Amp, oh my pi, and Zed are supported. If the install cannot write a config file safely, it prints the block for you to paste. Restart each agent after the install, or run /mcp in it.
--client cursor configures one agent instead of all of them.--print shows the config without writing anything.autor is the same product from a terminal, for people and scripts. Every CLI tab in these docs is an autor command. The install also creates gman, the earlier name of the CLI, as a link to the same binary. A script that runs gman works without a change.
curl -fsSL https://autoresearch.sfcompute.com/cli.sh | bash
autor login
autor node create --name tulip --chip h100 --wait
autor run tulip -- nvidia-smi
autor node stop tulipA command reads autor <noun> <verb> [--flags] [arguments]. -- ends the flags. Everything after it is the remote command. Most nouns take an ls verb that lists and a get verb that reads one thing.
autor <command> --help prints that command's syntax. autor reference prints every command at once.autor login opens the browser sign-in. With no browser, it prints a code to approve on another device.autor mcp install connects the agents on this machine to Autoresearch through that login.autor docs prints this manual.autor api <method> <path> calls any API endpoint directly, including endpoints with no subcommand yet.A node is a container on an H100 with a persistent disk. Every node type has a name. Pass that name as chip. h100-1 is one GPU. h100-8 is a full 8-GPU machine. The original spellings h100 and 8xh100 also work. The examples use the original spellings, and the API returns them.
autor node create --name tulip --chip h100create_node(name: "tulip", chip: "h100")create_node never blocks. It returns one of two statuses:
provisioning: capacity was free. The response carries the locked rate. rate.node_per_min is the cost of the whole node per minute. rate.per_min is the per-GPU rate. Poll get_node("tulip") until the status is running.queued: the type is full. The response carries a live queue position and a wait estimate. Your node is provisioned when a slot frees. Set a boot command now, so work starts the moment the node is ready (see On wake). Poll get_node. Do not create the node again. See Queue.list_nodes() (CLI: autor node ls) shows every node you have. get_node(name) (CLI: autor node get) shows one node with its state, rate, and queue position.
Two create-time flags are for benchmark work. clock_lock: true pins the GPU clock, so timings compare across runs. profiling: true unlocks the GPU profiler counters that ncu needs. Both are free. Both cannot change after create. See Profiling.
autor node stop tulip # stop paying now; disk parks, files intact
autor run tulip -- ls # wakes a stopped node automatically
autor node rm tulip # destroy the disk (crypto-erase, irreversible);
# stopped nodes only, and confirms unless -ystop_node("tulip") # stop paying now; disk parks, files intact
run_command("tulip", "ls") # wakes a stopped node automatically
delete_node("tulip") # destroy the disk (crypto-erase, irreversible);
# stopped nodes only, so stop_node firststop_node stops billing now. The disk parks with the files intact. Processes do not survive a stop.stopping until then, and stopped after. You do not pay for the stopping time.run_command on a stopped node wakes it. The disk is exactly as you left it. A run_command on a stopping node wakes it too, without waiting.stopped (no restorable disk copy) cannot start. A run_command on it fails and says why. The disk values below explain this state.delete_node destroys the disk. This cannot be undone. The node must be stopped first.get_node and list_nodes show the same date as deletes_at.get_node, is not a use.get_node reports where the disk is in the disk field. The field has four values.
intact. The disk is on the machine that runs the node, or on the machine where it is parked. A wake uses that disk. A node that has not started yet also reads intact, unless you create it from a snapshot. That node reads snapshotted until it starts.snapshotted. The disk is a copy in durable storage. The next wake restores the copy onto a machine, so that wake takes longer. A queued wake of such a node also reads snapshotted.unrestorable. The node is stopped, and we have no copy of its disk that we can restore. The status reads stopped (no restorable disk copy), and we refuse every wake. The note in the stopped section says why. Our operators see the node, and sometimes they can make the copy usable again. If you do not need the disk, delete the node and create a new one.lost. The disk went away with its machine. Run delete_node and then create_node.get_node reports what the host sees your node doing, under utilization. busy is our verdict. While it is true, we do not idle-stop the node, and the deadline in idle_stops_at moves forward.
The numbers behind the verdict are in the same section. On a GPU node, gpu_pct_max is the busiest GPU over the last ten minutes, and gpu_pct_latest is the newest minute. On any node, cpu_pct is percent of one core, so 300 means three cores are busy. Samples are taken once a minute. sampled_minutes says how many minutes of the window carry a GPU sample. If it is lower than window_minutes, the rest of the window is unknown, not idle. A missing field was never sampled. Treat a missing field or a missing section as no measurement, not as zero. The full series per GPU is the gmn_gpu_utilization metric.
gpu_memory says whether another workload will fit. It lists every GPU on the node with used_mib and free_mib, from the newest heartbeat (the as_of field). free_mib_min is the tightest card. Size a second workload against free_mib_min, not against a node-wide total. Two trainers on one node run out of memory on one device. The UUIDs match nvidia-smi -L inside the node. A card with no numbers was not sampled, which means unknown, not zero. The whole block is absent when the newest reading is too old to publish, so a missing block never means free memory. The full series per GPU is the gmn_gpu_memory_used_bytes metric.
A wait has two clocks. Know both, and polling is cheap.
expires_at, whether or not anyone polls. Pass max_wait on a re-call of create_node to extend the hold, forward only, up to 12 hours. Polling never extends the hold. max_wait is also a parameter on run_command, so a command that wakes a stopped node can claim its hold in the same call.rate.minimum_minutes. A ready node with no session reports running (idle) and idle_stops_at. If no run_command arrives before then, the node stops with the files intact. The window bills while it runs. There is no minimum charge. Stop the node, and billing ends at the stop. See rates.A node that is doing work is not idle. If the node uses its GPUs or its CPU, the window moves forward for as long as that lasts. A trainer you started with nohup or in tmux keeps running after the command that launched it returns. busy in get_node is the verdict we act on. A process that is alive but does nothing reads as idle. Use hold_node to keep a node like that awake.
If a node stops, get_node says what happened. The stopped section carries reason (a token such as idle_grace_expired or wake_hold_lapsed), at, a note with the next move, and ran_seconds, the billed length of that session. Next to it, wake_estimate quotes what a wake requested now would face: queued_wakes and the measured ready estimates. The same section is on every stop_node response. It reads the queue as it stands and reserves nothing. On a node that cannot start, wake_estimate is null, and the note says why.
Re-calling create_node with the same name is idempotent. You get the current status, never a duplicate. If a hold lapses, create the same shape under the same name within 15 minutes and you keep the place you waited for.
How the estimate is made. Your node can arrive in two ways. The queue drains as other people finish, or we add a machine for you. On a whole-node shape the second is often much faster, because eight free GPUs on one box rarely appear by chance. You are quoted the sooner of the two. capacity_inbound is true when the wait is bounded by capacity we are adding. A queued h100-8 with capacity_inbound: true is waiting on a machine. Hold your place. When the field is false, we are not adding capacity for that shape now, and a smaller shape can be the better plan. The field can turn true on a later poll.
Queue depth starts more machines. A queued node's GPUs count toward the fleet's machine target on the next procurement pass. A job backlog that persists for a shape raises the same target. Queued time is free and your place is held. Cancel a queued job or delete a queued node only when you no longer want the result. Every queued response says this in demand_note. This is a direction, not a promise. Spend ceilings and a quiet market can each mean no machine is coming right now, which is what capacity_inbound tells you.
Blocked on a resource. Sometimes the queue is not what holds your node. If the scheduler refuses the node on a resource, the queued response carries blocked_on. It names the resource, how much this node needs, how far short the closest machine is, and the move that changes it. Waiting does not clear most of these. A node that asks for a large scratch disk on a busy fleet is the common case. Ask for less scratch and it places at once.
autor node get trainerget_node("trainer")
→ {status: "queued", position: 1, wait_reason: "disk",
estimated_ready_seconds: null, unknown_because: "disk",
blocked_on: {resource: "disk", needs_bytes: 335007449088,
note: "gated on disk: this node needs 312 GiB free on one
machine, and the closest one is more than half short.
Ask for less scratch, or drop a snapshot you no longer
need; the requirement counts your scratch and a
restore's working copies together."}}The estimate is null in that case, and unknown_because names the resource. capacity is different. It means no machine is eligible right now for reasons on our side. Your estimate stays, your place is held, and there is nothing to change. The remedy sentence only names moves your own request allows.
gpu_graphics appears when your organization has GPU graphics on and no machine that has the graphics libraries has room. Keep the node queued: new machines install the libraries when they start, and capacity_inbound says whether one is coming. See GPU rendering.
snapshot_region appears when the node restores from a stored copy of its disk that is in one region. The stored copy is a snapshot or the replicated copy of the disk. No machine in that region can take the node now. blocked_on.region names the region. Your place is held, and you do not need to change anything.
Every queued response also carries wait_reason, one token that your code can branch on. It is queue_order when the node waits for its turn. When a resource holds the node, it is the same token as blocked_on.resource. It is null when we cannot tell on this poll. Poll again in that case. Queued jobs use the same field (see why a job waits).
A stopped node can also wait on disk when it wakes. You cannot change the size of its disk or its scratch after you create the node, so the note gives no move. Your place is held. The note tells you where the disk of the node is.
volume_quota means no machine has room in its volume budget for your node's volume. Every node has one persistent volume, and its size is the node_volume_gib limit for your org (250 GiB by default). A node holds that size on the machine that keeps its disk, running or stopped. A new node lands only on a machine with that much budget left.
There is no per-node volume size to ask for, and a raised limit fits fewer machines. Your place is held. The gate clears when a machine has budget for your volume. That happens when a node on that machine is deleted, when a parked disk leaves it, or when a new machine starts. capacity_inbound says whether a machine is coming.
Size the hold to the wait. Every queued response carries estimated_ready_seconds next to expires_at. When your hold is shorter than the estimate, the response says so in hold_note. Extend the hold, because a hold that lapses first means the node never arrives. A wake started by run_command holds for one hour by default, which is short for a busy shape.
When the estimate is longer than the 12-hour ceiling, renew the hold. Re-call create_node with the same name before each expires_at. Each renewal extends the live hold by up to 12 more hours. The deadline is in every response, so you can schedule the renewals. If a wake's hold lapses, the node returns to stopped with the files intact, and the next run_command queues it again. Work that can run unattended fits batch jobs better. The job queue holds a place for 3 days.
Set the boot command while you wait. A queued node takes a boot command. Register it as soon as create_node answers queued, and it starts the moment the node is ready, even when you are not polling. See On wake. Without one, poll get_node and send a run_command inside the idle window. A ready node with no command bills that window and then stops. Once the node runs, hold_node keeps it awake between your turns. Every queued response says this in arrival_note.
Do not delete and re-create a queued node. The new node waits from zero. Only a hold that lapsed hands its place to a re-create within 15 minutes. Your own delete or cancel does not. The delete also withdraws the demand that starts the next machine.
Poll every 2 to 5 minutes. That is inside the grace window, so you cannot lose the node between polls.
Poll from a shell instead. The node and job operations are also plain HTTP calls on the /preview API. A shell script, a CI step, or a cron job can hold the poll loop with a service token while your agent does other work:
# the prepackaged wait: blocks until ready, exits 0
autor node create --name tulip --chip 8xh100 --max-wait 6h --wait
# or the raw loop, no CLI needed
while true; do
status=$(curl -s -H "Authorization: Bearer $GMN_TOKEN" \
https://autoresearch.sfcompute.com/preview/nodes/tulip | jq -r .status)
case "$status" in running*) break;; esac
sleep 120
doneMCP and HTTP have separate rate limits, so a script's loop never spends your agent's tool budget. When the node is ready, one run_command from either interface attaches a session and ends the idle countdown.
If your MCP connection drops, every Autoresearch tool disappears until the session reconnects. Nodes, queue holds, grace clocks, and detached commands continue on the platform. When the tools return, read before you act. get_node shows where the node is. list_commands shows whether a detached command that never answered did start. A synchronous run_command that never answered leaves no record, and its process can still be running. Check its effects before you run it again. Do not create the node again, because that only buys a second queue wait.
The GPU shapes run on the same hardware: 8-GPU NVIDIA H100 SXM machines with 112 vCPUs and 2 TiB of RAM. Your node is a container on one machine, sized to the shape. One GPU buys 1/8 of the machine. Eight GPUs buy all of it.
| Shape | GPUs | CPU | RAM | Notes |
|---|---|---|---|---|
h100-1 | 1 H100 (80 GB HBM3) | 14 vCPUs | ~250 GiB | The GPU is assigned by UUID. You see your GPU only. |
h100-8 | 8 H100 (640 GB HBM3) | 112 vCPUs | ~1.9 TiB | The NVLink fabric between the GPUs. The only shape with full thermal isolation, and the only one that can unlock profiler counters. |
cpu-2 | none | 2 reserved physical cores | 8 GiB | Downloads, ingest, and preprocessing. |
cpu-8 | none | 8 reserved physical cores on one NUMA node | 32 GiB | Parallel CPU work with low inter-core latency. |
CPU nodes have a persistent, encrypted disk. Stop the node to park its disk. The next command wakes it. The reserved cores serve only your node, and their hyperthread siblings stay idle.
To move files between CPU and GPU nodes, use org storage. Call export_data on the source node. Call import_data on the destination node. These calls have no egress fee. Some snapshots also support direct restoration across node types. create_node checks compatibility before it creates a node.
A CPU node can be in a different region from your GPU nodes. Move data with storage and contexts. Do not assume a LAN. A private network reaches CPU nodes. clock_lock and profiling apply only to GPU nodes.
/dev/shm scales with your RAM instead of Docker's fixed 64 MiB. PyTorch dataloader workers and NCCL work without flags.node_volume_gib limit, one size for every node you create. An org admin can ask for more with request_limit_increase, and an operator decides. A raised limit narrows placement, because each node holds its full size on its machine (see Queue). Files survive stop and wake. Processes do not./scratch on the same NVMe. Ask for it at create time. It is free and it is destroyed at every stop. See Scratch.image at create time. Like every create-time property, it cannot change for the life of the node. See Images.dev user with passwordless sudo. Add distro packages with sudo apt-get update && sudo apt-get install -y <package>. Images ship with empty package lists, so the update is required. Commands run with no stdin, so the -y is required. Plain apt-get install fails with a dpkg lock error. Packages install outside /home/dev, so a stop resets them. Run the install again after a wake. Keep anything that must survive under /home/dev.Per-minute rates are on the front page and in every create_node and get_node response. Read ceilings such as node counts and volume size in limits before you hit them.
While the node is placed, get_node reports resources: the container's CPU and memory budget. Inside the container the same numbers are $GMN_CPU_LIMIT and $GMN_MEMORY_LIMIT_BYTES.
OMP_NUM_THREADS, RAYON_NUM_THREADS, MAX_JOBS, and PYTHON_CPU_COUNT to the CPU budget. OpenMP, Rust, and torch extension builds size to your slice without flags. The last three are clamped to the cores your container can run on, which on a shared machine can be fewer than $GMN_CPU_LIMIT. On the catalog images nproc answers the budget too, because GNU nproc reads OMP_NUM_THREADS. BusyBox and Alpine ship a different nproc that reports the host. A distroless image has none.ENV line in your Dockerfile. To use a different number, pass it in run_command's env. Your image build runs before any of this exists. Add ENV MAX_JOBS=8 to your Dockerfile if you build flash-attn or vLLM from source.os.cpu_count(), /proc/cpuinfo, psutil, and free all report the host. On a 1x node that is commonly eight times your budget. A thread pool sized from them is throttled. Size pools from $GMN_CPU_LIMIT. For a DataLoader, write num_workers=int(os.environ["GMN_CPU_LIMIT"]) instead of num_workers=os.cpu_count(). PYTHON_CPU_COUNT makes os.cpu_count() answer the budget, but only on Python 3.13 and newer. The default image is on 3.12. node_doctor reports the gap with your node's numbers./sys/fs/cgroup/pids.max. The default is 4096. Past it, every new thread fails with “Resource temporarily unavailable”. The ceiling is the node_pids_max limit. list_limits shows it, and an org admin can ask for more with request_limit_increase. A raise applies the next time the node starts or wakes.get_node reports environment with image_nvcc, the CUDA toolkit the image compiles with. Once the node is placed it also reports driver_version, cuda_major_max (the highest CUDA major the host driver runs), and compute_capability (the value for TORCH_CUDA_ARCH_LIST). Inside the container the same facts are $GMN_DRIVER_VERSION and $GMN_GPU_ARCH. A GPU node also reports gpu_graphics_enabled and gpu_graphics. See GPU graphics. image_torch is the torch the image ships. When it is null, the image ships no torch, and pip install torch picks one from the package index. Check its CUDA major yourself. A CUDA extension builds only when torch and nvcc agree on the CUDA major. Doctor checks this for you.Pass scratch_gib at create time to get a second filesystem at /scratch on the same local NVMe. Use it for caches and other data you can download again.
autor node create --name tulip --chip h100 --scratch-gib 200create_node(name: "tulip", chip: "h100", scratch_gib: 200)stop_node, an idle stop, a crash, an OOM kill, and a lost machine all count. The next wake mounts /scratch empty, and a stop note says so. Do not put the only copy of anything there.node_scratch_gib limit: 250 GiB by default. An org admin can ask for more with request_limit_increase, up to the 550 GiB platform hard cap, reported as hard_max by list_limits.scratch_gib can queue longer, and can hold up a wake. If you carry a lot of scratch, prefer delete_node and a fresh create over parking the node for days.Nothing in the environment points at scratch. Hardcode /scratch, or set the cache variable your library reads (HF_HOME, TRITON_CACHE_DIR, a StreamingDataset local=).
Streaming datasets. If a training run streams shards from object storage and decompresses them locally, re-streaming every epoch pays the egress and multiplies disk writes by the epoch count. Pick the first option that fits:
/scratch. Fill it once. Every later epoch is local NVMe reads and no network./scratch with a bounded decompressed cache. MosaicML StreamingDataset does this with keep_zip=True and a cache_limit, both pointed at /scratch. You pay decompress CPU instead of egress.pip install, and download from Hugging Face from inside the node. Two exceptions are refused at connect: outbound port 25 (SMTP), and a short list of crypto-mining pool ports.PIP_INDEX_URL and UV_DEFAULT_INDEX in your node and job environment. They point at a pull-through mirror of the public Python Package Index, so installs are faster. If you use your own index, set PIP_INDEX_URL yourself, in your environment or on the pip command line. Your value wins. To turn the preset off for your workspace, tell us.8xh100 node. The InfiniBand network of a cluster reaches only the nodes of that cluster. The InfiniBand network of a job on more than one machine reaches only the machines of that job.8xh100 has the machine to itself. We publish no committed Gbps figure. If your workload needs one, talk to us.create_node have no RDMA or InfiniBand between them. For InfiniBand between full machines, use a cluster. To run one job on more than one machine with InfiniBand, use a job with nodes. Private networks give ordinary IP between your nodes, for sharding, parameter servers, Ray, and anything that speaks TCP. Embarrassingly parallel work fits batch jobs over many machines.Run node_doctor on a running node before a long build. It checks the environment from inside the container.
autor node doctor tulip # exits 1 if it found anythingnode_doctor(node: "tulip")It checks five things:
nvcc. This decides whether a CUDA extension such as flash-attn, DeepSpeed, apex, or xformers can compile.free, /proc/cpuinfo, and psutil report.The answer carries findings, each with a problem and a fix. A check that does not apply is listed under skipped with the reason. The tool installs, repairs, and changes nothing. It does not wake a stopped node. On a running node it counts as activity and restarts the idle window.
While a node runs, we copy its changed disk blocks to encrypted storage outside the machine. There is nothing to configure. The one thing you see is a timestamp on get_node.
get_node("trainer")
→ { status: "running",
durable_as_of: "2026-09-14T02:11:48Z" } # 14 seconds agodurable_as_of is the newest instant of the disk that is already replicated off the machine. A restore gives back the disk exactly as it was at that instant. It advances only when the copy is written and verified. In steady state it trails the live disk by under a minute. When the disk takes writes faster than we can replicate them, it trails further and advances again when the writes slow down. A clean stop ships a final copy, so a parked node's replicated disk matches the disk exactly.
To copy one instant of the disk, we pause writes to it for a fraction of a second. This happens about every 30 seconds while the disk changes. Reads do not pause.
If the machine under your node fails, the node stops instead of being lost. Your next run_command brings it back on a healthy machine with the disk restored. At most, the writes after durable_as_of need redoing. If we cannot restore the copy, the node reads stopped (no restorable disk copy) and cannot start. Then durable_as_of gives the time of the copy that we cannot restore. Blocks are encrypted before they leave the machine. Every restored block is verified before your node reads it. delete_node destroys the replicated copy in the same instant it destroys the disk.
We keep the replicated copy in one of two places. One place is the storage cluster of one region. That is one storage cluster in one datacenter. The other place is central storage, which keeps copies in several availability zones.
A new copy starts when we first replicate the disk, and again after a restore from a snapshot. The copy stays in the place where it started. If a restore puts the node in a different region, the copy does not move to that region.
If the copy is on the storage cluster of a region, a restore of the copy uses a machine in that region. While no machine in that region can take the node, the wake waits with wait_reason: "snapshot_region". Your place is held. A restore can use another region if the node type requires it. Downloads across regions can take longer.
Replication protects against machine loss while a node exists. To keep a disk you can clone, or keep after the node is gone, take a snapshot. We delete a node that stays stopped with no activity for 60 days, with its disk and its replicated copy.
A snapshot is a copy of a node's disk that you keep. Take one after you have installed and downloaded everything. Every node you create from it starts with that work done. Stop the node first, because the snapshot copies the parked disk. Wait until get_node reads stopped. A node that reads stopping is still saving its last writes, and snapshot_node refuses it until the disk is parked.
autor node stop tulip
autor node snapshot tulip
autor node snapshots # poll until STATUS is readystop_node("tulip")
snapshot_node("tulip") # → { snapshot_id: "snap-...", status: "uploading" }
list_snapshots() # poll until status is readyThe call returns when the upload starts. Poll list_snapshots until the snapshot reads ready. Leave the node stopped until then. If you start the node, the upload is cancelled, the snapshot reads failed, and you take a new one after the next stop. Your /scratch disk is never in a snapshot. Work you do on the node after the snapshot is not in it. The upload runs through eight parallel connections. In production it moved snapshots of 32 to 35 GB at 107 to 110 MB/s to central storage. To in-region storage it moved a 35 GB snapshot at 157 MB/s (September 2026).
Some snapshots show a region. That value identifies the snapshot's storage location. A snapshot that shows a region is stored on one storage cluster in one datacenter. A restore can use another region if the selected node type requires it. Downloads across regions can take longer.
A snapshot without a region is in central storage, which keeps copies in several availability zones. Snapshots without a region can restore wherever compatible capacity is available.
To create a node from a snapshot, name the snapshot at create time. The new node has its own name and its own disk. The snapshot is unchanged, so you can clone the same environment many times.
A snapshot is also how you move a node off a failing host. A stopped node's disk stays on its host, so stop_node and run_command bring the node back on the same host. When get_node shows host_gpu_health and CUDA or nvidia-smi misbehaves, stop the node, snapshot it, and create a new node from the snapshot. A new node lands on a healthy host.
autor node create --name tulip-2 --chip h100 \
--from-snapshot snap-8k2mp --waitcreate_node(name: "tulip-2", chip: "h100", from_snapshot: "snap-8k2mp")The new node downloads the snapshot before it starts. A snapshot of 128 MiB or more downloads through parallel connections. The host uses one connection for each of its CPU cores, from 8 to 64 connections. A smaller snapshot downloads through one connection. While the download runs, get_node reports the bytes done and the connections in use in restore_progress, and the measured rate in provisioning.download_bytes_per_second. Plan your wait from that rate. In production, eight connections moved a 35 GB snapshot at 118 MB/s from central storage. From in-region storage the same eight connections moved it at 390 MB/s. One connection moved 17 to 30 MB/s (September 2026).
The platform checks the snapshot's sha256 checksum while the download runs. The node gets its disk only after the check passes. If the check is still running when the last byte lands, restore_progress shows step: verifying. Then items_done of items_total counts the bytes checked.
Do not stop a node to retry a restore that still runs. A stop ends the download. The host keeps the parts that passed the check for 6 hours, and the next restore of the node goes to that host if it has room. That restore continues from those parts. On a different host, the download starts again from the beginning. Read restore_progress.hint. If the host sends no progress report for 90 seconds, the hint tells you to wait. If the host is connected but sends no report for 15 minutes, the restore no longer runs. Then the hint tells you to stop the node and run a command again. Your files stay in the snapshot in both cases.
Snapshots bill for the storage they use. They never expire on our timer. You can set your own expiry, when you take the snapshot or later. At that instant the snapshot deletes itself. Deleting a snapshot frees the storage and stops the charge. Nodes already restored from it keep running.
list_snapshots shows who took each snapshot in triggered_by. The value is customer for a snapshot that your organization took. The value is system for a snapshot that we took to keep a copy of a parked disk. A system snapshot is yours like any other snapshot. You can create a node from it, set its expiry, or delete it. It does not bill storage, and it counts toward your organization's snapshot limits.
autor node snapshot-expiry snap-8k2mp 7d
autor node rm-snapshot snap-8k2mpset_snapshot_expiry(snapshot_id: "snap-8k2mp", expires_after: "7d")
delete_snapshot(snapshot_id: "snap-8k2mp")Deleting a snapshot cannot be undone. We destroy the key, so the stored copy becomes unreadable everywhere at once. A later attempt to create a node from it is refused.
The machines behind nodes retire on a schedule. A node on one has a deadline. get_node carries it as soon as the retirement is scheduled. That is about 45 minutes before the deadline, and sometimes hours before it. If the machine gets more time after that, the deadline goes away from get_node. An attached session then gets a notice that the retirement is off.
get_node("tulip")
→ { status: "running",
machine_retires_at: "2026-08-12T00:00:00Z",
machine_retirement_note: "this node's machine retires at 12:00am UTC (in 6h 12m).
tulip is stopped at that moment and its disk does not survive the machine.
Writing your results out (export_file, or a write connection) keeps them,
and stop_node then snapshot_node keeps the whole disk. After the machine
goes, tulip is lost with it: delete_node then create_node starts a fresh
one under the same name." }Poll for this field on a long unattended run. A stopped node carries it too, because a parked disk goes away with its machine. An attached session gets countdown notices in the terminal. Code inside the container can read the deadline from the metadata service.
The note says which of two things happens to your files. Some hosts can copy your disk off before they go. On those, the note says so. We stop your node up to fifteen minutes before the deadline, because the copy needs that time. Your next run_command restores the node. On other hosts the disk does not survive. Write your results out to storage or a write connection before the deadline, or stop the node and snapshot_node it. Plan to finish before the deadline, not at it.
The ending is on get_node. A node whose disk was preserved reads stopped with a reason of retirement, and your next command restores it. A node whose disk went with the machine reads lost. Run delete_node and then create_node under the same name. Nothing runs or bills in either state.
Rarely, the machine under a node stops answering the platform while the node itself keeps running. While that lasts, calls that reach into the node, like run_command and export_file, fail fast with "host isn't responding", and get_node carries a host section that says so.
get_node("tulip")
→ { status: "running (idle)", disk: "intact",
host: { responding: false,
last_heard_at: "2026-08-23T23:10:37Z", silent_seconds: 74,
note: "this node's host has stopped answering the platform ..." } }The status label reports the node's state, so it stays running through a silence. A monitoring loop must watch for the presence of host. A status change is not the signal. Most silences end within a minute or two. The host reconnects and the section disappears.
If the host is really gone, we end the node after a few minutes of silence and close billing back at the host's last heartbeat. We credit the interrupted session automatically. If the disk has a durable copy in object storage, the node parks on that copy and run_command restores it on another host. While the restore runs, get_node reports the bytes done in restore_progress. If the disk has no durable copy, we mark the node lost and credit the disk too. If the durable copy fails to restore after a host death, we credit the disk then, and the refusal says so.
The silence itself does not harm the container or the disk. If it outlives your patience, stop_node still works and ends billing now. The stop is delivered when the host reconnects.
Every node runs one image. Pass image to create_node to pick one from the catalog, or pass a registry ref to run your own. Pass nothing and you get the default. An unknown image name is refused with the current list.
CUDA version selection is image selection. This is the catalog:
| Image | CUDA | Ubuntu | Adds | Notes |
|---|---|---|---|---|
cuda-12.9 | 12.9 | 24.04 | - | The default when you name no image, and the profiling image. Its ncu is 2025.x, which works on current drivers. |
pytorch-2.13-cuda12.9 | 12.9 | 24.04 | torch 2.13 + torchvision (cu129), ffmpeg, libgl1 | The default plus current PyTorch. The start for image and video work: diffusion fine-tuning and generation, ComfyUI, and anything that imports cv2. The system libraries those stacks load are present. |
vllm-0.26-cuda12.9 | 12.9 | 24.04 | vLLM 0.26 | A pinned inference stack: vLLM and its torch, on a toolkit new enough for the kernels it compiles at import. |
pytorch-2.13-robotics-cuda12.9 | 12.9 | 24.04 | torch 2.13 (cu129), torchcodec, OpenCV, ffmpeg, GL/EGL/Vulkan loaders | Current PyTorch plus the video and vision toolchain that LeRobot and openpi load at import. |
jax-0.11-cuda12.9 | 12.9 | 24.04 | JAX with the cuda12 wheels | For openpi's JAX trainer and GPU-native simulators such as MuJoCo MJX. |
cuda-13.3 | 13.3 | 24.04 | - | The newest CUDA release. Needs a 580-series host driver, so it places on fewer machines than the default. |
pytorch-2.13-cuda13.3 | 13.3 | 24.04 | torch 2.13 | Current PyTorch on the newest toolkit (cu130). |
cuda-12.8 | 12.8 | 22.04 | - | A modern toolkit on the older Ubuntu userland, for stacks that want cu128 wheels or glibc 2.35. |
cuda-12.4 | 12.4 | 22.04 | - | Legacy (March 2024), kept for stacks pinned to it. Its CUDA headers are older than some current wheels expect. |
pytorch-2.4-cuda12.4 | 12.4 | 22.04 | torch 2.4 | Legacy, with PyTorch 2.4 preinstalled. |
Every catalog image has the same contract:
nvcc and the full toolkit are present, so source-built extensions compile. The GPU driver comes from the host. You never install one.pytorch-* and vllm-* variants and pip install onto your persistent disk are for that. Nodes provision fast.Install anything the image lacks at first boot. It lands in /home/dev (or a venv there) and survives stop and wake.
Snapshots and images. A node created with from_snapshot and no image runs the image its snapshot was taken on. Virtualenvs and compiled packages resume on the toolchain they were built under. list_snapshots shows each snapshot's image. You can name a different image. The response then warns that packages built on the original image can fail to import.
Use create_node with from_snapshot to restore a snapshot. Some snapshots support both CPU and GPU nodes. Others support CPU nodes with a catalog image only. An incompatible node type or image causes a refusal before node creation. Use export_data and import_data for file transfers when a snapshot cannot restore directly. Downloads across regions can take longer.
A restore onto the other node family can keep the file owners of the host the snapshot came from. Then the node's user cannot write the restored files. The create_node response warns you when this can happen, and it gives the fix for your restore. On a GPU node, run sudo chown -R "$(id -u):$(id -g)" "${GMN_HOME:-$HOME}" after the node starts. If the image runs as root, remove sudo from the command. On a CPU node, or if the image has no sudo and does not run as root, no command in the node can change the owners. Restore onto the family the snapshot came from, or copy the files with export_data and import_data.
Use org storage instead when more than one node needs the same data. Call export_data on the first node and import_data on each of the others. There is no egress fee on either call.
If your stack ships as a container image, run it directly. Pass a registry ref as image. Anything with a /, :, or @ reads as a ref, so ubuntu:24.04 and ghcr.io/org/repo:tag both work. Bare words stay catalog names.
autor node create --name miles --chip 8xh100 --image nvcr.io/nvidia/pytorch:24.01-py3 --home /workspacecreate_node(name: "miles", chip: "8xh100", image: "nvcr.io/nvidia/pytorch:24.01-py3", home: "/workspace")repo@sha256:... to pin a digest yourself, which also skips the registry read at create./bin/sh for run_command to work.home is where your disk lives. The persistent disk mounts at the home path you declare. If you do not declare one, the disk mounts at the home directory of the image's user, from the image's /etc/passwd. If the image has no such user, the disk mounts at /root. Only that path survives a stop. Everything else is the image, and it resets on every wake.home. A new disk is empty, and the image files at that path are not visible on the node. If your image installs its code in its user's home directory, declare a home that the image does not use, such as /data. After the first start, get_node names the image files that the disk hides.autor connection create --kind registry, pinned to a host and repository list. When your ref is inside a pin, the platform presents that credential at the pull. Credentials never ride the create call. See Registry.get_node reports what your image carries. After the first start, the custom_image block says whether python3, sudo, nvcc, and bash are present. It also gives a warning when the disk hides image files. node_doctor needs python3. sudo needs the image to ship sudo. CUDA extensions need nvcc.Use the catalog where it fits. Catalog images are pre-pulled on every host, so starts are instant, and their CUDA contract is checked in CI. Profiling with your own image works only if the image ships nsight-compute 2025 or newer. The platform cannot verify that for you.
A pip-installed kernel library can be built against a newer CUDA toolkit than your image ships. Recent flashinfer builds reference cuda::ptx symbols that exist only in a newer CCCL than CUDA 12.4 carries. On a hybrid or linear-attention model (Qwen3.5's gated-DeltaNet layers, for example), vLLM selects FlashInfer, tries to compile a GDN prefill kernel, nvcc reports missing symbols, and the engine dies before it generates a token. It looks like a broken install. It is a toolkit-age mismatch.
The default image does not have this problem. The default is cuda-12.9, and every catalog entry except the legacy cuda-12.4 line is checked at build time for these symbols. You hit this only on a node pinned to cuda-12.4. The image cannot change per node, so an old node keeps its old toolkit.
On such a node, do one of two things. Tell vLLM to skip the compile (gdn_prefill_backend=triton uses the Triton path and never runs nvcc), or create the node again on a current toolkit:
# either: skip the JIT on a node pinned to the old toolkit
autor run tulip -- python -m vllm.entrypoints.openai.api_server \
--model Qwen/Qwen3.5-8B --gdn-prefill-backend triton
# or: create on a current toolkit (the default already is one)
autor node create --name tulip --chip 8xh100
autor node create --name tulip-served --chip 8xh100 --image vllm-0.26-cuda12.9# either: skip the JIT on a node pinned to the old toolkit
run_command("tulip", "python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen3.5-8B --gdn-prefill-backend triton", detach: true)
# or: create on a current toolkit (the default already is one)
create_node(name: "tulip", chip: "8xh100")
create_node(name: "tulip-served", chip: "8xh100", image: "vllm-0.26-cuda12.9")When a pip-installed kernel library fails to compile at import, compare the toolkit it wants with the one your image has (nvcc --version) before you debug your environment. If a stack you rely on needs a newer toolkit or a different pinned version than the catalog offers, tell us. Adding an image is a small change on our side.
Three pinned stacks are entering the catalog for post-training and RL work. trl-0.21-cuda12.9 has TRL and vLLM in one image. verl-0.7-cuda12.9 has verl, which pins its own engine. sglang-0.6-cuda12.9 serves judges and reward models. Until they appear in create_node's image list, the post-training guides run on vllm-0.26-cuda12.9 plus one pip line.
If the stack you need already ships as a container image, you do not have to wait. create_node takes a registry ref as image (see Custom). Batch jobs build their own image from a context (the Dockerfile's FROM can name any public ref), or name a registry ref the same way.
The pytorch-2.13-robotics-cuda12.9 image carries torch plus the video and vision toolchain that robot-learning stacks load at import. That is ffmpeg, torchcodec, OpenCV, and the EGL and Vulkan loaders (libegl1, libvulkan1). They are present before your first command.
The camera renderers in these stacks need GPU graphics on for your organization. GPU graphics is off by default. An org admin turns it on. See GPU graphics.
MuJoCo (MJX), Genesis, and ManiSkill run their physics on H100 nodes. Their camera renderers need GPU graphics on: ManiSkill renders through SAPIEN and Vulkan, MuJoCo and Genesis render through OpenGL or EGL. NVIDIA Isaac Sim does not run here. It requires ray-tracing cores that H100 hardware does not have. If your pipeline is Isaac-based, only its headless physics paths work here. Plan visual sim-to-real training elsewhere.
For the JAX path (openpi's primary fine-tune trainer, and GPU-native simulators such as MuJoCo MJX) use jax-0.11-cuda12.9. It has JAX with the cuda12 wheels preinstalled.
A node computes on the GPU by default. With GPU graphics on, the node also renders on the GPU through Vulkan, EGL and OpenGL. Unreal Engine, SAPIEN and ManiSkill, and the camera renderers in MuJoCo and Genesis need it. GPU graphics is off by default. An org admin turns it on for the whole organization. Then create a new node.
autor gpu-graphics on
autor node create --name renderer --chip h100-8set_gpu_graphics(enabled: true)
create_node(name: "renderer", chip: "h100-8")You can also use the GPU graphics card on Settings, or send PUT /preview/org/gpu-graphics with the body {"enabled": true}. Only an org admin can change the setting, and each change is in the audit log. A node gets the setting at its next start. A node that exists now keeps its machine, so create a new node after you turn the setting on. To turn it off, run autor gpu-graphics off or call set_gpu_graphics(enabled: false).
autor gpu-graphics shows the setting, and list_team reports it as org.gpu_graphics. get_node reports two facts in environment. gpu_graphics_enabled says whether your organization has GPU graphics on. gpu_graphics says whether the machine under the node has the driver's graphics libraries. A node renders on the GPU only when both are true. GET /preview/nodes/{name} returns the same fields. node_doctor reports when GPU graphics is off for your organization. With GPU graphics on, it names the client libraries that your image does not have.
When GPU graphics is on, each new GPU node of your organization starts on a machine that has the driver's graphics libraries, and the node has NVIDIA_DRIVER_CAPABILITIES=all. Inside the node, /etc/vulkan/icd.d/nvidia_icd.json is present and vulkaninfo lists the H100. Your container image must carry the client libraries libvulkan1, libegl1, libglvnd0, libx11-6 and libxext6. The robotics image has them. The plain CUDA images do not. If libegl1 is not installed, vulkaninfo reports ERROR_INCOMPATIBLE_DRIVER. Use EGL with the surfaceless or device platform, or Vulkan headless. GLX through Xvfb stays a software renderer. H100 has no video encoder, so NVENC sessions fail. NVDEC works.
Without GPU graphics on, nodes do not provide GPU-accelerated Vulkan, OpenGL or EGL, and the loaders find no GPU device. Renderers that run on CUDA directly work, for example nvdiffrast in CUDA mode, PyTorch3D, and NVIDIA Warp. Unreal Engine and other Vulkan renderers need GPU graphics on. The same rule applies to a simulator that renders camera images through Vulkan, OpenGL or EGL.
A node that waits for a machine with the graphics libraries reports blocked_on.resource: gpu_graphics. Keep the node queued. New machines install the libraries when they start. A node that stopped on a machine without the libraries cannot wake there while GPU graphics is on. Create a new node instead. If the node does not need GPU graphics, an org admin can turn the setting off. The node can then start on any GPU machine.
There is no SSH. Commands run in the node's container, and the output is the interface.
autor run tulip -- sh -c 'git clone https://github.com/you/yourrepo && cd yourrepo && pip install --break-system-packages -e .'
# exit code = the remote exit coderun_command("tulip", "git clone https://github.com/you/yourrepo && cd yourrepo && pip install --break-system-packages -e .")
→ {exit_code: 0, stdout: "...", stderr: "...", truncated: false, timing_ms: {exec: 41230, server: 41288}}A synchronous run waits for the command to exit and returns its exit code and output. Running a command on a stopped node wakes the node first.
Every successful answer also carries timing_ms. server is the time we spent on the call, in milliseconds, from the request reaching us to the answer leaving. Tools that run something on a node add exec for that leg. Time beyond server is in your client or on the network.
Put secrets in env, never in the command string. Command lines are recorded. Env values are not.
autor run tulip --env HF_TOKEN=hf_... -- huggingface-cli download meta-llama/Llama-3.1-8Brun_command("tulip", "huggingface-cli download meta-llama/Llama-3.1-8B",
env: {HF_TOKEN: "hf_..."})Use short-lived tokens with the smallest scope that works. Treat a token that rode a chat as burned. To keep a value out of the chat entirely, store it once with autor secret create hf-write and reference it by name: env: {HF_TOKEN: {secret: "hf-write"}}.
Anything that outlives a tool-call timeout (training, a server) runs detached:
autor run tulip -d -- python train.py # returns immediately with the command id
autor logs tulip -f # newest command's output; -f follows to exit
autor ps tulip # all detached commands, newest first
autor kill tulip # TERM, then KILL after 10srun_command("tulip", "python train.py", detach: true)
→ {command_id: "cmd_...", log_path: "..."} # returns immediately
get_command("tulip", "cmd_...") # status + incremental output
get_command("tulip", "cmd_...", offset: 4096) # just the new bytes
list_commands("tulip") # all detached commands, newest first
kill_command("tulip", "cmd_...") # TERM, then KILL after 10sFiles survive stop and wake. Processes do not. It works like tmux on a machine you can power off. get_command never wakes a stopped node, so a check on finished work is free. A detached command runs with no deadline.
Read output in pages with offset. That is steadier than one synchronous command that prints all of it.
An env HOME reaches your command but not the command's markers. The log, the exit marker and a declared result stay under the image's home, where get_command and the result harvest read them.
A synchronous run gets 60 seconds by default. Pass timeout (CLI: --timeout) to raise it, up to 300 seconds from the CLI and the HTTP API. Over MCP the ceiling stays at 60 seconds. Agent clients enforce a per-call deadline of their own, commonly 60 seconds, and nothing comes back until the command finishes. A longer budget would die in your client and lose the output, so it is refused before anything runs. Work that needs longer runs detached. Leave timeout off when you detach. On a detached run it budgets only the wake of a stopped node.
autor run tulip --timeout 300 -- ./cleanup.sh# 300s is over the 60s MCP ceiling, so run it detached
run_command("tulip", "./cleanup.sh", detach: true)
→ {command_id: "cmd_...", log_path: "..."}
get_command("tulip", "cmd_...") # status, output, exit codeA synchronous command is killed when its timeout expires. The error carries what the command printed before the kill. Most programs buffer stdout when it is not a terminal, so a script that prints progress can die with nothing captured. Python holds back about 8 KiB. Run scripts with python -u, or set PYTHONUNBUFFERED=1, or print with flush=True. The same buffering applies to a detached command's log.
A call that times out at your own client's deadline says nothing about whether the command ran. Retry after a short backoff when the command is safe to run twice. Use a detached command plus get_command when it is not.
Killed line usually means the container's memory ceiling. When the kernel's OOM killer fired during the run, get_command carries an oom_kills count and a note that names the cause. The budget to size against is on get_node under resources.launch_error means the command did not start. The command writes its log and its markers in ~/.givemeanode/commands, and that directory was not writable. launch_error names the directory, its owner and the error. The cause is usually a disk that is full or read-only, or files that a restore left with another owner. get_command and list_commands show launch_error for every organization, whatever it keeps of command output. A command with restart: "on-failure" does not restart after this error.lost, with no exit code and no declared result. stop_node warns you and names every detached command that was still running. get_command on a lost command says what survived: your volume's files and the command's log, unless the node itself was lost with its host.list_audit_events. A stop, a spend cap, and an OOM kill look the same on the command row, but each lands its own audit event. data.actor.type is customer for one of your people or your key, and system for the platform (grace lapsed, the container died, the host went away). Platform rows belong to the org, so reading them takes the admin or billing role.$GMN_RESULT_PATH verdict are in ~/.givemeanode/commands/. Call get_command while the node is up. It reads that record and closes the command with the real exit code and result. A recovered exit code does not place a hold on the node. Wake the node first if it is stopped. When the disk holds no record, the note says so. Only a node lost with its host takes the record with it.To keep a verdict on the tool surface, poll get_command until it carries an exit_code, then stop the node.
A node with nothing running stops after its grace window. When you work by hand and think between commands, hold the node instead.
autor hold tulip --until 2h
autor hold tulip --releasehold_node("tulip", until: "2h") # or an RFC3339 time
→ {held_until: "2026-08-12T18:00:00Z", node_per_min: "$0.672/min",
cost_if_held_to_expiry_usd: 80.64}
release_node("tulip") # back to the normal idle timeruntil is required and has a ceiling. list_limits shows the ceiling.release_node ends a hold early and puts the node back on its normal idle timer. stop_node stops billing now and clears the hold.stop_node ends that bill.A hold defers two things: the idle timer, and your workspace's idle-burn auto-stop if somebody armed one. Both look at CPU as well as GPU on their own, so a file transfer, a dependency install, or a dataset download defers them without a hold. A running detached command defers them too. A hold is for a node you want awake while nothing runs on it. Use a hold instead of a background sleep loop. A sleep loop costs a command slot and shows on your mission page as work.
The idle-burn alert counts only the minutes after the node became ready. The minutes while a node starts, wakes or restores from a snapshot do not count, and neither do the minutes before its last stop. A node runs for the full window you set before the alert can stop it.
By default, a failed command ends like any other. The node idles out and /scratch is destroyed. Ask for a hold on failure and the node stays awake with the failed attempt's data on disk.
autor run tulip -d --hold-on-failure 120 -- python train.pyrun_command("tulip", "python train.py", detach: true,
hold_on_failure_minutes: 120)A clean exit is unaffected, and the node idles out as usual. A nonzero exit, a kill by a signal, and a supervised command whose restart budget runs out all count as failures. get_node shows the hold and names the command that caused it. A command whose first launch did not start (exit code 127 with a launch_error and no restarts) did not run, and it does not start a hold. If a restart did not start, the attempts before it ran and failed, and the hold starts.
A node you wake or create can arrive in two minutes or in six hours. If nothing runs on it within its grace window, it stops again and bills that idle time. Register the command in advance, and it starts the instant the node is ready. A boot command works on a stopped node and on a queued node. Register it as soon as the create or the wake answers queued.
autor run tulip -d --on-wake --max-duration 240 -- ./bootstrap.shrun_command("tulip", "./bootstrap.sh", detach: true, on_wake: true,
max_duration_minutes: 240, max_wait: "12h")
→ {status: "queued", position: 3, estimated_ready_seconds: 16800,
boot_command: {command_id: "cmd-4k2x9", state: "pending_wake"}}
get_command("tulip", "cmd-4k2x9") # answers now, before the node exists
→ {status: "pending_wake"}get_command answers for it from that moment. get_node shows the registration next to the queue facts. When the node is ready, the command starts as an ordinary detached command with the env, restart, and mission you gave it.max_duration_minutes is required. When it expires, the command is killed and the node returns to its normal grace window. list_limits shows the ceiling.kill_command cancels a registration that has not run. Nothing was billed, because no process started.on_wake to run the command now.Small files go up with write_file. Results come back with export_file. Large inputs that exist somewhere reachable are cheapest pulled from inside the node with curl, git, or huggingface-cli. A large input that exists only on your own machine goes through storage.
Use write_file for configs and entry scripts. Do not use it for secrets or datasets.
autor cp ./run.sh tulip:~/run.sh --executablewrite_file("tulip", "~/run.sh", "#!/bin/sh\npython train.py\n", executable: true)autor cp uses the scp grammar. The side that names a node is written node:path. The other side is a path on your own machine.~/run.sh and /home/dev/run.sh are the same file.max_write_file_bytes in autor limits. Anything larger goes through storage, which streams.sha256 of the content. The node hashes the file again before it moves it into place, and refuses the write if the two differ. If you built the content by hand, hash your copy and compare. If the image has no sha256sum, the response says sha256_verified: false, and the file was checked by length only.written: false. Send the content again when the node is running. To push several files, wake the node, hold it, then write.base64 and pass the content as base64 text. The sha256 in the response is the hash of the decoded bytes, and the node checks it before the file lands. autor cp does this for you. Do not decode base64 on the command line with run_command. That records the bytes and has no integrity check.A file over the single-call cap goes up in parts. Send each part with write_file_part, then assemble them with commit_file_parts. The node holds the parts and puts the file together. autor cp takes this path by itself when the file is over the cap.
autor cp ./corpus.zip tulip:~/data/corpus.zip # over the single-call cap, so it goes up in 15 parts# split the file into parts of at most 4 MiB, base64 each
write_file_part("tulip", upload_id: "3f9a0c1e2b7d4a5f", part_index: 0, content: "<base64>", encoding: "base64")
write_file_part("tulip", upload_id: "3f9a0c1e2b7d4a5f", part_index: 1, content: "<base64>", encoding: "base64")
# ... one call per part, in any order
commit_file_parts("tulip", upload_id: "3f9a0c1e2b7d4a5f", path: "~/data/corpus.zip", part_count: 15, sha256: "3f9a0c1e...")
→ {written: true, path: "data/corpus.zip", bytes_written: 62914560, sha256: "3f9a0c1e...", sha256_verified: true, parts: 15, timing_ms: {exec: 640, server: 702}}timing_ms. A part lands in well under a second on our side. When a part call takes much longer in your chat, the rest of the time is in your client or on the network. The agent writes the content of every part as text in the call. Base64 text is 4/3 the size of the file. Send text files as utf8. The bytes cost the same however you split them. Fewer, larger parts mean fewer round trips, so use the largest part your client accepts.max_write_file_bytes. The whole file is at most max_chunked_write_bytes, and an upload has at most max_chunked_write_parts parts. autor limits shows the three values.upload_id. Use the sha256 of the file, or its first 16 characters. A retry with the same id continues the same upload.part_index starts at 0. A part sent again with the same index replaces the earlier part, so a retry is safe.~/.givemeanode/uploads/<upload_id>/ on the node. Nothing is stored on our side between the calls. The parts survive a stop and a wake.sha256 you give, moves the file into place, and removes the parts. If a part is missing, the commit refuses and names the missing parts and the size of each part on the node. Send those parts again, then commit again.written: false. Parts already on the node stay. Send the call again when the node is running.run_command. The next commit on the node removes part directories that nobody touched for 7 days.export_file returns a presigned download URL. A directory arrives as a tar.
autor cp tulip:~/yourrepo/checkpoints ./ # arrives as a tar
autor api post /nodes/tulip/exports --data '{"path": "~/yourrepo/checkpoints"}'
→ {"url": "https://...", "expires_at": "..."} # the link, if you want it insteadexport_file("tulip", "~/yourrepo/checkpoints")
→ {artifact_id: "art-...", state: "ready", download: {url: "https://...", expires_at: "..."}}autor cp does the export and the download in one step and checks the digest. Use export_file directly when you want the link, which is what an agent usually wants. The link is private and made for machines. If you lose it, get_artifact returns a fresh one for the same export for about 30 days.
A large path can take minutes to export. Over MCP, export_file waits about 45 seconds, which is less than the time an MCP client usually lets one call run. If the export is not done then, the call returns the artifact id with state: "pending". The export continues. Call get_artifact with that id until the state is ready, and the answer then has the download link. The HTTP API waits up to 5 minutes before it returns the same pending answer.
autor cp tulip:~/ckpt/step-9000.pt ./ # waits until the export is ready
autor api get /artifacts/art-... # the state, and the link when it is readyexport_file("tulip", "~/ckpt/step-9000.pt")
→ {artifact_id: "art-...", state: "pending", next_step: "Still running. Call get_artifact ..."}
get_artifact("art-...") # call again until state is "ready"
→ {artifact_id: "art-...", state: "ready", download: {url: "https://...", expires_at: "..."}}If you do not have the artifact id, list_artifacts lists your recent exports, newest first. Give it a node name to list only the exports from that node.
Each export refusal has an error code next to the sentence that explains it. Use the code to decide what to do next. Over MCP the code is in the tool result as {error: {code, message, retryable}}. Over the HTTP API it is the error.code field, and the HTTP status follows it. When retryable is true, the same call can work if you try it again. get_artifact and get_export show the same error_code and retryable for an export that failed after the call returned. The codes apply to export_file, export_data, and the export part of copy_file.
NOT_FOUND means the path is not in the container. A symlink whose target is gone gives this code too. So does a file that your trainer renamed between your call and the read. You can retry.NOT_A_FILE means the path is a socket, a named pipe, or a device. Export the file or the directory itself.READ_FAILED means the read in the container failed, usually because of permissions. The message shows the error that the read printed.TOO_LARGE means the path is over the size limit. export_file stops at 5 GiB, and no limit increase goes past that. Use export_data for a larger path. It uploads in parts to your own storage.RATE_LIMITED means you are over your export limit for the hour. Wait, or export the whole directory in one call.NODE_NOT_RUNNING means the node is stopped, starting, or lost. An export reads the running container. On a stopped node, the message gives the cost of a wake. A run_command wakes the node, and the wake bills, so wake it only when you need the file now.NODE_NOT_FOUND means your workspace has no node with that name. list_nodes shows your nodes.HOST_UNAVAILABLE means the machine of the node did not answer, or stopped during the export. Retry in about a minute.WORKER_DISK_FULL means the machine had no disk space to stage the export. Retry.TRANSFER_FAILED means the upload stopped or the storage refused it. Retry.TIMEOUT means the export did not finish in time. From copy_file, the export can still finish, so call get_artifact with the artifact id in the message before you copy again. On a failed artifact, it means that the export never reported a result.INVALID_PATH means the path was empty or relative. Give an absolute path, or a path that starts with ~/.INVALID_MISSION means the mission argument does not name a mission in this workspace.DESTINATION_DENIED means your own storage refused the write. The connection can be missing, disabled, or read-only, or the provider can refuse the credentials.NOT_ENABLED means exports are not available on this deployment.EXPORT_FAILED means the failure has no code yet. Read the message.An export does not refuse a file that is open. It reads the bytes that are on disk at that moment. If your trainer is still writing a checkpoint, the export can be short and incomplete, and it still reports success. Export only files that the trainer has closed. Write each checkpoint to a temporary name, then rename it when the write is complete.
copy_file moves a file or directory between two running nodes of your workspace in one call. The nodes do not need a shared network.
autor cp tulip:~/app poppy:~/app # polls until it landscopy_file(src_node: "tulip", src_path: "~/app", dst_node: "poppy", dst_path: "~/app")
→ {import_id: "imp-...", status: "transferring", size_bytes: ..., sha256: "..."}
get_import("poppy", "imp-...") # poll until completedst_path exactly.dst_path, merged file by file over what is there. A second copy after edits overwrites the changed files only.get_import on the destination node until it reports complete.share_artifact makes one exported file public at a clean URL. The link carries the real filename and content type. Slack unfurls an image, <img src> works in a page, and a browser shows the file instead of downloading it.
autor api post /artifacts/art-df5cj/shares
→ {"share_id": "shr-9k2fq", "url": "https://s.givemeanode.io/shr-9k2fq-vx7t3m8dk4qwrz2n/cat1.jpg", ...}export_file("tulip", "~/out/cat1.jpg")
→ {artifact_id: "art-df5cj", ...}
share_artifact("art-df5cj")
→ {share_id: "shr-9k2fq", url: "https://s.givemeanode.io/shr-9k2fq-vx7t3m8dk4qwrz2n/cat1.jpg",
filename: "cat1.jpg", content_type: "image/jpeg", expires_at: "..."}list_shares shows everything your org has public. unshare revokes a link at any time.expires_after. Sharing the same artifact again refreshes the same URL..tar download.Transfers use a second hostname. Every presigned URL the platform returns points at its object store, which is fly.storage.tigris.dev. That includes the upload URL from create_context, the download URLs from get_context and export_file, and the artifact, preview, and sample links on jobs. If your agent runs behind an egress allowlist, allow that host too. With only the API allowed, every tool call works and every transfer fails.
A blocked host usually shows as a 403 from your own proxy when curl connects, before any bytes reach the store. Check the hostname against your allowlist before you debug the signature. These are the hostnames the product uses:
autoresearch.sfcompute.com and mcp.autoresearch.sfcompute.com serve the web pages, the HTTP API, and MCP. Existing clients can still use givemeanode.com and mcp.givemeanode.com.fly.storage.tigris.dev serves the presigned upload and download URLs.storage.givemeanode.com and <bucket>.storage.givemeanode.com serve the S3 endpoint, if you use it.A connection is a stored credential for data behind a login: a private bucket, a gated Hugging Face repo, a private registry, or an authenticated URL. An org admin creates it once from a terminal or CI. The platform uses the credential on the node's host. Agents see connection names, never secrets.
curl -fsSL https://autoresearch.sfcompute.com/cli.sh | bash && autor login # once, ~2 minutes
autor connection create corpus --kind s3-keys \
--scope s3://my-bucket/datasets/ \
--access-key-id AKIA... --material-stdin < secret-key.txtEvery connection follows the same rules:
--material-stdin), in the GMAN_CONNECTION_MATERIAL environment variable, or at a hidden prompt. Never as a flag.--scope. Imports outside the scope are refused by the platform, whatever the credential itself allows.--verbs read,write. Write lets agents export results through them./team). Every create, get, and list answer names the organization it acted in, and autor connection ls prints it above the rows. A lookup that misses tells you when the name exists in another of your organizations.A run does not have to wait for a connection. Until an admin has set one up, pass the cloud credentials in the environment of the command and sync from inside the node. The values ride --env, never the command line, and they are not stored.
autor run my-node \
--env AWS_ACCESS_KEY_ID=AKIA... --env AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
-- aws s3 sync s3://my-bucket/datasets/ ~/data/The kinds:
| Kind | Reaches | Scope |
|---|---|---|
s3-keys | Any S3-compatible bucket: AWS, R2, MinIO, B2 | s3://bucket/prefix/ |
huggingface | Gated or private Hugging Face repos | owner/name, owner/* |
civitai | Community models and LoRAs on Civitai | model:123, version:456 |
registry | A private Docker or OCI registry | ghcr.io/acme/* |
https | One GET with a token in a header | https://host/path/ |
An s3-keys connection holds an access key pair for one bucket prefix. The key id rides --access-key-id. The secret key is the material.
# AWS
autor connection create corpus --kind s3-keys \
--scope s3://my-bucket/datasets/ \
--access-key-id AKIA... --material-stdin < secret-key.txt
# Cloudflare R2, MinIO, Backblaze B2: add the endpoint
autor connection create r2-corpus --kind s3-keys \
--scope s3://my-bucket/datasets/ --endpoint https://<account>.r2.cloudflarestorage.com \
--access-key-id ... --material-stdin < secret-key.txt
# a connection agents can also export to
autor connection create acme-out --kind s3-keys \
--scope s3://acme-ml/ --verbs read,write \
--access-key-id ... --material-stdin < secret-key.txts3://bucket/prefix/. The prefix can be empty: s3://bucket/.--endpoint names a non-AWS endpoint. --region names the bucket's region. The default is us-east-1.s3-keys connection are full s3:// URLs, because the bucket is not implied.A huggingface connection reaches gated and private repos. On huggingface.co, create a fine-grained token (Settings, then Access Tokens) with read access to the repos in the scope. Add write access if the connection will push checkpoints.
# read a private adapter repo
autor connection create loras --kind huggingface \
--scope acme/loras --material-stdin < token.txt
# read and write a checkpoint repo (the repo must already exist)
autor connection create ckpts --kind huggingface \
--scope acme/ckpts --verbs read,write --material-stdin < token.txt
# no token at all: an org admin authorizes once in the browser
autor connection create ckpts --kind huggingface \
--scope acme/ckpts --verbs read,write --auth oauthowner/name is one model repo. datasets/owner/name is one dataset repo. owner/* is every repo of that owner. * is everything the token reads.--auth oauth stores no token. An org admin authorizes in the browser, and short-lived tokens are minted per use. The command prints the link and waits for the approval.autor connection ls # loras huggingface read acme/loras
autor import tulip --connection loras --source acme/loras \
--dest '~/models/loras' --waitlist_connections()
→ [{name: "loras", kind: "huggingface", scope: "acme/loras", ...}]
import_data("tulip", connection: "loras",
source: "acme/loras", dest_path: "~/models/loras")
→ {import_id: "i-7pd12", status: "running"} # poll get_importA civitai connection holds a Civitai API key from civitai.com/user/account. It authorizes gated and early-access downloads of community models and LoRA adapters.
autor connection create community-loras --kind civitai \
--scope model:1115064 --material-stdin < api-key.txtmodel:1115064 pins one model, and an import of it pulls the newest published version. version:1253021 pins one exact version. Several pins are separated by commas. * allows any.autor connection ls # an admin created one: kind civitai, scope model:1115064
autor import tulip --connection community-loras \
--source model:1115064 --dest '~/models/loras' --waitimport_data("tulip", connection: "community-loras",
source: "model:1115064", dest_path: "~/models/loras")
→ {import_id: "i-9wk21", status: "running"} # poll get_importA registry connection holds a pull credential for a private Docker or OCI registry. Jobs, nodes, and sandboxes that name an image inside its scope pull with it. Nothing about the image reference changes.
autor connection create acme-images --kind registry \
--scope 'ghcr.io/acme/*' --username acme-ci --material-stdin < token.txtghcr.io/acme/trainer pins one repository. ghcr.io/acme/* pins an owner. ghcr.io/* pins a whole registry. Every entry must name the same host. Create a second connection for a second registry.--username is the login half of the credential. On GHCR it is your username, with a personal access token as the material. On Quay and Harbor it is a robot account. On Google Artifact Registry it is _json_key, with the service account JSON as the material.context.FROM. A job that builds from a context needs a base it can pull anonymously.autor job submit --chip h100 --count 8 \
--image ghcr.io/acme/trainer:v4 --command 'python train.py'submit_job(chip: "h100", chip_count: 8, mission: "trainer-v4",
image: "ghcr.io/acme/trainer:v4",
command: "python train.py",
idempotency_key: "trainer-v4-run1")
→ {job_id: "job-8x2mf", status: "building"}An https connection fetches one URL with a token in a header. It is for an authenticated download that is not S3 or a hub.
autor connection create exports --kind https \
--scope https://data.corp.example/exports/ --material-stdin < token.txt
# a header other than Authorization
autor connection create exports --kind https \
--scope https://data.corp.example/exports/ --header X-Api-Key --material-stdin < token.txthttps://. An import names a URL under it.Authorization: Bearer <token>. --header names a different header. The value is still Bearer <token>.Agents import by connection name. The import runs on the node's host and streams into the container.
autor connection ls # names, kinds, scopes - never material
autor import tulip --connection corpus \
--source s3://my-bucket/datasets/train/ --dest '~/data/train' --waitlist_connections()
→ [{name: "corpus", kind: "s3-keys", scope: "s3://my-bucket/datasets/", ...}]
import_data("tulip", connection: "corpus",
source: "s3://my-bucket/datasets/train/", dest_path: "~/data/train")
→ {import_id: "i-4kd93", status: "running"}
get_import("tulip", "i-4kd93")
→ {status: "complete", total_bytes: 5368709120, files: 128, dest_path: "~/data/train"}--dest that starts with ~. An unquoted ~ expands to your local home directory before the command runs, and the import fails on the node.A connection created with --verbs read,write can receive results through export_data: model checkpoints, generated datasets. The platform uses the credential on the node's host, so it never enters the container. Write is available on s3-keys and huggingface connections.
autor connection ls # ckpts huggingface read,write acme/ckpts
autor export tulip --connection ckpts \
--source ~/out/checkpoints --dest acme/ckpts@main:runs/run7/ --waitlist_connections()
→ [{name: "ckpts", kind: "huggingface", verbs: ["read", "write"],
scope: "acme/ckpts", ...}]
export_data("tulip", connection: "ckpts",
source: "~/out/checkpoints", dest: "acme/ckpts@main:runs/run7/")
→ {export_id: "exp-4kd93", status: "running"}
get_export("tulip", "exp-4kd93")
→ {status: "complete", total_bytes: 4831838208, dest: "acme/ckpts@main:runs/run7/"}owner/name[@revision][:path/in/repo]. A directory lands as individual files in one commit, never a tar, so from_pretrained works on the result.If your training code pushes to the Hub itself (push_to_hub in the trainer loop), the token has to be inside the container. Store it as a secret and reference it by name in env, so it stays out of the chat.
autor connection ls # names, kinds, verbs, scopes
autor connection get corpus # one connection; material is never shown
autor connection arm corpus --material-stdin < new-secret.txt # replace the secret
autor connection disable corpus # in-flight imports fail at their next chunk
autor connection enable corpus
autor connection rm corpus # the sealed material is shredded noworg:write.list_connections over MCP. They cannot change them.autor connection arm prints the link.Org storage is buckets on Autoresearch's own object store. There is no second vendor and no credential to manage. Upload from a laptop or CI with the CLI. Every bucket appears to agents as the connection storage/<name>, so nodes pull from it with import_data.
autor storage bucket create corpus
autor storage cp ./train-data corpus/train/ # a directory becomes one object per file
autor storage cp ./icons corpus/icons/ --concurrency 64
autor storage ls corpus/
autor storage cp corpus/train/a.bin ./ # download one object
autor storage cp corpus/train/ ./train # download everything under a prefix
autor storage cp corpus/ ./corpus.zip # download the whole bucket as one zip
autor storage rm corpus/train/a.bin # delete one object
autor storage bucket ls # footprint and estimated cost per bucket--concurrency sets how many files are in flight. The default is 32 and the maximum is 256. An empty file uploads as a zero-byte object..zip destination gives one archive instead.--force, which cannot be undone.autor import tulip --connection storage/corpus --source train/ --dest '~/data/train' --waitimport_data("tulip", connection: "storage/corpus",
source: "train/", dest_path: "~/data/train")source is a key inside the bucket, written as autor storage ls prints it. s3://corpus/train/ means the same thing. A dest on export_data works both ways too./ imports every object under that prefix. An exact key imports one object. A source with no trailing slash that names no exact object imports every key it prefixes.max_import_files). Past that, split by first letter: import items/a, then items/b, and so on. The pieces land side by side under the same dest_path.imports_per_hour). The count is per person, across every node, workspace, and org you use. The window slides. The refusal says how many you started, when the next slot frees, and how long that is. A whole-prefix import counts as one call, however many objects it carries. An operator can raise both limits.An agent can look inside a bucket before it pulls anything. A list reads the index only. It costs no node time and needs no running node.
autor storage ls corpus/
autor storage ls corpus/train/list_storage_objects("corpus")
list_storage_objects("corpus", prefix: "train/")
list_storage_objects("corpus", delimiter: "/") # one directory level at a timelist_storage_objects returns the keys under a prefix with their sizes.delimiter: "/" to answer one directory at a time. Everything below the next / folds into a list of prefixes, such as ["icons/", "train/"]. Pass one of them back as the prefix to descend. A bucket with a million objects is a few calls this way.truncated: true has more. Call again with after set to the next_after it returned.autor storage ls always lists flat. On a very large bucket, use a narrower prefix from the terminal.Every bucket is also reachable over the S3 protocol, so aws s3, boto3, DuckDB, rclone, and anything else that speaks S3 works directly. Mint a scoped access key and point your tool at the endpoint the mint prints.
autor storage key create --bucket corpus --access rw
# prints AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (once) / endpoint
aws s3 cp ./train-data s3://corpus/train/ --recursive \
--endpoint-url https://storage.givemeanode.com
duckdb -c "SELECT count(*) FROM read_parquet('s3://corpus/train/*.parquet')"
autor storage key ls
autor storage key revoke gmnk_... # immediate; the secret is shreddedbucket.storage.givemeanode.com) work. Set AWS_REGION=auto.import_data through the storage/<name> connection is credential-free.These numbers bound an upload. Most are defaults that support can raise for your org. Your live values are in limits.
SlowDown: back off and retry.An example for a 12.6 GB training checkpoint. A plain aws s3 cp of the whole file works. The shard-and-manifest pattern adds integrity you verify on readback, and shard-level resumability:
# upload: 4 GiB shards plus a checksum manifest
tar -cf - ckpt/ | split -b 4G - ckpt.tar.part-
sha256sum ckpt.tar.part-* > MANIFEST.sha256
aws s3 cp . s3://ckpts/run7/ --recursive \
--exclude '*' --include 'ckpt.tar.part-*' --include 'MANIFEST.sha256' \
--endpoint-url https://storage.givemeanode.com
# restore: download, verify byte-for-byte, reassemble
aws s3 cp s3://ckpts/run7/ . --recursive \
--endpoint-url https://storage.givemeanode.com
sha256sum -c MANIFEST.sha256 # every shard must print OK
cat ckpt.tar.part-* | tar -xf -The same works with autor storage cp and no key. autor storage cp ./shards ckpts/run7/ uploads the directory in parallel, and autor storage cp ckpts/run7/ ./restore brings it back. From a node, use import_data and export_data through the storage/<name> connection.
The platform verifies these on every upload:
IncompleteBody) and never becomes an object.x-amz-content-sha256 is recomputed and a mismatch is refused. Streaming (aws-chunked) uploads, the aws CLI v2 default, carry CRC trailers instead. Those are accepted but not recomputed. TLS covers the transport and the length check still applies.For bytes you cannot re-derive, checkpoints above all, write your own sha256sum manifest beside the data and verify it on readback, as in the example above.
Retries are safe:
CompleteMultipartUpload whose response you lost. The commit converges on the assembled result.AbortMultipartUpload cleans up staged parts now. Single-PUT staging debris is swept after about 24 hours. We abort an abandoned multipart upload 7 days after you start it. A part or a CompleteMultipartUpload sent after that fails. Always abort what you do not finish. Deletes take effect at once.CompleteMultipartUpload on a multi-GB object can take a while. Like S3, we hold the connection open with whitespace and deliver the outcome in the body of a 200, which is what the AWS SDKs expect. You do not need to raise your client's read timeout. If the connection drops before the answer, the Complete is cancelled where it was. HeadObject the key to see whether it committed. Your parts are still there, and a second Complete assembles them again.Storage bills per GB-month at each day's peak footprint, prorated daily. The first 100 GB each month are free. Then the rate steps down with size, marginally, like tax brackets:
| Footprint | Rate per GB-month |
|---|---|
| First 100 GB | free |
| To 10 TB | $0.10 |
| To 100 TB | $0.07 |
| To 1 PB | $0.05 |
| Beyond | $0.035 |
autor storage bucket ls shows each bucket's footprint and the month's estimated cost. get_billing() reports storage spend beside compute.
Put a port on a running node at a public HTTPS URL: TensorBoard, Jupyter, or a vLLM endpoint that serves a model to a judge.
autor run tulip -d -- python -m vllm.entrypoints.openai.api_server --port 8000
autor api post /nodes/tulip/endpoints --data '{"port": 8000}'
→ {"url": "https://<endpoint-id>.<edge>", "expires_at": "..."}run_command("tulip", "python -m vllm.entrypoints.openai.api_server --port 8000", detach: true)
expose_port("tulip", port: 8000)
→ {url: "https://<endpoint-id>.<edge>", expires_at: "..."}auth: "bearer" also requires an Authorization: Bearer token at the edge. The token is in the response once and never shown again.expires_at is the deadline. Call expose_port again to extend it.expose_port again for a fresh URL.list_endpoints(node) shows what is live. unexpose_port(node, port) takes it down.vanity: "mylabel" also claims a readable hostname beside the capability URL. A vanity name is guessable, so pair it with auth: "bearer" when the port must not be public. The namespace is global and first-come. Reserved words refuse. The name frees when the endpoint dies. A re-expose with the same vanity keeps it.A private network lets your own nodes reach each other over stable private addresses. Nothing on it is visible from the internet.
autor node create --name head --chip h100 --network mesh1
autor node create --name worker1 --chip h100 --network mesh1
autor run head -- ping -c 3 worker1create_node("head", chip: "h100", network: "mesh1")
create_node("worker1", chip: "h100", network: "mesh1")
run_command("head", "ping -c 3 worker1")autor network attach worker2 mesh1 (attach_network) converges a running node in seconds with no restart. autor network detach worker2 removes the interface just as live. autor network ls (list_networks) shows every network and its members.worker1 and worker1.givemeanode.internal both resolve to that peer, so MASTER_ADDR=head works. The metadata service serves the whole roster at /v1/network.get_node, list_networks, the attach_network response, and /v1/network from inside a member show a data_plane field. up means the interface gmn0 is in the container. converging means the interface installs on the host's next heartbeat, normally within a few seconds; if it still reads converging after about a minute, file a ticket with the node name. not_placed means the node is stopped or queued, and it joins at its next wake. unavailable means the host that runs the node has no mesh support; the member keeps its membership and addresses, and the interface appears when the host gains mesh support or when the node runs on a host that has it.A cluster is two or more full 8-GPU machines on a private InfiniBand network that we make for that cluster alone. Your nodes move data to each other over InfiniBand RDMA. Clusters are a preview, and every account can make them. To run one batch job on more than one machine, use a job with nodes.
autor cluster create --name train --chip h100-8 --nodes 2 --gpus 16 --wait
autor cluster show traincreate_cluster(name: "train", chip: "h100-8", nodes: 2, gpus: 16)
get_cluster("train") # poll until state is runningA cluster uses full machines only. create_cluster needs all three of chip, nodes, and gpus. If one value does not agree with the rules below, the call fails with the rule. It never rounds.
chip must be h100-8. The spelling 8xh100 also works. Each node of the cluster is one full machine.nodes is at least 2. The maximum is 8, and list_limits shows the maximum for your account. For one machine, use create_node with chip: "h100-8".gpus must equal nodes times 8. A request for 2 nodes is a request for 16 GPUs.image, home, mission, and max_wait work as they do on create_node. They apply to every node. max_wait applies only while the cluster waits for machines.network, clock_lock, profiling, scratch_gib, or from_snapshot. The cluster makes its own private network.create_cluster never blocks. Each response carries the cluster state and the rate. rate.cluster_per_min is the cost of all the nodes per minute. A call with the name of a live cluster and the same shape returns that cluster.
| State | Meaning |
|---|---|
queued | The cluster waits for free machines. |
preparing | The machines join the new InfiniBand network for this cluster. |
starting | The node containers start. Each node bills from the moment it is ready. |
running | Every node runs. |
ending | The nodes stop and their disks are deleted. |
ended | The cluster is gone. |
failed | The cluster ended without your request. The failure field says why. |
Preparation can take 10 to 45 minutes. The wait field says what the cluster waits for. No node bills while the cluster is queued or preparing.
The nodes of cluster train are train-0 and train-1. The number is the rank. Each node is an ordinary h100-8 node. run_command, ports, files, get_node, and hold_node work as on any node. list_nodes and get_node show the cluster and rank of each node.
Every node of a cluster gets these environment variables.
| Variable | Value |
|---|---|
GMN_CLUSTER | The cluster name. |
GMN_CLUSTER_RANK | The rank of this node, from 0 to nodes - 1. |
GMN_CLUSTER_SIZE | The number of nodes. |
GMN_CLUSTER_MASTER_ADDR | The private network address of rank 0. |
NCCL_SOCKET_IFNAME | gmn0, the private network interface. NCCL uses it to start. The data moves over InfiniBand. |
NCCL_IB_HCA | mlx5 |
NCCL uses InfiniBand only when the node has the InfiniBand user libraries. If they are not there, NCCL sends the data over the private network, which is much slower. Install them on each node before you start the job.
for r in 0 1; do
autor run train-$r -- sh -c 'sudo apt-get update && sudo apt-get install -y ibverbs-utils ibverbs-providers'
donerun_command("train-0", "sudo apt-get update && sudo apt-get install -y ibverbs-utils ibverbs-providers")
run_command("train-1", "sudo apt-get update && sudo apt-get install -y ibverbs-utils ibverbs-providers")Start the same command on every node. This example runs one torchrun on each node of a two-node cluster.
for r in 0 1; do
autor run train-$r -d -- sh -c 'torchrun \
--nnodes $GMN_CLUSTER_SIZE --node-rank $GMN_CLUSTER_RANK \
--nproc-per-node 8 \
--master-addr $GMN_CLUSTER_MASTER_ADDR --master-port 29500 \
train.py'
donerun_command("train-0", "torchrun --nnodes $GMN_CLUSTER_SIZE --node-rank $GMN_CLUSTER_RANK --nproc-per-node 8 --master-addr $GMN_CLUSTER_MASTER_ADDR --master-port 29500 train.py", detach: true)
run_command("train-1", "torchrun --nnodes $GMN_CLUSTER_SIZE --node-rank $GMN_CLUSTER_RANK --nproc-per-node 8 --master-addr $GMN_CLUSTER_MASTER_ADDR --master-port 29500 train.py", detach: true)To see the InfiniBand device from a node, run ibv_devinfo. The port state is PORT_ACTIVE. To make sure that NCCL uses InfiniBand, set NCCL_DEBUG=INFO before torchrun. The output then shows NET/IB.
A cluster is all its nodes or none of them. If one node stops, exits for good, or is lost, the whole cluster ends. Then every node stops and every disk is deleted. The failure field says why.
hold_node on each node across a long wait.max_wait, the cluster ends. We do not charge you.autor export train-0 --connection storage/ckpts \
--source ~/out/checkpoints --dest run7/ --wait
autor cluster rm trainexport_data("train-0", connection: "storage/ckpts",
source: "~/out/checkpoints", dest: "run7/")
get_export("train-0", "exp-...") # wait for complete
delete_cluster("train")Deleting a cluster deletes the disks of all its nodes. Save what you need first. delete_cluster stops billing on every node at once and returns ending. The cluster reaches ended in the background.
h100-8 node, per GPU-minute, from the moment it is ready. There is no separate cluster price. rate.cluster_per_min is rate.node_per_min times the number of nodes.delete_cluster, the idle rule, a program that exits, a spend cap or balance stop, and a fault on our side, for example a lost machine. There is no minimum charge. We do not give credit for a run that a fault ended.list_limits shows the limits.Nodes are interactive. Jobs run without you. submit_job builds your image, queues it fairly against your org's share, runs your command, and bills only the running attempt. Queued and preempted time is free.
autor job submit --chip h100 --context ./eval \
--command 'python eval.py' --idempotency-key eval-2026-07-25
→ job_... # the id on stdout; rate and cost ceiling on stderr (--wait blocks to running)submit_job(
chip: "h100",
command: "python eval.py",
context: {"Dockerfile": "FROM pytorch/pytorch\nCOPY eval.py .", "eval.py": "..."},
idempotency_key: "eval-2026-07-25")
→ {job_id: "job_...", rate: "$/min", cost_ceiling: "..."}image ref instead of context to skip the Dockerfile build. The platform still imports the image. It pulls the image and pushes it into the platform registry, and this import uses build_timeout_minutes (default 30, maximum 120). Give a large image several minutes. If the limit stops an import while its pull moves at a normal speed, the import fails with no retry, and its failure_reason tells you what to change.build_note, and get_job repeats it while the wait lasts. Identical contexts share one build.queued. Its build log has no output. The note on that read names the job whose build made the image, when you can see that job.idempotency_key makes resubmission safe. The same key returns the same job.submit_job refuses a new job past this limit and creates nothing. The limit never refuses a resubmit with the same idempotency_key. list_limits shows your value as max_queued_jobs_per_org.$GMN_RESULT_PATH, and put files to keep in $GMN_OUTPUT_DIR. Both come back on get_job when the attempt completes: the result inline, the output directory as a download URL.$GMN_CPU_LIMIT and $GMN_MEMORY_LIMIT_BYTES, like a node. OMP_NUM_THREADS and RAYON_NUM_THREADS are preset to the CPU budget, so nproc, OpenMP, BLAS, torch, and Rust thread pools size to your slice. A value you pass in env wins. This matters most for small-tensor test suites, where a pool sized to the whole host can make tiny operations hundreds of times slower.The queue is built for long waits. Hours queued is normal. A job holds its place for queue_ttl_minutes (default 3 days). Poll minutes apart. Do not resubmit or cancel because the wait is long.
While it waits, get_job reports queue.position, estimated_wait_seconds and queue.wait_reason. The position is the job's place in the dispatch order under org fairness, so it can move both ways. The estimate is measured from recent throughput and the fleet's drain capacity. All three are null until the queue computes them, seconds after the job enters the queue. A held job reports null for the position and the estimate.
wait_reason tells you why the job waits. queue_order gives a number in estimated_wait_seconds. whole_machine gives the time to start a new machine, or null when no new machine is coming for queued jobs. Every other reason gives a null estimate, because the reason does not clear when the jobs ahead of yours finish. A job that is held, is over a cap, has a balance at zero, is past due, or waits on runway also reports a null position. Every other job keeps its position. When a reason has more detail, held_reason explains it in a sentence.
queue_order means that the job waits for its turn or for a free slot, and a machine in the fleet can start it when both come. A null estimate here means that there is no signal yet.whole_machine means that the job needs a whole machine, and each machine that can take it has a node on it. We never stop a node to start a job, so the job waits for a new machine. Your place is held.capacity means that no machine in the fleet can start this job now, for reasons on our side. Your place is held.profiling means that the job asks for GPU profiling, and no machine with the profiling counters can take it now.sole_tenancy means that only machines that serve one tenant each are left for this shape, and the job is too small to take one whole.runway means that no machine has enough time left for max_duration_minutes. A shorter max_duration_minutes or resume: "checkpoint" can let the job start.spend_cap means that the job is held for the spend cap, or that its maximum cost is more than a cap.spend_cap_headroom means that this month's spend and your running jobs leave too little room under the cap.balance means that your prepaid balance does not cover the maximum cost of the job. Add credit (see credit and automatic top-up, or machine payments for an agent that pays for itself). When the balance is above zero, the held_reason sentence names the ways that work for your org.payment means that the account is past due or suspended.gpu_limit means that the running jobs of your org leave too little of the max_running_job_gpus_per_org limit for this job. A job with nodes counts all its GPUs from the moment Autoresearch starts to get its machines.cluster_machines means that Autoresearch gets the machines for a job with nodes. If no machines are free, Autoresearch buys them. You do not pay while the job waits.cluster_network means that the machines of a job with nodes join a private InfiniBand network. This usually takes 10 to 45 minutes. You do not pay while the job waits.autor job get job-d4bx9get_job("job-d4bx9")
→ {status: "queued", start_failures: 0, infra_start_failures: 0,
queue: {position: 1, estimated_wait_seconds: null,
wait_reason: "profiling"},
held_reason: "waiting for a machine with GPU profiling counters. No
machine that has these counters can take this job now.
The job keeps its place in the queue."}start_failures counts the starts that failed because of the image of the job or its reference. After three, the job fails with start_error. infra_start_failures counts the starts that failed on our side, for example because another process still held a GPU. Such a start sends the job back to the queue. It costs nothing and does not count toward the three. While the job waits again, held_reason says start failed (infrastructure, retrying).
Sometimes a machine refuses a start with an error that we cannot connect to the image. If that machine did not refuse the job before, the start counts as ours, two times per job at most. After a refusal, the job tries a different machine first. It goes back to the same machine only when no other machine can take it.
A job backlog that persists for a shape raises the fleet's procurement target for that shape. Submitting the work is how capacity arrives, and cancelling retracts the signal. Submit the whole shape of the work at once. Eight jobs queued is a truer signal than one job queued eight times in series. The signal reads a sustained backlog, so give it an hour rather than a minute. See Queue.
A job can wait for another job. Pass depends_on with the parent job id and, if you want one, a condition over what the parent declared at $GMN_RESULT_PATH. The dependent job does not build, does not queue and never places until the parent succeeds and the condition holds. A cheap probe in front of an expensive run therefore costs nothing when the probe says no.
# the probe; two cards, twenty minutes, writes {"ok": true, "peak_gib": 17.4}
submit_job(chip: "h100", chip_count: 2, context_id: "ctx-...",
command: "python probe.py", idempotency_key: "mem-probe",
mission: "memory-ceiling")
# → {job_id: "job-8x2mf", ...}
# the run; four cards, nine hours, and it starts only if the probe passes
submit_job(chip: "h100", chip_count: 4, context_id: "ctx-...",
command: "python train.py", idempotency_key: "arm-1",
mission: "memory-ceiling",
depends_on: {job: "job-8x2mf", require: "result.ok == true"})
# → {job_id: "job-4kq7p", status: "submitted",
# depends_on: {job: "job-8x2mf", require: "result.ok == true",
# state: "waiting"}}require is one comparison. Write result, then an optional dotted path, then ==, !=, <, <=, > or >=, then a literal. "result.ok == true" and "result.metrics.peak_gib < 72" are both valid. Leave it out to wait on the parent succeeding alone.canceled and failure_reason says why, down to the value the condition read. It built nothing and ran nothing, so it cost nothing.allow_different_context: true inside depends_on.get_job reports depends_on.state as waiting, satisfied, or closed, so a polling agent can tell a gate from a queue. The waiting job's own queue_ttl_minutes still bounds how long it waits, and a job whose TTL runs out before the parent finishes reads expired_waiting rather than closed, because the gate never decided.submit_jobs takes the same declaration on shared, so one probe can gate a whole sweep.autor job get job_... # status; autor job logs job_... -f follows
autor job get job_... --no-links # status with no download URL in the answer
autor job ls --label lr-sweep # every job under the label
autor job cancel job_... # immediate; queued/building stops freeget_job("job_...") # status; add stream:"run" for logs
get_job("job_...", links: false) # status with no download URL in the answer
list_jobs(label: "lr-sweep", summary: true) # rollup with every declared result
cancel_job("job_...") # immediate; queued/building stops freestream: "run" to read a job's output as it goes. Add attempt to read an earlier attempt after a preemption.links: false to leave the download URLs out of the answer. A download URL is a credential for one file. The output URL lives 7 days and the preview URL 15 minutes. Do not print either into a log or a transcript. Poll artifact.state without them, and call get_artifact from the job's workspace for a URL when you download. In the CLI, pass --no-links.attempt_state says whether to read again. "not_started": still building, queued, or starting. Keep polling the job. "live": the attempt is executing. "complete": the attempt ended and the stored output is final. "never_ran": the attempt is over and never got a container, so an empty response is the final answer. "unknown": an error on our side stopped the read. That response has no output and keeps next_offset. Read the log again in a few seconds.eof: true means you reached the end of a stream that nothing is writing. Stop polling.host_gpu_health appears on a live attempt only when the machine under it has a problem, such as a GPU that stopped computing correctly. You do not have to act on it. An attempt cut short by such a device does not bill, draws on the restart budget, and re-places on a healthy host. The field lets a polling agent tell our hardware apart from its own bug.receipt.image_digest is the manifest digest of the image we resolved and pinned for the run. Use it to tie a result to the exact bytes that produced it. receipt.billed_seconds and receipt.charged_usd_micros are the totals. receipt.attempts holds one row per billed attempt, each with the rate it billed at. All of these come from the same billing records your invoice totals to. Billed seconds are the seconds for one GPU. The charge is seconds x gpu_count x rate / 60. An attempt we ended, such as a preemption, bills only the progress that survived. It bills nothing unless a checkpoint capture or a task report landed inside that attempt's own window. If one did, the attempt bills up to that mark. A job with nodes bills each attempt that started to its end. An attempt with no row was free.A check declares what your job must do. The platform validates the declaration when you submit, and then applies it. On the MCP interface every job must declare at least one check.
autor job submit --chip h100 --context ./eval --command 'python eval.py' \
--checks '{"success":{"check":{"all_of":[
{"type":"file_exists","path":"/output/rows.jsonl"},
{"type":"jsonl_field","path":"/output/rows.jsonl","field":"logprob_mean","require":"nonnull"}]}}}'submit_job(
chip: "h100",
command: "python eval.py",
image: "my-registry/eval:v3",
idempotency_key: "eval-2026-09-02",
checks: {success: {check: {all_of: [
{type: "file_exists", path: "/output/rows.jsonl"},
{type: "jsonl_field", path: "/output/rows.jsonl",
field: "logprob_mean", require: "nonnull"}]}}})A success check runs one time, when the job stops. If it passes, the job gets the succeeded state. If it fails, the job gets the check_failed state, and the result is on get_job. A downstream reader cannot take an unusable file from a succeeded job.
Start with a success check. It runs one time, it has no interval to configure, and it cannot stop a job that works.
A health check runs on an interval while the job is live. It finds the failure a loss curve hides, such as a training run that writes logs and changes no weights.
autor job submit --chip h100 --image my-registry/train:v7 \
--command 'python train.py' --max-duration 8h --checks ./checks.json
# checks.json:
# {"health": {"check": {"type": "tensor_changed",
# "path": "/checkpoint/adapter_model.bin"},
# "interval_s": 600, "grace_period_s": 900,
# "fail_threshold": 2, "on_fail": "kill"}}submit_job(
chip: "h100", image: "my-registry/train:v7",
command: "python train.py", max_duration_minutes: 480,
resume: "checkpoint", idempotency_key: "train-v7",
checks: {health: {
check: {type: "tensor_changed", path: "/checkpoint/adapter_model.bin"},
interval_s: 600, grace_period_s: 900,
fail_threshold: 2, on_fail: "kill"}})interval_s is the time between checks, from 30 to 3600 seconds. The default is 300.grace_period_s is the time at the start of the attempt when no check counts. Set it above your start time. The default is 300.fail_threshold is the number of consecutive failures before the platform acts. The default is 2.on_fail is required. Use record to write the failure to the job record and do nothing more. Use kill to stop the job and close the meter. Use reschedule to stop the job and run it again on different hardware.Before a kill or a reschedule, the platform captures your checkpoint slot. Write your state to $GMN_CHECKPOINT_DIR and pass resume: "checkpoint". A check that stops the job then costs you the time to resume. A job that writes no checkpoint has nothing to capture, and it loses the run.
The push pattern is the shortest health check to add. Your loop writes a file, and the platform confirms the file changed.
# one line per step in your training loop
Path("/output/heartbeat").touch(){"health": {"check": {"type": "heartbeat", "path": "/output/heartbeat"},
"interval_s": 600, "on_fail": "kill"}}A check is one type, or a group of types under all_of or any_of. Groups hold groups, to four levels. Each phase takes at most 16 checks.
file_exists passes if the path is present. Both phases.file_size passes if the size is between min_bytes and max_bytes. Both phases.jsonl_field passes if the field satisfies require in every row. require is nonnull, present, finite, or nonempty. Both phases.exec passes if your command exits 0. Both phases.file_grows passes if the file or the directory is larger than at the last check. Health only.jsonl_rate passes if the file gained min_per_interval rows since the last check. Health only.heartbeat passes if the path changed since the last check. Health only.tensor_changed passes if the bytes at the path differ from the last check. Health only.gpu_util passes if every GPU is at min_pct or above. Health only.gemm_sane passes if every GPU computes a finite bf16 matrix multiplication. Use it with on_fail: "reschedule" to move off a bad GPU. Health only.http passes if GET :port/path returns a 2xx status. Health only.The platform runs each check from outside your container. You install nothing in your image, and a check works when your job's own environment is the part that is broken.
Use exec only when no other type can state your assertion. Exit code 1 means your check ran and the assertion is false. A command that cannot run, such as a missing binary, is a different result: the platform records it and takes no other action. An exec command that can only pass, such as true, is refused when you submit.
A failed exec check records the last 400 characters that your command printed. This is your own output, so your data policy controls it: if your organization turns off command output, the platform records the result and a note in place of the output. The check still runs and on_fail still applies.
The first check of an attempt has nothing to compare against, so a file_grows, heartbeat, jsonl_rate, or tensor_changed check passes and records the first measurement.
get_job reports the declaration, the number of checks that ran, the last result, and the failures. A job that a health check stopped gets the self_terminated state. A job whose success check could not run gets the check_errored state. Your checks continue when we update the software on a host during your job, and the success check runs when the job ends.
The job_scratch_bytes limit caps the container's writable layer. The default is 100 GiB. Files that your job writes into the image's own paths count against it, and so do model weights that it downloads without hf_cache.
$GMN_CHECKPOINT_DIR, $GMN_OUTPUT_DIR, and the hf_cache mount are outside the quota.ENOSPC. Size your working set to the quota, or ask for more.autor limits request job_scratch_bytes 322122547200 \
--reason "8xH100 training: 179 GiB dataset plus working copies"request_limit_increase(limit: "job_scratch_bytes", value: 322122547200,
reason: "8xH100 training: 179 GiB dataset plus working copies")Preflight runs on every submission. It adds about 2 seconds and it is free. It has two checks, and each one is quiet when the data is not sufficient.
The cost check multiplies the node rate by the job timeout. If the result is more than the money that remains under your workspace cap or your organization cap, the platform refuses the job and shows the arithmetic. Without the refusal, the cap stops the same job in the middle of the run, after you spend the money. To submit a job that stops early, pass force.
BLOCK preflight/cost-cap: worst case $214.00 (h100-8 x 420 min) exceeds
workspace remaining cap $150.00
Override: force: true (this submission only) - raise the cap: set_workspace_capThe disk check identifies the model your job downloads, resolves its size, and compares the size with the job's disk (see Job disk). If the model with a 10% margin is larger, preflight warns and submits the job. The warning states the model bytes, the bytes it needs, and the limit that caps the disk. Declare model to make the check exact. Otherwise the platform reads a --model flag, a model name in env, or a from_pretrained value from your command. If the platform cannot identify or size the model, the check says nothing.
autor job validate --chip h100 --count 8 --max-duration 7h \
--model Qwen/Qwen3-VL-72B --command 'vllm serve' --checks ./checks.json
→ ok preflight/cost-cap: worst case $214.00 fits the organization remaining cap $900.00
→ WARN preflight/disk-model: Qwen/Qwen3-VL-72B resolves to ~145 GiB (revision main,
31 weight files). With a 10% margin it needs about 160 GiB of disk. The job's
writable layer holds 100 GiB (job_scratch_bytes, 107,374,182,400 bytes).
The job can run out of disk. ...validate_job(
chip: "h100", chip_count: 8, max_duration_minutes: 420,
model: "Qwen/Qwen3-VL-72B", command: "vllm serve",
checks: {health: {check: {type: "gemm_sane"}, on_fail: "reschedule"}})
→ {preflight: [...], would_submit: true}validate_job submits nothing and costs nothing. Use it in a loop while you write a job spec.force submits a job that a preflight check blocks. The platform records the override on the job.no_preflight skips both checks. The job record then says that it skipped them, so a later postmortem knows which checks did not run.Every job gets a durable checkpoint slot. $GMN_CHECKPOINT_DIR points at a directory on the job's scratch disk. Write a consistent checkpoint there, then declare it with one credential-free call to the metadata service. The platform captures it, encrypted and checksummed, beside your running process.
# 1. write a consistent checkpoint into $GMN_CHECKPOINT_DIR, then declare it
curl -s -X POST -H 'Metadata-Flavor: givemeanode' "$GMN_METADATA_URL/v1/checkpoint"
# → 202 {"status": "accepted"} ("coalesced" if one is already in flight;
# retry a transient 503 in the first seconds of a run)
# 2. poll until the capture COMMITS
curl -s -H 'Metadata-Flavor: givemeanode' "$GMN_METADATA_URL/v1/job"
# → ... "checkpointed_at": "2026-07-30T23:57:12Z" ...
# 3. optional, for qualification: end this attempt and get it back
curl -s -X POST -H 'Metadata-Flavor: givemeanode' "$GMN_METADATA_URL/v1/yield"
# → 202 {"status": "yielding", "attempt": 1, "checkpointed_at": "..."}
# 4. the requeued attempt starts with $GMN_CHECKPOINT_DIR non-empty:
# recompute your checksum there and compare across the two attempt logs
# (get_job with attempt: 1, then attempt: 2)checkpoint_slot_bytes limit sets that maximum. $GMN_CHECKPOINT_DIR is outside the job_scratch_bytes quota (see Job disk).checkpointed_at appears or advances on GET /v1/job and on get_job, next to checkpoint_bytes.Retry-After header. Nothing failed, and the last committed capture stands.$GMN_CHECKPOINT_DIR before your process starts. The resume contract is one line: if $GMN_CHECKPOINT_DIR is not empty, resume from it.max_restarts. Provider faults (worker death, allocation loss, GPU failure) draw on that budget, which is 1 by default and up to 3. A job past it fails.resume: "checkpoint". Rank 0 saves the checkpoint, and every machine restores it.resume: "checkpoint" for resumable placement. The job can be scheduled onto capacity windows shorter than its max_duration. A window lapse is a planned sever: your container gets a machine_retirement countdown on /v1/events, and the interruption bills only up to your last committed capture. A resumable job severed twice without a capture loses the relaxed placement, so checkpoint early. Once one job has checkpointed in your org, later jobs with the same image and the same command inherit resume: "checkpoint". get_job shows resume_source: "inferred" and the job it learned from. Pass resume: "none" to run a job as one attempt.POST /v1/yield proves restoration on demand. It refuses with 409 when no capture has committed, while a capture is in flight, or when the budget of 2 yields per job is spent. It refuses with 429 under the one-per-5-minutes limit. A yielded attempt bills its full span. The requeued job keeps its queue standing, can land on a different machine, and gets its own max_duration clock. get_job reports yields_used beside preempted_count.get_job always shows attempt_history. The list has one row for each attempt that ended, and it is empty until the first attempt ends. Each row has the attempt's start and end, its outcome (completed: your job ended it, and it bills; severed: we ended it), and its end_cause, for example preempt, worker_lost, spot_reclaim, or process_exit.end_reason gives more detail about an attempt that we ended. For preempt, it tells you why we needed the capacity. preempted_interactive means that an interactive node needed it. preempted_whole_node_hold means that we kept a whole machine empty for a queued 8-GPU job. For a fault on our side, it gives the class of the fault: provider_fault_host_lost, provider_fault_allocation_lost, provider_fault_host_expired, provider_fault_gpu, spot_reclaimed, or allocation_boundary. It is null when your job ended the attempt, and for a preemption before October 9, 2026.build_history shows each build that failed under the job. free: true means the fault was ours, and the image builds again at no charge. While the image builds again, build.retry_count gives the number of free rebuilds, and build.retry_reason tells you why the last build failed.$GMN_OUTPUT_DIR, or push it to org storage with the shard-and-manifest pattern.A job can run on 2 to 8 whole machines with 8 H100 GPUs each. Set nodes to the number of machines. Autoresearch puts the machines on a private InfiniBand network for this job only. Each machine runs one copy of your command with 8 GPUs, and the copies move data to each other over InfiniBand.
This feature is in preview, and we turn it on for one account at a time. To ask for it, use file_ticket. Until it is on, a submit with nodes fails with a sentence that says so.
autor job submit --chip h100 --count 8 --nodes 2 --resume checkpoint \
--context ./train --mission llama-pretrain --max-duration 2h \
--checks '{"success":{"check":{"type":"file_exists","path":"/output/model.safetensors"}}}' \
--command 'torchrun --nnodes $NNODES --node-rank $NODE_RANK --nproc-per-node 8 --master-addr $MASTER_ADDR --master-port $MASTER_PORT train.py'submit_job(chip: "h100", chip_count: 8, nodes: 2, resume: "checkpoint",
context: {"Dockerfile": "FROM nvcr.io/nvidia/pytorch:25.01-py3\nCOPY train.py /app/train.py\nWORKDIR /app",
"train.py": "..."},
command: "torchrun --nnodes $NNODES --node-rank $NODE_RANK --nproc-per-node 8 --master-addr $MASTER_ADDR --master-port $MASTER_PORT train.py",
max_duration_minutes: 120, mission: "llama-pretrain",
idempotency_key: "pretrain-2x8-v1",
checks: {"success": {"check": {"type": "file_exists", "path": "/output/model.safetensors"}}})The machine with NODE_RANK 0 is rank 0. In torchrun, the process with global rank 0 runs on it. Every machine loads the checkpoint at start. Rank 0 alone saves it.
ckpt = os.path.join(os.environ["GMN_CHECKPOINT_DIR"], "state.pt")
# at start, on every machine: resume when the checkpoint exists
if os.path.exists(ckpt):
model.load_state_dict(torch.load(ckpt, map_location="cpu"))
# every N steps, on global rank 0 only: save the whole checkpoint, then declare it
if dist.get_rank() == 0:
torch.save(model.state_dict(), ckpt)
requests.post(f"{os.environ['GMN_METADATA_URL']}/v1/checkpoint",
headers={"Metadata-Flavor": "givemeanode"})A job with nodes has these rules.
chip to h100. Set chip_count to 8, or omit it.nodes from 2 to 8. Your account limit can be lower. list_limits shows it as max_cluster_nodes.resume to "checkpoint". A submit without it fails.command. Each machine runs the same command.max_duration_minutes to 1200 or less, also when your account allows longer jobs.nodes cannot use task sets, profiling, yield, or a check with on_fail: "reschedule". Use on_fail: "kill" or on_fail: "record".submit_jobs does not accept nodes. Submit each job that uses more than one machine with submit_job.validate_job takes nodes and resume, and it applies the same rules with no cost.Each machine gets these environment variables.
| Variable | Value |
|---|---|
NODE_RANK | The rank of this machine, from 0 to nodes - 1. |
NNODES | The number of machines. |
MASTER_ADDR | 100.64.0.1, the private network address of rank 0. |
MASTER_PORT | 29500. You can set a different port in env. |
GMN_CLUSTER | The job id. |
GMN_CLUSTER_RANK | The rank, as in NODE_RANK. The nodes of a cluster use the GMN_CLUSTER names too, so one script works in both. |
GMN_CLUSTER_SIZE | The number of machines, as in NNODES. |
GMN_CLUSTER_MASTER_ADDR | The address of rank 0, as in MASTER_ADDR. |
NCCL_SOCKET_IFNAME | gmn0, the private network interface. NCCL uses it to start. GLOO_SOCKET_IFNAME has the same value. |
NCCL_IB_HCA | mlx5 |
NCCL_NET | IB. NCCL moves the data over InfiniBand. If it cannot, it stops with an error. |
MASTER_PORT and the four NCCL_ and GLOO_ variables in env. Your value wins.NODE_RANK, NNODES, MASTER_ADDR, or a GMN_CLUSTER variable in env. Autoresearch sets them, and a submit that sets one fails.WORLD_SIZE or RANK. torchrun sets them for each process.libibverbs and the mlx5 provider. The NGC PyTorch images (nvcr.io/nvidia/pytorch) contain them. A job container cannot use sudo, so install the libraries in your Dockerfile. If they are not there, NCCL stops at start, and the log of each machine shows the error.The job waits in the queue in the same way as other jobs. When its turn comes, Autoresearch gets the machines. If no machines are free, Autoresearch buys them. Then the machines join the private InfiniBand network. This usually takes 10 to 45 minutes. You do not pay while the job waits.
wait_reason is cluster_machines while Autoresearch gets the machines. It is cluster_network while the machines join the network. Before its turn, the job shows the same reasons as other jobs (see why a job waits).queue.position and estimated_wait_seconds are always null.queue_ttl_minutes counts while the job waits for its turn and for machines. It stops while the machines join the network and the job starts. After a retry, it counts again from the moment the job is queued again.max_running_job_gpus_per_org.Every machine waits until all the machines are on the network. Then every machine starts your command at the same time. Rank 0 decides when the attempt ends.
max_restarts. When none are left, the job fails.cancel_job stops every machine. A cancel while the machines prepare costs nothing.Rank 0 keeps the results of the job.
$GMN_RESULT_PATH and $GMN_OUTPUT_DIR, so code that writes them on every machine works. Only the files of rank 0 come back on get_job. output_to copies the output of rank 0.get_job shows the run log of rank 0. Add rank to read the log of a different machine.get_job shows nodes and gpus in chip. The cluster.ranks list shows each machine: its rank, its state, its start time, and its exit code. The state is starting, running, exited, stopped, or lost.autor job get job-8x2mf
autor job logs job-8x2mf --rank 1get_job("job-8x2mf")
→ {status: "running", chip: {model: "h100", count: 8, nodes: 2, gpus: 16},
cluster: {state: "running", ranks: [
{rank: 0, state: "running", started_at: "2026-10-06T20:41:09Z", exit_code: null},
{rank: 1, state: "running", started_at: "2026-10-06T20:41:08Z", exit_code: null}]}}
get_job("job-8x2mf", stream: "run", rank: 1) # the log of machine 1A job with nodes keeps one checkpoint slot, as every job does.
$GMN_CHECKPOINT_DIR. Then rank 0 calls POST $GMN_METADATA_URL/v1/checkpoint. On a different machine, the call answers 409.list_limits shows the slot size as checkpoint_slot_bytes.$GMN_CHECKPOINT_DIR before your command starts.$GMN_CHECKPOINT_DIR, and that try bills as usual.max_restarts. This is also true after a checkpoint commits.GET $GMN_METADATA_URL/v1/job shows a cluster block with the rank, the size, and the master_addr.An attempt bills every GPU on every machine. Billing starts when every machine starts your command, and it stops when the attempt ends.
h100-8 node. There is no job discount. A promotional discount on nodes does not apply. The Friday and weekend discounts apply as they do for a node.max_cost_usd is the cost of all the GPUs for max_duration_minutes. It applies to one attempt. Each new attempt can bill up to that amount again.max_restarts. So at most max_restarts + 1 attempts bill. Together they cost at most max_restarts + 1 times max_cost_usd.A preview is a look at a running job. Your process writes something small (a sample grid, a metrics file) into $GMN_OUTPUT_DIR and declares it. The platform captures it into the job's one preview slot, and get_job returns a short-lived download link. Look at it and decide: keep going, resubmit with new settings, or cancel_job now.
# from inside the job, once the file is fully written:
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
-H 'Content-Type: application/json' \
-d '{"path": "/output/samples/step-500.png"}' \
"$GMN_METADATA_URL/v1/preview"
# → 202 {"status": "accepted"} ("coalesced" if a capture is already running)
# from your agent: poll get_job and fetch the peek
get_job("job-8x2mf")
# → ... "preview": {"captured_at": "...", "size_bytes": 51200,
# "download": {"url": "https://...", "expires_at": "..."}} ...$GMN_OUTPUT_DIR: a regular file or a small directory, which arrives as a tar. Declare it only after the bytes are fully written.preview.error on get_job. Your run continues, and the previous preview stays downloadable.attempt that took it. After a preemption, check it before you judge the current run by an earlier run's preview.A sample is one small file a running job adds to a timeline. Previews replace. Samples append. A training job that saves a sample grid every N steps builds a series you can scroll through. If the job belongs to a mission, the mission page gets a Samples tab that renders the timeline.
# from inside the job, once the file is fully written:
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
-H 'Content-Type: application/json' \
-d '{"path": "/output/samples/step-500.png", "kind": "image", "label": "step 500"}' \
"$GMN_METADATA_URL/v1/sample"
# → 202 {"status": "accepted"}
# from your agent: list the series, newest first (by job, or mission: for the whole mission)
list_samples(job: "job-8x2mf")
# → {"samples": [{"kind": "image", "label": "step 500", "captured_at": "...",
# "download": {"url": "https://...", "expires_at": "..."}}, ...], "next": "..."}kind is one of image, audio, video, text, or json. Every kind renders on the mission page. If a sample cannot be shown, the tile offers the download.label is the caption, up to 120 characters./v1/preview instead.$GMN_OUTPUT_DIR. Declare it after the bytes are fully written.samples.error on get_job and never ends your run. A job keeps up to 256 samples. Past that, captures are refused with the reason in the same place.list_samples returns live for minutes. Fetch what you list, and re-list for fresh links.A sweep is up to 256 variants of one job in one call: one build, N runs.
autor api post /jobs/sweeps --data '{
"shared": {"chip": "h100", "context": {...}},
"variants": [{"env": {"LR": "1e-4"}}, {"env": {"LR": "3e-4"}}, {"env": {"LR": "1e-3"}}],
"label": "lr-sweep",
"idempotency_key": "lr-sweep-1"}'submit_jobs(
shared: {chip: "h100", context: {...}},
variants: [{env: {LR: "1e-4"}}, {env: {LR: "3e-4"}}, {env: {LR: "1e-3"}}],
label: "lr-sweep",
idempotency_key: "lr-sweep-1")Validation is all or nothing. list_jobs(label: ..., summary: true) is the comparison table when the variants finish.
Every variant counts against the queued-jobs limit. If the new variants would take your org past the limit, submit_jobs submits none of them. The refusal says how many new jobs fit now.
A job can carry a list of tasks instead of one command. Your command runs once per machine and asks for work as it goes. One container start and one image pull cover thousands of tasks. A lost machine costs the tasks in flight, not the whole run.
autor job submit --chip h100 --context ./render \
--command 'python render.py' --tasks ./prompts.jsonl --task-runners 8 \
--label nightly-render --idempotency-key nightly-render-1submit_job(
chip: "h100",
context: {...},
command: "python render.py",
tasks: [{prompt: "a red bicycle"}, {prompt: "a blue door"}, ...],
label: "nightly-render",
idempotency_key: "nightly-render-1")Your command leases tasks from inside the container. There is no credential, because the platform knows which job you are.
# this runs INSIDE your container, not from your shellimport json, os, time, urllib.request
META = os.environ["GMN_METADATA_URL"]
def post(path, body=None):
req = urllib.request.Request(
META + path, method="POST",
data=json.dumps(body).encode() if body else None,
headers={"Metadata-Flavor": "givemeanode",
"Content-Type": "application/json"})
with urllib.request.urlopen(req) as r:
return r.status, json.loads(r.read() or "{}")
while True:
status, body = post("/v1/tasks/lease", {"count": 4})
if status == 204:
break # every task is done and the set is sealed
if not body["tasks"]:
time.sleep(5) # the set is open, nothing claimable yet
continue
for task in body["tasks"]:
data = render(task["params"]["prompt"])
urllib.request.urlopen(
urllib.request.Request(task["output_put_url"], method="PUT", data=data))
post(f"/v1/tasks/{task['id']}/done")output_put_url in the lease. Report done only after the upload returns. A task can run more than once, and a done report before the bytes land is the one way to get a wrong answer. Each lease has its own URL, so two runs of one task never overwrite each other. The set keeps the output of the run whose done report we accept.503. The metadata service is briefly unavailable while your container starts, so the first poll of a run often gets one.POST /v1/tasks/{id}/extend before the deadline, or raise task_lease_minutes at submit.POST /v1/tasks/{id}/fail. A task is tried up to task_max_attempts times (3 by default), then recorded as failed. A failed task does not fail the job. get_job lists which tasks failed and why. If you get a 409 that says you no longer hold a task, drop the result and lease again.task_max_runners sets the ceiling. It is 4 by default and can go to 32. A set of 20 tasks never starts 32 machines. The ceiling bounds spend and reserves nothing.autor job get job-8x2mfget_job(job_id: "job-8x2mf")
{"state": "running",
"tasks": {"total": 5000, "done": 4193, "failed": 2, "leased": 8,
"pending": 797, "drained": false, "runners": 4, "max_runners": 8}}state is the result of your own container, not of the set. When your runner loop exits 0, the job reads succeeded, but other machines can still hold leased tasks. drained: true means that the set is sealed and every task is done or failed. Read drained to know that the set is finished.outputs/<task_id>, and manifest.json lists every done task and where its output is. It shows on get_job as tasks.outputs, with a download link, a few minutes after the last machine on the set stops.get_job shows the reason. For a set that large, write each output to your own storage from the task.tasks.outputs for the full set of results. The artifact of a run holds only the files that one machine left in its own $GMN_OUTPUT_DIR.If you do not know every task up front, pass tasks_open: true at submit and add tasks with add_tasks. Your runner keeps leasing while the set is open. The job finishes when you call seal_tasks and the pool drains.
autor job submit --chip h100 --context ./render --command 'python render.py' \
--tasks ./first-batch.jsonl --tasks-open \
--label stream-render --idempotency-key stream-render-1
autor job add-tasks job-8x2mf --tasks ./next-batch.jsonl
autor job seal-tasks job-8x2mf
autor job tasks job-8x2mf # progress, and the failed tasks with their reasonssubmit_job(chip: "h100", context: {...}, command: "python render.py",
tasks: [{prompt: "the first batch"}], tasks_open: true,
label: "stream-render", idempotency_key: "stream-render-1")
add_tasks(job_id: "job-8x2mf", tasks: [{prompt: "..."}, ...])
seal_tasks(job_id: "job-8x2mf")A sweep over one base model does not need to download it N times. hf_cache: true on submit_job, or once in a sweep's shared, mounts a cache shared by your org's jobs on the same machine and points HF_HUB_CACHE at it. The first job on each machine pays the download. The rest load the weights from local disk. It is free, off by default, and never shared across orgs.
autor job submit --chip h100 --context ./train --hf-cache \
--command 'python train_lora.py' --idempotency-key flux-r16submit_jobs(
shared: {chip: "h100", context: {...}, hf_cache: true},
variants: [{env: {RANK: "4"}}, {env: {RANK: "8"}}, ...],
label: "flux-rank-sweep",
idempotency_key: "flux-rank-sweep-1")HF_HUB_CACHE in env alongside hf_cache. The submit is refused, because the platform sets that variable.env as usual.Inline contexts cap at 4 MiB. Past that, presign an upload:
autor job submit --chip h100 --context ./ctx --command '...'
# one command: tars ./ctx deterministically, uploads (create_context →
# finalize_context), then submits - any size up to 10 GiB. A pre-built
# tar.zst rides the same dance by hand via autor api post /contexts.create_context(sha256: "...", size_bytes: 123456789) → upload URL
# PUT the tar to the URL, then:
finalize_context(context_id)
submit_job(chip: "h100", command: "...", context_id: "...", idempotency_key: "...")The same upload can land on a node. Use this for a dataset or eval set that exists only in your sandbox: too big for write_file, and not fetchable from inside the node.
autor api get /contexts/ctx_... # → download URL
autor run tulip -d -- curl -o ~/data.tar.zst '<url>'get_context(context_id) → download URL
run_command(node, "curl -o ~/data.tar.zst '<url>'", detach: true)fly.storage.tigris.dev, not the API hostname. If your sandbox filters egress, see Allowlists.list_contexts shows what you have uploaded: ids, sizes, and when each stops being fetchable.To end an upload early, delete it:
autor api get /contexts # → your uploads
autor api delete /contexts/ctx_... # → status: deleted
autor api get /contexts/ctx_... # → bytes_deleted_at, once confirmedlist_contexts()
delete_context(context_id) → status: "deleted"
get_context(context_id) → bytes_deleted_at (the confirmation)get_context gains bytes_deleted_at once the store confirms it.A sandbox is a small isolated virtual machine that runs shell commands for your agent. Code your model wrote can run and fail in one without touching anything you care about.
autor sandbox create
# → sbx-0a1b2c3d4e5f, plus where it came from
autor sandbox exec sbx-0a1b2c3d4e5f -- python3 -c 'print(6*7)'
# → 42, and the command's own exit code becomes autor's
autor sandbox rm sbx-0a1b2c3d4e5fcreate_sandbox()
sandbox_exec(sandbox: "sbx-0a1b2c3d4e5f", cmd: "python3 -c 'print(6*7)'")
delete_sandbox("sbx-0a1b2c3d4e5f")reused_snapshot. A setup command makes every create boot again, because the snapshot has to carry your setup. For more than one sample, fork.pooled: true, and billing starts when you receive it.shared_base: true. A pooled response carries it too. The base belongs to us, so the response carries no snapshot for it, and there is nothing for you to fork. To make many copies at once, bake your own snapshot.Bake the environment once. Fork it once per sample. Setup runs one time in the bake, and each fork resumes that exact state, memory and running processes included. Forks are free.
autor sandbox bake --image sbx-task --ram-gib 2 --warm-file ./setup.sh
# setup.sh writes /workspace/grade.py and imports numpy, so every fork has both
# → env-1f4c2ab90d3e, baked in 41.2s
autor sandbox fork env-1f4c2ab90d3e --count 64
# → 64 sandbox ids, one per line, ready in 690ms
autor sandbox exec --batch rollouts.jsonl
# one line per sandbox, each with its own seed:
# {"sandbox": "sbx-0a1b2c3d4e5f", "cmd": "python3 /workspace/grade.py 0"}
# → results in request order: exit_code, stdout, stderr, duration_ms
autor sandbox rm sbx-0a1b2c3d4e5f sbx-6a7b8c9d0e1f
# → deleted 2 of 2, the whole batch in one callcreate_sandbox_env(image: "sbx-task", ram_gib: 2,
warm: "mkdir -p /workspace && ...write grade.py... && python3 -c 'import numpy'")
fork_sandbox(from: "env-1f4c2ab90d3e", count: 64)
sandbox_exec(execs: [
{sandbox: "sbx-0a1b2c3d4e5f", cmd: "python3 /workspace/grade.py 0"},
{sandbox: "sbx-6a7b8c9d0e1f", cmd: "python3 /workspace/grade.py 1"}])
delete_sandbox(sandboxes: ["sbx-0a1b2c3d4e5f", "sbx-6a7b8c9d0e1f"])prefix_note that names the snapshot you could have forked.deleted: true or false. A false entry is a sandbox that is still running and still billing, because the id was not yours or the host refused the teardown. The call still succeeds. Check the entries, not only the status. autor sandbox rm reads them for you and exits nonzero when any entry came back undeleted.| Image | Carries |
|---|---|
sbx-base (default) | python3 and pip, Node.js 24 and npm, git, curl, and a C compiler |
sbx-min | bash and coreutils, nothing else |
sbx-task | python3 with numpy |
Name an image we do not have, and the error lists the ones we do. You can also bake from a container image of your own.
A sandbox holds 2 GiB of RAM by default. A workspace starts with an 8 GiB ceiling. An org admin can ask for more with request_limit_increase, up to the 64 GiB platform maximum. list_limits shows yours. Every fork inherits the size its snapshot was baked at.
The default sandbox has one vCPU, which fits an agent that does one thing at a time. A parallel build, a dependency install, or a typecheck across a monorepo needs more. size names a bigger machine.
| Size | vCPU | RAM | Bills as | Network ceiling |
|---|---|---|---|---|
sandbox-sm (default) | 1 | 2 GiB | 2 GiB | 1 Gbps |
sandbox-md | 4 | 8 GiB | 16 GiB | 2 Gbps |
sandbox-lg | 8 | 32 GiB | 32 GiB | 5 Gbps |
sandbox-xl | 16 | 64 GiB | 64 GiB | 10 Gbps |
size and ram_gib are alternatives. Pass one. Your ceiling is the sandbox_vcpus limit, which starts at 4. sandbox-md needs no permission. sandbox-lg and sandbox-xl need a raise.On sandbox-lg and sandbox-xl, /tmp is memory, not disk. Scratch writes and deletes are much faster. The capacity is half the shape's memory: 16 GiB on sandbox-lg and 32 GiB on sandbox-xl. /tmp competes with your process for the same memory, so a job that fills both can be OOM-killed where it would have got “no space left on device”. If you write tens of GiB of scratch, put it under your working directory, which stays on disk. sandbox-sm and sandbox-md keep a disk-backed /tmp.
npm installs use a cache on the machine. It is a pull-through cache of the public registry, shared by everything on that host. Your install is still a real download of every package your lockfile names. You do not configure it. If the cache is unavailable, the sandbox uses registry.npmjs.org directly. We set the default registry only, by writing ~/.npmrc when there is no file there. A home-directory .npmrc of your own is never overwritten. Scoped registry lines are unaffected.
One case to know. Your project can have an auth-only .npmrc: an _authToken line for registry.npmjs.org and no registry= line. npm keys that token to the registry URL it resolves, so it does not send the token to the cache, and the install fails. Add an explicit registry= line to that .npmrc, or put one in ~/.npmrc.
package_cache: true serves npm and bun installs from a cache on the machine instead of from registry.npmjs.org. It is off by default, because it rewrites your lockfile. The cache serves tarballs from a loopback address, so bun records that address for every package. A bun.lock produced under it differs from the one you committed, and a --frozen-lockfile check fails. Turn it on for throwaway installs. Leave it off for anything whose lockfile you keep.
If the environment you want already exists as a container image, bake from the image instead of rebuilding its setup as a warm script.
autor sandbox bake --from-image ghcr.io/acme/task@sha256:8f43...c21a
autor sandbox fork env-7fa2 --count 64create_sandbox_env(from_image: "ghcr.io/acme/task@sha256:8f43...c21a")
→ {snapshot: "env-7fa2", ram_gib: 2}
fork_sandbox(from: "env-7fa2", count: 64)fork_sandbox, sandbox_exec, and snapshots behave the same way.docker buildx imagetools inspect or crane digest. Write the registry host in full.warm.sandbox_converted_images_per_hour). A repeat conversion of the same image does not count.autor connection create acme-ghcr --kind registry \
--scope 'ghcr.io/acme/*' --username acme-bot --material-stdin < token.txt# Registry connections are created from the CLI, so the credential
# never rides an MCP argument.A sandbox can reach the internet. Your task code can git clone, pip install, and call an API. DNS works. A sandbox cannot reach anything private: your nodes, your networks, this API, and every other sandbox are unreachable from inside one. Each sandbox gets up to one gigabit per second.
Inside a sandbox, a server your code starts is reachable at 127.0.0.1 and localhost from another command in the same sandbox. Nothing outside the sandbox can reach it.
Pass egress: "none" for a sandbox with no network at all. Use this when a rollout must be hermetic, or when code your model wrote must have no way out. Everything such a sandbox needs must be baked in.
autor sandbox bake --image sbx-task --egress none --warm-file ./setup.sh
# → env-1f4c2ab90d3e, and every fork of it has no network
autor sandbox create --egress nonecreate_sandbox_env(image: "sbx-task", egress: "none",
warm: "pip install -q -r requirements.txt")
create_sandbox(egress: "none")The choice is made when the env is baked. Every fork inherits it. A fork cannot turn the network on or off. Bake a second env for the other posture. The response says which one you got in egress.
Bake in anything a fork needs on every sample. A bake runs your setup once, and every fork inherits the result. A model or a large dataset per sample is slow to download anywhere. That work belongs on a node or in a batch job, with the sandbox grading what comes back.
A server inside a sandbox can be reached from outside at a public HTTPS URL. This is the link a coding agent hands to a person to look at a dev server.
autor sandbox exec sbx-0a1b2c3d4e5f -- sh -c 'cd app && (npm run dev > /tmp/dev.log 2>&1 &)'
autor sandbox expose sbx-0a1b2c3d4e5f 3000
# → https://sbe-9k2fq-vx7t3m8dk4qwrz2n.givemeanode.io
autor sandbox unexpose sbx-0a1b2c3d4e5f 3000sandbox_exec(sandbox: "sbx-0a1b2c3d4e5f",
cmd: "cd app && (npm run dev > /tmp/dev.log 2>&1 &)")
expose_sandbox_port(sandbox: "sbx-0a1b2c3d4e5f", port: 3000)
unexpose_sandbox_port(sandbox: "sbx-0a1b2c3d4e5f", port: 3000)127.0.0.1, which is where a dev server listens by default.egress: "none" too. Nothing inside has to reach out for the URL to work.npm run dev & holds the exec open until its deadline and then kills the server with it. Redirect the output, as in the example, and the exec returns at once.auth: "bearer". The token is in that response once.list_limits shows how many you can have at a time. Exposing the same port again returns the same URL with a fresh expiry, so ask for the URL whenever you need it.404. A retry does not help. Create a new sandbox, start the server, and expose the port again. A 502 means that the host can still answer. Send the request again after one minute.autor sandbox exec sbx-0a1b2c3d4e5f -- python3 /workspace/run.py --until 278
autor sandbox snapshot sbx-0a1b2c3d4e5f
# → env-77c1d0e9ab24, captured while the sandbox keeps running
autor sandbox fork env-77c1d0e9ab24 --count 8sandbox_exec(sandbox: "sbx-0a1b2c3d4e5f", cmd: "python3 /workspace/run.py --until 278")
snapshot_sandbox("sbx-0a1b2c3d4e5f") # → snapshot: "env-77c1d0e9ab24"
fork_sandbox(from: "env-77c1d0e9ab24", count: 8)
# eight continuations from one identical statesnapshot_sandbox captures a running sandbox without stopping it. Every branch starts from finished work.rehydrate_ms with the wait. Forks after it are fast again.expires_after to create_sandbox_env or snapshot_sandbox, with a value such as "24h" or "7d" and a minimum of one hour. Set or clear it later with set_sandbox_snapshot_expiry. "never" is the default. An expired snapshot deletes itself.delete_sandbox_snapshot ends the storage charge and stops new forks. Sandboxes already forked keep running.{"sandbox": "sbx-0a1b2c3d4e5f",
"outcome": {"outcome": "void", "cause": "uffd_handler_died"},
"billed": false}cancelled: true means your own delete_sandbox won a race against the exec. The verdict is unknowable. The time is billed, and the sandbox is gone.Killed line usually means the sandbox's memory budget. When the OOM killer fired, the result carries an oom_kills count and a note. That is your command's own failure. It bills, it scores as a real low reward, and an unchanged retry dies the same way. Bake with more ram_gib, or use less memory. get_sandbox_stats counts these under oom_execs.get_sandbox_stats finds the forks a loop forgot.sandboxes block in get_usage carries the same days and splits the charge per tier. The CSV export at /usage.csv carries the same rows per workspace and member, marked kind=sandbox.list_audit_events records each create, fork, and delete call as a sandbox.create or sandbox.delete event. A batched call is one event that carries the count. Read these to see what your harness did on the days the usage page names.| Limit | Value |
|---|---|
| Sandboxes at once, per workspace | 4,096 |
| RAM per sandbox | 8 GiB, raised on request as far as 64 GiB |
| Fork depth per lineage | 64 generations |
| Sandboxes per fork call | 256 |
| Commands per exec call | 4,096 |
| Command deadline | 60 seconds by default, 10 minutes at most |
| Output per command | 1 MiB of stdout and 1 MiB of stderr |
| Cold snapshot storage, per organization | 2 TiB |
Commands inside one sandbox run one at a time. Different sandboxes run in parallel. The sandbox endpoints have their own request budget, separate from the rest of the API, so an exec batch across a thousand sandboxes is one request. list_limits shows the values that apply to you. We raise them on request. See Help.
A rollout is one episode of a policy in an environment. It starts from a state, runs under a judge, and ends in a verdict. You bring the environment and one spec for each episode. We record three things for each rollout. We record its identity, computed from what it is. We record its lineage, when it starts from a state that another rollout declared. We record its verdict, with a stamp that says who declared it and from where. We do not read your spec or your state. Rollouts run on the task sets and sandboxes you already use.
The rollout tools are on for every organization. If your agent does not see them in its tool list, reconnect it, then ask with file_ticket.
Three commands take you from one spec to a scored report. This example uses mjx-rollout, a container that we ship. It runs each episode and it declares each verdict, so you write no runner code.
# 1. Submit one spec over eight seeds. A submit never blocks.
autor rollout submit --environment mjx-rollout \
--spec '{"scene": {"builtin": "ball-to-goal"},
"horizon_steps": 100,
"init": {"qvel": [1.2, 0, 0, 0, 0, 0], "qvel_noise": 0.05},
"judge": {"body": "ball", "center": [1, 0, 0.1], "radius": 0.5}}' \
--seeds 0..8 --judge goal-region@v1 \
--evaluation hello-rollouts --idempotency-key hello-rollouts
# → evaluation hello-rollouts: 8 rollouts
# 8 queued as task set job-7c1kd
# 2. Watch. Each row goes from queued to running to scored.
autor rollout ls --evaluation hello-rollouts
# → ID STATUS SEED SCORE PASS JUDGE NOTE CREATED
# ro-4k9f2 scored 0 1.0 yes goal-region@v1 - 2026-09-03T18:04:11
# ro-4k9f3 scored 1 0.0 no goal-region@v1 - 2026-09-03T18:04:11
# ...
# 3. Read the scores. "final": true means each rollout has finished.
autor evaluation report hello-rollouts
# → {"evaluation": "hello-rollouts", "final": true,
# "rollouts": {"total": 8, "scored": 8, "failed": 0, "void": 0, "memoized": 0},
# "environments": [{"all": {"n": 8, "mean_score": 0.625, "pass_rate": 0.625}}],
# ...}submit_rollouts(
environment: "mjx-rollout",
spec: {scene: {builtin: "ball-to-goal"}, horizon_steps: 100,
init: {qvel: [1.2, 0, 0, 0, 0, 0], qvel_noise: 0.05},
judge: {body: "ball", center: [1, 0, 0.1], radius: 0.5}},
seeds: "0..8", judge: "goal-region@v1",
evaluation: "hello-rollouts", idempotency_key: "hello-rollouts")
list_rollouts(evaluation: "hello-rollouts")
get_evaluation_report(evaluation: "hello-rollouts")0..8 gives seeds 0 to 7. See Submit.mjx-rollout reads. Your own environment reads its own fields.eval, so memoization is on. Submit the same eight rollouts again and each one returns its stored verdict. It does not run again and it costs nothing. See Memoization."environment": {"kind": "task", "env": "mjx-rollout"}
"environment": {"kind": "task", "env": "ghcr.io/acme/my-sim@sha256:41c8..."}
"environment": {"kind": "task", "env": "openenv:hub.openenv.dev/acme/swe-mini@sha256:2ab4..."}
"environment": {"kind": "sandbox", "env": "env-8x2mf3a9c1d4"}There are two kinds of environment. A task environment is a container that you run. It starts once on each machine, leases episodes, runs them, and declares their verdicts from inside. Name your own image at a digest, or name one of the two containers we ship. mjx-rollout runs MuJoCo MJX scenes in batches on a GPU and grades them with its goal-region judge. openenv-rollout runs an OpenEnv environment image at a digest. See OpenEnv.
A sandbox environment is a snapshot that you baked with create_sandbox_env. Each rollout is one fork of that snapshot. Your harness drives the fork with sandbox_exec and declares the verdict from outside. Use this kind for code that your model wrote, for software engineering agents, and for tool use.
Name an image at a digest. A tag can move, so we refuse a bare tag. The kind sets where an episode runs and who can declare its verdict. It does not change how you submit, read, or report rollouts.
autor rollout submit \
--environment ghcr.io/acme/mjx-reach@sha256:41c8... \
--specs ./episodes.jsonl --seeds 0..8 \
--judge reach-goal@v1 --profile eval \
--chip h100 --runners 8 \
--evaluation ckpt-eval-9 --mission policy-eval \
--idempotency-key ckpt-eval-9
# → evaluation ckpt-eval-9: 49,152 rollouts
# 12,808 memoized from ckpt-eval-8 (same identity); unbilled, not re-run
# 36,344 queued as job-7c1kdsubmit_rollouts(
environment: "ghcr.io/acme/mjx-reach@sha256:41c8...",
# each line of the manifest: the spec your container reads, plus labels
specs: [{spec: {scene: "reach-v2", body: "hand", target: [0.3, 0.1, 0.4],
checkpoint: "acme/reach-ckpt41"},
labels: {policy: "ckpt41"}}, ...],
seeds: "0..8",
judge: "reach-goal@v1", profile: "eval",
chip: "h100", runners: 8,
evaluation: "ckpt-eval-9", mission: "policy-eval",
idempotency_key: "ckpt-eval-9")A submit makes one rollout for each spec and each seed, under one judge. A spec is a JSON value. We give it to your environment unchanged. The seed range 0..8 gives seeds 0 to 7. In a --specs file, each line is a spec, or {"spec": ..., "labels": {...}, "seed": n} when a spec needs its own labels. Labels group rows in a report. They do not change the identity of a rollout.
The profile sets the defaults. eval is the default. It turns memoization on and stamps each verdict with the reproducibility class you declared. Use it for benchmarks and regression suites. train turns memoization off. A memoized training sample reduces the diversity of your samples, so we refuse train with memoization on. You can set each default yourself.
Queued time is free. A backlog of rollouts is one of the inputs that starts more machines, so do not resubmit to go faster. Give each submit an idempotency key. A retry with the same key returns the first response and does not run a second set.
# lease: each task carries a rollout block; specs arrive once, by hash
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
-d '{"count": 32}' "$GMN_METADATA_URL/v1/tasks/lease"
# declare: one call per batch; a verdict completes its task
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
-d '{"verdicts": [{"rollout": "ro-4k9f2", "score": 1.0, "pass": true, "basis": "environment"},
{"rollout": "ro-4k9f3", "score": 0.0, "pass": false, "basis": "environment"}]}' \
"$GMN_METADATA_URL/v1/rollout/verdicts"
# an episode that failed on your side
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
-d '{"reason": "physics diverged"}' "$GMN_METADATA_URL/v1/tasks/t-8123/fail"# A task runner works from inside its container over HTTP.
# The MCP tools are for the harness outside; see Verdicts below.
get_job(job: "job-7c1kd") # progress of the runner job{"tasks": [{"id": "t-8123", "params": {"rollout": "ro-4k9f2"}, "output_put_url": "...",
"rollout": {"id": "ro-4k9f2", "identity": "blake3:...", "seed": 3,
"spec_hash": "blake3:9f2a...", "judge": "reach-goal@v1",
"policy": "", "repro_class": "seeded-stochastic",
"parent_state": {"id": "st-91mm2", "digest": "blake3:...",
"bytes": 2148, "get_url": "https://..."},
"group": "grp-a41"}}],
"specs": {"blake3:9f2a...": {"scene": "reach-v2", "body": "hand", "target": [0.3, 0.1, 0.4],
"checkpoint": "acme/reach-ckpt41"}},
"lease_expires_at": "..."}A task runner uses the same loop as a task set runner. Each leased task carries a rollout block with the rollout id, the seed, the judge, the policy, the reproducibility class, the hash of its spec, and the parent state to start from. The lease response carries a specs map with each distinct spec once, keyed by that hash. A lease of thousands of seeds over a few scenes stays small.
Declare verdicts in batches. A verdict completes its task. If an episode fails on your side, report the task as failed. The rollout is recorded as failed with a null score. It is billed and it is not memoized. The metadata service allows 5 requests per second for each container, with a burst of 10. A 429 answer carries retry_after_secs. Wait that long and send the same request again.
autor rollout submit --environment sandbox:env-8x2mf3a9c1d4 \
--spec ./task-1443.json --group 16 --seeds 0..16 \
--judge swe-tests@v3 --profile train \
--idempotency-key task-1443-step-40
# → 16 sandboxes forked from env-8x2mf3a9c1d4; group grp-a41
# rollouts: [{rollout: "ro-9d41k", sandbox: "sbx-...", deadline: "..."}, ...]submit_rollouts(
environment: {kind: "sandbox", env: "env-8x2mf3a9c1d4"},
spec: {task: "swe-py-1443", checkpoint: "acme/agent-ckpt41"},
group: 16, seeds: "0..16",
judge: "swe-tests@v3", profile: "train",
idempotency_key: "task-1443-step-40")
# the response lists one entry per rollout, in the order you asked:
for r in resp.rollouts:
if r.sandbox is None: continue # memoized: the verdict is already in the row
sandbox_exec(sandbox: r.sandbox, cmd: "...")
declare_verdict(rollout: r.rollout, score: ..., basis: "harness")A sandbox submit forks one sandbox for each rollout. The response lists one entry for each rollout, in the order you asked. A memoized entry has no sandbox. Its verdict is already in the row, so skip it. Drive each other sandbox with sandbox_exec, then declare its verdict from outside.
Under eval, a sandbox submit with memoization on needs --policy with a label for the model that drives the sandbox. The model is outside your environment, spec, seed, and judge. Without the label, the next checkpoint gets the verdicts of this one. The rule is simple. What we cannot see, we cannot tell apart. If a thing changes the result and is not in your environment, spec, seed, judge, or policy, two different rollouts get one identity. A URL in a spec is not a policy, because the model behind a URL can change.
autor rollout submit \
--environment openenv:hub.openenv.dev/acme/swe-mini@sha256:2ab4... \
--specs ./tasks.jsonl --seeds 0..4 --policy ckpt41 \
--judge upstream:rubric@sha256:2ab4... --profile eval \
--evaluation swe-mini-ckpt41 --idempotency-key swe-mini-ckpt41
# → 8,192 rollouts; the environment runs on sandbox forks
# 1,847 memoized (the baseline and control arms); only changed identities runsubmit_rollouts(
environment: "openenv:hub.openenv.dev/acme/swe-mini@sha256:2ab4...",
specs: [...], seeds: "0..4", policy: "ckpt41",
judge: "upstream:rubric@sha256:2ab4...", profile: "eval",
env: {POLICY_API_KEY: {secret: "policy-key"}},
evaluation: "swe-mini-ckpt41", idempotency_key: "swe-mini-ckpt41")An OpenEnv environment runs from its image at a digest. The environment runs in a sandbox forked for each rollout. The adapter calls the policy endpoint that your spec names. Put the key for that endpoint in env on the submit as a secret reference. Only the adapter sees it. The sandbox that runs the environment does not.
The reward comes from the code of the environment, in the same sandbox as the agent. We stamp these verdicts basis: environment and mark them as a value the policy could influence. If your rubric matters, run it in your harness. That verdict is stamped basis: harness.
autor rollout verdict ro-9d41k --score 1.0 --pass --basis harness --data ./verdict.json
# a batch, one result per item
autor rollout verdicts --file ./verdicts.jsonldeclare_verdict(rollout: "ro-9d41k", score: 1.0, pass: true, basis: "harness",
data: {tests_passed: 41, tests_total: 41})
# or a batch, with one result per item:
declare_verdict(verdicts: [{rollout: "ro-9d41k", score: 1.0, pass: true},
{rollout: "ro-9d41m", score: 0.0, pass: false}])A verdict is a small JSON object that you declare against a rollout id. You give a score. Use null for a scoring gap. reward is accepted as an alias. You can add pass, a basis, and a data body of up to 16 KiB. We store data and return it. We do not read it. We stamp the judge, the way the verdict arrived, who sent it, the reproducibility class, and the time. A body that sets those fields itself is refused.
A task environment declares from inside its container, with no credential. See Task runner. A sandbox environment runs untrusted output. Nothing inside a sandbox can declare a verdict. The code under test cannot grade itself. The harness outside the sandbox declares, from a job, from an agent over MCP, or from a script with a workspace token.
A verdict is written once. A second declaration for the same rollout answers with the first. A retry of the same declaration after a dropped connection changes nothing. A batch returns one result for each item. One item that you no longer hold does not discard the rest. A judge fix is a new judge version and a new run.
Never run autor with a token inside a sandbox. A sandbox is for untrusted code. We refuse a verdict declared from inside a sandbox, and we treat the token as leaked. We refuse each write it attempts until we clear it. Rotate the token, then file a ticket to have it cleared. A harness that runs inside a sandbox cannot declare verdicts. Let your policy write its result to the sandbox filesystem. Let the harness outside read it and grade it.
autor rollout state ro-9d41k --label step-40 --expires-after 30d
# → st-8821a (a sandbox snapshot; it bills as one until it expires)
autor rollout states ro-9d41k
# task kind, from INSIDE your container: declare a file under $GMN_OUTPUT_DIR
curl -s -X POST -H 'Metadata-Flavor: givemeanode' \
-d '{"rollout": "ro-4k9f2", "label": "t=8.0s", "path": "states/ro-4k9f2-8.bin"}' \
"$GMN_METADATA_URL/v1/rollout/state"declare_rollout_state(rollout: "ro-9d41k", label: "step-40", expires_after: "30d")
list_rollout_states(rollout: "ro-9d41k")A running rollout can declare a state. A state is an opaque snapshot that other rollouts can start from. For a task environment, a state is a file that your runner writes, for example a MuJoCo position and velocity array. For a sandbox environment, a state is a sandbox snapshot with memory, processes, and disk. The sandbox pauses for a moment while we take it. We digest the bytes. We do not parse them.
A state bills as storage until the expiry you set. One state for each rollout is one snapshot for each rollout on your bill. Set --state-expires-after when you submit a sweep that declares states.
autor rollout fork st-8821a --branches 64 \
--spec-patch ./recovery-variants.jsonl --group
# → 64 children from st-8821a; group grp-c7f2; restore quoted before they startfork_rollout_state(state: "st-8821a", branches: 64,
specs: [{spec: {...}, labels: {variant: "retry"}}, ...], group: true)A fork makes children in the environment of the parent. Each child starts from the parent state, with its own spec patch and seed. A sandbox fork runs on the machine that holds the snapshot. A task child prefers the machine that made the state. Otherwise it downloads the state, and a large state takes time to restore. If the children would start from a state over 256 MiB that has to move, the response quotes the measured restore time from recent forks and asks you to confirm. A fork bills the execution of its children and nothing else.
autor rollout group grp-c7f2get_rollout_group(group: "grp-c7f2")A group is a set of rollouts that started from one state in one environment. We attest that, because group-relative training methods need it and cannot check it themselves. --group N on a submit or a fork makes the children and the group. The verdict of each member carries the group id. A group does not attest which policy drove its members or what their verdicts were. Group members are never memoized. A group under eval needs --no-memo.
autor rollout ls --evaluation ckpt-eval-9 --memoized
autor rollout qualify ro-4k9f2 --times 4
# → seeded-stochastic (scores equal, data differs); declare --repro-class seeded-stochasticlist_rollouts(evaluation: "ckpt-eval-9", memoized: true)
qualify_rollout(rollout: "ro-4k9f2", times: 4)Under --profile eval, a rollout with an identity that already ran in your workspace returns the stored verdict. It does not run again and it is not billed. The identity comes from the environment digest, the digest of the runner container, the spec hash, the seed, the judge, the policy label, the parent state, and the reproducibility class. A new container build, an edited spec, or a new judge version is a new identity. Each memoized row names the original rollout, its evaluation, and its workspace in memo_of. The submit response counts the hits.
Memoization is per workspace. An organization can ask us to widen it to all of its workspaces. The report then says which workspace each hit came from. --no-memo skips the lookup for one submit. Use it for calibration runs that measure variance.
Only a declared verdict is memoized. A failed rollout, a void, a training rollout, a group member, and an unrepeatable rollout are never memoized. We stamp the reproducibility class you declare. Under eval the default is seeded-stochastic. We never stamp bit-exact unless you say so. autor rollout qualify runs an identity several times with memoization off, compares the verdicts, and tells you the class to declare. Live services and the live web are unrepeatable. Memoization does not apply to them.
A new checkpoint named in a spec is a new identity on each of its rollouts. It runs in full and bills in full. The arms with an unchanged identity come back memoized. That includes baselines, control checkpoints, the missing rollouts of a resumed suite, and a suite you made larger. Memoization does not carry a verdict from one checkpoint to the next.
autor rollout submit --environment mjx-rollout --specs ./tasks.jsonl \
--seeds 0..8 --judge goal-region@v1 --evaluation ckpt-eval-9 --open
autor rollout add ckpt-eval-9 --environment mjx-rollout \
--specs ./more-tasks.jsonl --seeds 0..8 --judge goal-region@v1
autor evaluation get ckpt-eval-9
autor evaluation seal ckpt-eval-9
autor evaluation lssubmit_rollouts(..., evaluation: "ckpt-eval-9", open: true)
add_rollouts(evaluation: "ckpt-eval-9", specs: [...], seeds: "0..8")
get_evaluation(evaluation: "ckpt-eval-9")
seal_evaluation(evaluation: "ckpt-eval-9")
list_evaluations()An evaluation is a named set of rollouts that you compare, under one judge. Name it on a submit to make it or to add to it. Submit with --open when you do not know each rollout in advance. Add rollouts with autor rollout add. Close it with autor evaluation seal. An identity already in the evaluation is refused. get_evaluation shows the counts by status, the judges, and whether the evaluation is final.
autor evaluation real ckpt-eval-9 --file ./trials.jsonl
autor evaluation report ckpt-eval-9 --by policy
# → rollouts: 49,152 total, 49,118 scored, 14 void, 12,808 memoized
# reach-goal@v1, one environment digest, final: true, void rate 0.03%
# by policy: ckpt41 mean 0.412 (stderr 0.007) ... ckpt40 mean 0.387 ...
# real: rho 0.83 (Spearman) over 12 policies and 96 matched trialssubmit_evaluation_real(evaluation: "ckpt-eval-9",
rows: [{labels: {policy: "ckpt41"}, outcome: 0.71}, ...])
get_evaluation_report(evaluation: "ckpt-eval-9", by: "policy")We compute the report on request over the verdicts we recorded. It carries the counts by status, the void rate, one table for each environment digest, and an integrity block. Pass --by with a label key to split each table by that label. The integrity block says whether one judge graded each row, whether that judge is bound to code by a digest, how many rows were memoized and from which workspaces, and how many verdicts the environment graded itself. final is true when each member has finished.
A report refuses to average rows graded under different judges unless you pass --allow-mixed, and it names the mix. If two label values share an identity, the report refuses to compare them, because it would compare one number to itself. Environments always get one table each.
To compare with the real world, send rows of labels and a number. The report adds the Spearman correlation between mean score and mean outcome for each label group. It shows the matched and unmatched counts next to it, so a join that dropped half the data cannot print a confident number. From the CLI, autor evaluation real posts a JSONL file of rows.
autor rollout ls --evaluation ckpt-eval-9 --failed
autor rollout get ro-9d41klist_rollouts(evaluation: "ckpt-eval-9", failed: true)
get_rollout(rollout: "ro-9d41k")A rollout has two ways to end without a score. If your side fails, for example a harness exit that is not zero or a judge error, the rollout is recorded as failed. It is billed and it is not memoized. Resubmit it and it runs again. If our side fails, for example a lost machine, a void sandbox, or a state that did not restore, the rollout is recorded as void. It is not billed. It is excluded from its evaluation and its group, and it is safe to run again. A task episode goes back to the pool by itself. A scoring gap is a null score, and that is real signal about your policy.
Each rollout has a deadline of 60 minutes by default. The clock starts when the rollout starts to run. Queued time does not count. A rollout still running at its deadline is recorded as failed and billed. Its sandbox is deleted, so a harness that dies cannot leave a sandbox that bills. A snapshot declared as a state stays.
Rollouts add no new line to your bill. A task rollout costs the GPU minutes its runner uses, at the job rate. A sandbox rollout costs the RAM its sandbox holds, per minute. A state costs what a snapshot or a stored file costs. Queued time, voids, and memoized rollouts cost nothing.
| Limit | Value |
|---|---|
| Rollouts per evaluation | 65,536 |
| Members per group | 512 |
| States per rollout | 16 |
| State size (task kind) | 8 GiB. Sandbox states follow the sandbox limits. |
| Children forked from one state | 512, across calls |
| Spec size | 64 KiB |
| Rollout deadline | 60 minutes |
| Rollouts per lease | 32 by default, up to 8,192 for a vectorized simulator. 16 by default and 64 at most for OpenEnv environments. |
| Verdict data | 16 KiB per verdict |
One submit is also bounded by the task-set size of 50,000. An evaluation at the ceiling is two submits, or a submit and an add. list_limits shows the values that apply to you. An org admin can ask for more with request_limit_increase, and an operator decides.
A mission groups the nodes, jobs, commands, and files that answer one question. It gives you a page to watch while your agent works. Ask your agent to open one before it starts. Everything it does then rolls up at autoresearch.sfcompute.com/missions/<name>. The front of the page is what your agent has said. The other tabs carry the cost so far, commands as they run, declared results, and charts.
autor api post /missions -d '{"name": "llama-serving", "title": "Serve Llama 70B under $2/hr"}'
autor node create --name tulip --chip 8xh100 --mission llama-serving
autor run tulip --mission llama-serving -- python bench.py
autor job submit --chip 8xh100 --context ./bench \
--command 'python bench.py' --mission llama-servingopen_mission(name: "llama-serving", title: "Serve Llama 70B under $2/hr")
→ {name: "llama-serving", url: "https://autoresearch.sfcompute.com/missions/llama-serving", ...}
create_node(name: "tulip", chip: "8xh100", mission: "llama-serving")
run_command("tulip", "python bench.py", mission: "llama-serving")
submit_job(..., mission: "llama-serving")
finish_mission("llama-serving", result: {tokens_per_second: 8140, usd_per_hour: 1.84})finish_mission records the verdict on the page.get_mission returns the whole receipt in one call.autor command without a mission gets one named for the day, such as auto-2026-08-10-3f9ab2c1. Missions never cap or gate spending.mission on those calls to count the work somewhere else, for example when you reuse an old node for a new question.Your agent posts updates to a mission while it works. They are the first thing you see on the mission page. An update has a title and a body in markdown.
autor api post /missions/lora-portraits/updates -d '{
"title": "Run 7 is the best LoRA so far",
"body": "Seven of ten runs finished. Rank 32 at 1e-4 wins on FID.\n\n"
}'post_update(
mission: "lora-portraits",
title: "Run 7 is the best LoRA so far",
body: "Seven of ten runs finished. Rank 32 at 1e-4 wins on FID.\n\n"
)
→ {update_id: "upd-9k2fq",
url: "https://autoresearch.sfcompute.com/missions/lora-portraits#upd-9k2fq", ...}export_file and write  in the body. The artifact id becomes a share link, which unshare revokes. The stored body holds a real link, so the same text pasted into Slack shows the image too.## headings, bold, italic, inline code, fenced code blocks, lists, quotes, links, tables, and horizontal rules. Anything else appears as the characters you typed. An image marker inside a code fence publishes nothing.kind to "alert" when a person needs to act. An alert is marked on the mission page and on your dashboard until the agent posts again or finishes the mission. Everything else is a note.list_updates reads them back, which is how an agent that lost its context catches up.Research is in private preview. It is off by default for an organization. Ask us to turn it on.
Research attaches a researcher to a mission. You give it an objective, your source, a check set, and a budget. The managed researcher measures a baseline, changes one factor per hypothesis, runs each experiment as a job from your source, and reads the evaluations the platform computes. When an experiment beats its control and passes every check, the platform runs it again on fresh executions. If those runs pass, the mission completes with a confirmed result. A negative or inconclusive result is also a valid outcome.
autor research start \
--name inference-latency \
--source . \
--objective 'Reduce p95 latency below 120 ms. Preserve task quality and API compatibility.' \
--benchmark './bench.sh --json results.json' \
--check-set ./production-evals.toml \
--budget-usd 2000 \
--deadline 8hcreate_context(size_bytes: 48213, sha256: "...") # upload the source tar, then
finalize_context(context_id: "ctx-...")
start_research(
name: "inference-latency",
objective: "Reduce p95 latency below 120 ms. Preserve task quality and API compatibility.",
source_context: "ctx-...",
benchmark: "./bench.sh --json results.json",
checks: {schema: "research.checks/v1", checks: [...]},
budget_usd: 2000,
deadline_hours: 8
)
→ {name: "inference-latency", state: "ready", version: 1, control_version: 1,
planner: "managed", url: "https://autoresearch.sfcompute.com/missions/inference-latency", ...}--planner managed is the default. The researcher uses intelligence-1 at $0.01 per unit. See the rate card for compute prices.h100 by default. Pass --chip cpu-2 or --chip cpu-8 for work that needs no GPU. The number is the count of reserved CPU cores. See Shapes. Pass --image to build on a different base image.--source . uploads the directory. In a Git checkout the upload holds the tracked files and the untracked files that Git does not ignore. A .gmanignore file excludes more paths, one path or *.suffix per line. Git metadata never uploads. Executable bits are kept.
Each experiment runs its command with sh -c from the root of your source at /workspace/source. The command must write a JSON object to its results file. The default file is results.json in the source root. The platform reads that object. It is the only measurement the evaluator trusts.
{"p95_ms": 108, "throughput_tokens_per_s": 4120, "quality": {"score": 0.987}}A check set lists the conditions a result must pass. Each check names a metric in the results JSON with a dotted path, a comparison, and one bound. A threshold is an absolute bound. A ratio compares the experiment to its control. The evaluator takes the median over an experiment's repetitions before it compares. Files may be TOML, YAML, or JSON.
schema = "research.checks/v1"
[[checks]]
id = "quality"
metric = "quality.score"
op = ">="
threshold = 0.98
[[checks]]
id = "latency"
metric = "p95_ms"
op = "<="
threshold = 120
[[checks]]
id = "faster-than-control"
metric = "p95_ms"
op = "<="
relative_to_control = 0.9passed, failed, or unknown. A missing metric or a missing control is unknown. Unknown is never a pass.autor research get inference-latency
autor research steer inference-latency --expected-version 7 \
--question-id question-7a2 --answer 'continue without search'
autor research steer inference-latency --expected-version 8 \
--instruction 'Prioritize prefix caching. Keep the current checks and budget.'
autor research stop inference-latencyget_research(name: "inference-latency")
→ {state: "needs_input", version: 7, control_version: 1, spent_usd: 614.22,
question: {id: "question-7a2", text: "May I add external search to the allowed network policy?"},
experiments: [{id: "baseline", verdict: "passed", measurements: {p95_ms: 141}}, ...], records: [...]}
steer_research(name: "inference-latency", expected_version: 7,
question_id: "question-7a2", answer: "continue without search")
steer_research(name: "inference-latency", expected_version: 8,
instruction: "Prioritize prefix caching. Keep the current checks and budget.")
stop_research(name: "inference-latency")get_research as often as you like. The states are ready, running, waiting, needs_input, verifying, stopping, stopped, completed, and failed.version you read. If the mission changed since then, the change is refused. Read again and retry.You can select the experiments yourself. Start with --planner customer and a plan, or hand control over on an existing mission. The platform runs, evaluates, and retains each experiment. When your plan finishes, the mission waits for the next plan or for --complete.
cat > experiments.toml <<'TOML'
schema = "research.plan/v1"
[[experiments]]
id = "manual-baseline"
command = "./bench.sh --precision bf16 --json results.json"
[[experiments]]
id = "manual-fp8"
control = "manual-baseline"
command = "./bench.sh --precision fp8 --json results.json"
repetitions = 5
TOML
autor research steer inference-latency --expected-version 8 \
--expected-control-version 1 --planner customer --accepted-work continue
autor research steer inference-latency --expected-version 9 --plan ./experiments.toml
autor research steer inference-latency --expected-version 11 \
--expected-control-version 2 --completesteer_research(name: "inference-latency", expected_version: 8, expected_control_version: 1,
planner: "customer", accepted_work: "continue")
steer_research(name: "inference-latency", expected_version: 9,
plan: {schema: "research.plan/v1", experiments: [
{id: "manual-baseline", command: "./bench.sh --precision bf16 --json results.json"},
{id: "manual-fp8", control: "manual-baseline", repetitions: 5,
command: "./bench.sh --precision fp8 --json results.json"}]})
steer_research(name: "inference-latency", expected_version: 11, expected_control_version: 2, complete: true)id, a command, and an optional control that names an earlier experiment. repetitions defaults to 3 and timeout_seconds to 3600. chip, chip_count, image, and results override the mission defaults.--planner managed hands control back. The researcher continues with the same objective, evidence, and remaining budget.The budget is a hard cap across everything the mission spends. Before any experiment or researcher turn starts, the platform reserves its maximum charge against the budget. A fifth of the budget is held for the confirmation of a selected result. Pass --confirmation-reserve-usd to change the amount.
GPU performance counters, which Nsight Compute (ncu) reads, are locked on every GPU by default. Ask for them at create time and the platform unlocks them.
autor node create --name counters --chip 8xh100 --profiling --clock-lock
autor run counters -- ncu --metrics gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed ./saxpycreate_node(name: "counters", chip: "8xh100", profiling: true, clock_lock: true)
run_command("counters", "ncu --metrics gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed ./saxpy")There is nothing to configure inside the container. A memory-bound kernel reports what you expect:
dram__bytes_read.sum 134.23 Mbyte
gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed 78.10 %
gpu__time_duration.sum 70.94 us
sm__throughput.avg.pct_of_peak_sustained_elapsed 35.33 %ncu --set full), and so does attaching to a running process.8xh100). On a shared machine they could report on a neighbor. A profiling request on a single-GPU shape is refused with the fix.An unlocked machine also needs a profiler new enough to use it. ncu below 2025 fails on current drivers with Failed to prepare kernel for profiling. So profiling: true with no image lands on the catalog's profiling entry, which is cuda-12.9. That is also the plain default. When you name an image, only entries whose profiler works are accepted. An image that cannot profile is refused, and the refusal names the entry that works.
The same rule covers from_snapshot. When profiling selects a newer image than the snapshot was built against, the response says so.
Batch jobs take the same flag, so a kernel sweep can profile every variant.
autor job submit --chip h100 --count 8 --profiling --clock-lock \
--image ghcr.io/you/kernels:cu129 --command ./sweep.shsubmit_job(chip: "h100", chip_count: 8, profiling: true, clock_lock: true,
image: "ghcr.io/you/kernels:cu129", command: "./sweep.sh")A job runs your image, so bring your own profiler. Base the image on CUDA 12.9 or newer, or apt-get install cuda-nsight-compute in your Dockerfile. Below ncu 2025 the profile fails on every machine. chip_count: 8 (CLI: --count 8) is required. The flag is free and cannot change after submit.
Counters say why a kernel is slow. clock_lock: true makes how long it takes comparable across runs. It pins the GPUs to the model's published sustained clock instead of letting them drift with boost and thermal state. Kernel-search work usually sets both flags.
The lock is a ceiling, not a guarantee. The hardware still throttles under power or thermal pressure. On an 8-GPU SXM baseboard, your neighbors' heat reaches your GPUs, so a full 8xh100 is the only fully isolated shape. The flag is free and cannot change after create.
Code on a node can read its own facts and stop itself with no credential. The address is http://169.254.42.1 (also metadata.givemeanode.internal). Every request needs the header Metadata-Flavor: givemeanode.
curl -s -H 'Metadata-Flavor: givemeanode' http://169.254.42.1/v1/node
curl -s -H 'Metadata-Flavor: givemeanode' http://169.254.42.1/v1/events
curl -s -H 'Metadata-Flavor: givemeanode' http://169.254.42.1/v1/egress
curl -s -H 'Metadata-Flavor: givemeanode' http://169.254.42.1/v1/network
curl -s -X POST -H 'Metadata-Flavor: givemeanode' http://169.254.42.1/v1/stop/v1/node is identity and money: name, state, rate, grace window./v1/events carries structured notices with deadlines, such as machine retirement, spend-cap warnings, and a price increase ahead (price_change). A checkpoint-aware training script polls it like cloud spot metadata./v1/egress is the network policy. The default is open internet. The response lists the blocked destinations: outbound SMTP on port 25, a small set of mining pool ports, and the cloud metadata IP. The blocked ports fail fast. The metadata IP is dropped silently, so a connect to it hangs. DNS is unfiltered and TLS is never intercepted, so a certificate error or a failed name comes from the origin. It answers for nodes and jobs./v1/network is the roster of your private network.POST /v1/stop ends the node from inside with stop_node semantics. A training run's last line can be that curl.When a node starts, its metadata arrives on the host about one second later. A call in that time waits for it, up to 5 seconds. After 5 seconds, the call returns HTTP 503 with metadata temporarily unavailable; retry. Retry that response with a bounded wait. POST /v1/stop can wait up to 5 more seconds while the control plane records the stop, so give it a timeout of more than 10 seconds. A refused POST /v1/stop does not stop the node, so retry it too.
HTTP 403 has two causes. The request does not have the header Metadata-Flavor: givemeanode, or the request does not come from the container itself. If your code uses an HTTP proxy, set NO_PROXY=169.254.42.1,metadata.givemeanode.internal.
Job containers get the same service at $GMN_METADATA_URL, with job routes. GET /v1/job is identity and time: job id, attempt, deadline, checkpointed_at. POST /v1/checkpoint declares $GMN_CHECKPOINT_DIR consistent and captures it. POST /v1/yield ends the attempt and resumes it from the slot. POST /v1/stop is cancel, and it is terminal. The whole workflow is under Checkpoints.
At job start or resume, metadata calls can return HTTP 503 with metadata temporarily unavailable; retry. Retry that response with a bounded wait. The refused call does not execute a job control request. Successful metadata describes the current attempt.
The guides combine the features above into complete workflows. Each one names the image it runs on and links to the docs for every capability it uses.
These guides cover LoRA SFT, GRPO, agentic rollouts, Prime-RL, and multi-stage training. Each names its image, its topology, and its checkpoint discipline. All of them run on one 8x node or as batch jobs. Multi-node training is not offered. The pinned stacks for this work are under Post-training in the image catalog.
A dataset goes in and a LoRA adapter comes out. This works up to the largest models that fit one 8x in bf16. Larger bases fit with QLoRA.
autor job submit --chip h100 --count 8 --context ./sft \
--command 'python sft.py' # ./sft: Dockerfile (FROM your training base,
# pip install trl peft) + sft.py + config.
# sft.py: SFTTrainer; state to $GMN_CHECKPOINT_DIR
# each epoch; final adapter to $GMN_OUTPUT_DIRsubmit_job(
chip: "h100", chip_count: 8,
command: "python sft.py", context_id: "ctx_...", # Dockerfile + code ride the context
hf_cache: true, # base weights cached per-org
resume: "checkpoint", idempotency_key: "sft-run-1")Two lines in sft.py matter. Write the trainer state to $GMN_CHECKPOINT_DIR and declare it (Checkpoints), so a preemption costs minutes instead of the whole run. Write the adapter and eval numbers to $GMN_OUTPUT_DIR and $GMN_RESULT_PATH, so the verdict comes back on get_job.
RL post-training runs as three cooperating processes: a trainer, a rollout orchestrator, and an inference server the policy samples from. On one 8x they share the machine. vLLM serves the policy on a GPU slice, TRL's GRPO trainer updates weights on the rest, and rollouts loop between them. Once built, trl-0.21-cuda12.9 carries both halves. Until then, start from vllm-0.26-cuda12.9 and add TRL with the pip line below.
autor node create --name grpo-box --chip 8xh100 --image vllm-0.26-cuda12.9 --wait
autor run grpo-box -- pip install --break-system-packages trl peft datasets accelerate
autor run grpo-box -d -- python serve_policy.py # vLLM (for restart supervision
autor run grpo-box -d -- python grpo_train.py # use the MCP form below)
autor run grpo-box -- tail -20 rewards.jsonl # or scrape it: register_scrapecreate_node(name: "grpo-box", chip: "8xh100", image: "vllm-0.26-cuda12.9")
get_node("grpo-box") # poll to running; create never blocks
run_command("grpo-box", "pip install --break-system-packages trl peft datasets accelerate")
run_command("grpo-box", "python serve_policy.py", detach: true, restart: "on-failure")
run_command("grpo-box", "python grpo_train.py", detach: true)
query_metrics(...) # reward curves via the free platform telemetryGroup the run under a mission. Call finish_mission with the final reward, so the mission receipt records the whole run.
Agentic RL needs three services: environments to act in, code execution for the agent's tools, and a reward signal. Run environment rollouts as sweeps. Each variant is one rollout batch, with verdicts written to $GMN_RESULT_PATH. Run untrusted tool code in a sandbox, one fork of a baked environment per sample. Your agent drives sandboxes over MCP. A job drives them over the HTTP API with a service token in env, never on the command line. If your reward is a judge model, serve it like the policy server above (sglang-0.6-cuda12.9 or vLLM on a 1x or a slice of your 8x) and point rollout workers at it.
autor node create --name judge --chip h100 --image vllm-0.26-cuda12.9 --wait
autor run judge -d -- python -m vllm.entrypoints.openai.api_server \
--model acme/reward-judge --port 8000
autor api post /nodes/judge/endpoints --data '{"port": 8000, "auth": "bearer"}'
autor api post /jobs/sweeps --data '{
"shared": {"chip": "h100", "context_id": "ctx_...", "command": "python rollout.py",
"env": {"JUDGE_URL": "https://...", "GMN_TOKEN": "..."}},
"variants": [{"env": {"SHARD": "0"}}, {"env": {"SHARD": "1"}},
{"env": {"SHARD": "2"}}, {"env": {"SHARD": "3"}}],
"label": "rollouts-r1", "idempotency_key": "rollouts-r1"}'
# each variant is one rollout batch; verdicts land in $GMN_RESULT_PATHcreate_node(name: "judge", chip: "h100", image: "vllm-0.26-cuda12.9")
get_node("judge") # poll to running; create never blocks
run_command("judge", "python -m vllm.entrypoints.openai.api_server --model acme/reward-judge --port 8000", detach: true)
expose_port("judge", port: 8000, auth: "bearer")
submit_jobs(
shared: {chip: "h100", context_id: "ctx_...", command: "python rollout.py",
env: {JUDGE_URL: "https://...", GMN_TOKEN: {secret: "gmn-token"}}},
variants: [{env: {SHARD: "0"}}, {env: {SHARD: "1"}}, {env: {SHARD: "2"}}, {env: {SHARD: "3"}}],
label: "rollouts-r1", idempotency_key: "rollouts-r1")
# each variant is one rollout batch; verdicts land in $GMN_RESULT_PATH,
# so list_jobs(label: "rollouts-r1", summary: true) is the reward tablePrime-RL runs its trainer, its orchestrator, and its vLLM inference servers together on one 8x node. Run it from its own image. The persistent disk mounts at /home/appuser, the home directory of the user that the image runs as.
autor node create --name prl --chip 8xh100 --wait \
--image ghcr.io/primeintellect-ai/prime-rl:v0.9.0 --home /home/appuser
autor cp ./rl.toml prl:~/rl.toml
autor run prl -d -- sh -c 'cd ~ && rl @ rl.toml' # outputs/ goes on the diskcreate_node(name: "prl", chip: "8xh100",
image: "ghcr.io/primeintellect-ai/prime-rl:v0.9.0", home: "/home/appuser")
get_node("prl") # poll to running; create never blocks
write_file("prl", "~/rl.toml", "...")
run_command("prl", "cd ~ && rl @ rl.toml", detach: true) # outputs/ goes on the diskhome out of /app. The image keeps its code and its Python environment in /app, which is also its working directory. A disk at /app hides them. /home/appuser is also the default home for this image, so you can leave it out.~. Commands start in /app. Prime-RL writes its checkpoints and logs under outputs/ in the directory where it starts, and /app resets at every wake. Start with cd ~, or set PRL_OUTPUT_DIR to a path under ~.~/.cache/huggingface. vLLM keeps its compile results in ~/.cache/vllm. Both survive a stop, so the next start skips the download and most of the compile.Prime-RL sets how many rollouts run at once from the free KV cache on its inference servers. Training and evaluation rollouts share this bound. It starts low and goes up. An evaluation runs group_size rollouts for each example, and the default is 1. If one evaluation needs more rollouts than the bound, it runs in waves and takes longer. Set initial_inflight to at least num_examples times group_size for one evaluation. max_inflight is the ceiling. If the servers run short of KV cache, Prime-RL lowers the bound itself.
# rl.toml
[orchestrator.concurrency]
initial_inflight = 256 # num_examples x group_size of one evaluation
max_inflight = 512
[inference.vllm]
enforce_eager = true # for smoke runs; starts faster, generates more slowlyA standalone evaluation has its own bound. The image's evals command reads it from [eval.concurrency]. That table has the same fields as [orchestrator.concurrency]. The verifiers eval command uses max_concurrent (the -c flag). Set it to the number of examples times the rollouts for each example (the -r flag). Older verifiers releases default to 32 requests at once.
A vLLM server compiles the model and captures CUDA graphs each time it starts. A short smoke run can skip both with enforce_eager = true. The server then starts faster and generates tokens more slowly. Use it only when the start time is a large part of the run.
A model cache on the disk makes the disk larger. When a wake restores the disk onto another machine (see disk under Stop and wake), a larger disk takes longer. The other choice is scratch. Set HF_HOME to a path on /scratch. Scratch is empty after every stop, so the model downloads again at each start.
Multi-stage post-training (distillation or SFT first, RL second) chains two jobs. Stage 1 runs to a verified artifact. Stage 2 starts from it. The stage 1 job writes final weights to $GMN_OUTPUT_DIR, or pushes large ones to org storage. Your agent checks the stage 1 eval verdict from $GMN_RESULT_PATH. The stage 2 job pulls the artifact and trains. Use the same image for both stages, and one mission over both.
autor job submit --chip h100 --count 8 --context ./train \
--command 'python sft.py' --idempotency-key stage1-sft
autor job get job_... # poll to done; the result is the stage 1 eval
# verdict, the output URL is the weights
autor job submit --chip h100 --count 8 --context ./train \
--command 'python grpo.py' --env STAGE1_URL='https://...' \
--idempotency-key stage2-rl # one build context serves both stagessubmit_job(chip: "h100", chip_count: 8, context_id: "ctx_...",
command: "python sft.py", idempotency_key: "stage1-sft")
get_job("job_...") # poll to done; result = the stage 1 eval verdict,
# output = the weights as a download URL
submit_job(chip: "h100", chip_count: 8, context_id: "ctx_...",
command: "python grpo.py", env: {STAGE1_URL: "https://..."},
idempotency_key: "stage2-rl") # grpo.py pulls the weights firstImage generation here means a large shared base checkpoint (SDXL-class, Flux-class) plus one or more small LoRA adapters. The pytorch-2.13-cuda12.9 catalog image carries torchvision, ffmpeg, and libgl1, so diffusers pipelines and ComfyUI import without setup. Base weights use the model cache, so a sweep pulls Flux once per machine.
Adapters are small (tens to hundreds of MB) and come from three places:
get_job, or lands on your Hugging Face repo through a write connection.For a batch sweep, carry adapters in an uploaded context, which takes up to 10 GiB. Upload once with create_context then finalize_context, and put the id in the sweep's shared.context_id. autor job submit --context ./dir uploads inline but submits one job. Sweeps take the id form. The inline context file map caps at 4 MiB, which fits code but not weights.
Batch generation sweeps over prompt sets, seeds, and LoRA strengths, and it tolerates queueing, so it fits the job rate. Put the variant parameters in env, share the base weights with hf_cache, and land every variant's images in your own bucket with output_to:
autor api post /jobs/sweeps --data '{
"shared": {"chip": "h100", "context_id": "ctx-...", "hf_cache": true,
"command": "python generate.py",
"output_to": {"connection": "acme-out", "dest": "s3://acme-ml/grids/"}},
"variants": [{"env": {"STRENGTH": "0.6"}}, {"env": {"STRENGTH": "0.8"}},
{"env": {"STRENGTH": "1.0"}}],
"label": "lora-strength", "idempotency_key": "lora-strength-1"}'submit_jobs(
shared: {chip: "h100", context_id: "ctx-...", hf_cache: true,
command: "python generate.py",
output_to: {connection: "acme-out", dest: "s3://acme-ml/grids/"}},
variants: [{env: {STRENGTH: "0.6"}}, {env: {STRENGTH: "0.8"}}, {env: {STRENGTH: "1.0"}}],
label: "lora-strength", idempotency_key: "lora-strength-1")output_to takes a write-enabled s3-keys connection and a prefix. Each variant's $GMN_OUTPUT_DIR capture lands at <dest><job_id>-<attempt>.tar, so a 256-variant sweep becomes one dataset in your bucket with no key collisions and no credential in any container. Without it, each job's images come back as a download link on get_job. Either way, write a verdict (image count, NSFW filter hits, CLIP score) to $GMN_RESULT_PATH, so list_jobs(label, summary: true) is the comparison table.
Check samples before a long batch finishes. Have the script write a small grid every N images and declare it as the job's preview. Poll get_job, look at the grid, and keep going or cancel_job. A 2,000-image batch whose LoRA strength is wrong at image 50 then costs minutes.
autor job submit --chip h100 --context ./gen \
--command 'python generate.py' --idempotency-key batch-2k
autor job get job_... # → preview.download.url once a grid is captured
autor job cancel job_... # wrong strength; pay only the minutes so farsubmit_job(chip: "h100", context_id: "ctx_...",
command: "python generate.py", idempotency_key: "batch-2k")
get_job("job_...") # → preview.download.url; fetch and look
cancel_job("job_...") # wrong strength; pay only the minutes so farThe declaring side, inside generate.py:
# every 50 images: write the grid, then declare it
grid.save(f"{os.environ['GMN_OUTPUT_DIR']}/grid.png")
requests.post(f"{os.environ['GMN_METADATA_URL']}/v1/preview",
json={"path": f"{os.environ['GMN_OUTPUT_DIR']}/grid.png"},
headers={"Metadata-Flavor": "givemeanode"})For interactive generation, run ComfyUI on a node behind an exposed port. There is no pinned ComfyUI catalog entry, because its custom node ecosystem moves too fast for a pin. Install a version you have tested onto the torch image, which carries the torch stack and the system libraries it loads:
autor node create --name comfy --chip h100 --image pytorch-2.13-cuda12.9 --wait
autor run comfy -d -- sh -c 'git clone --depth 1 --branch v0.31.0 \
https://github.com/comfyanonymous/ComfyUI && pip install \
--break-system-packages -r ComfyUI/requirements.txt requests && \
pip install --break-system-packages --force-reinstall --no-deps \
torchaudio --index-url https://download.pytorch.org/whl/cu129'
autor logs comfy -f # pin a RECENT tag you've tested (current model
# architectures need one; an old pin fails to load them);
# PEP 668 on the 24.04 base wants --break-system-packages
# (the install lands in ~/.local, which persists); the
# torchaudio line matches the wheel to the image's CUDA
autor run comfy -d -- python ComfyUI/main.py --listen 0.0.0.0 --port 8188
autor api post /nodes/comfy/endpoints --data '{"port": 8188, "auth": "bearer"}'create_node(name: "comfy", chip: "h100", image: "pytorch-2.13-cuda12.9")
get_node("comfy") # poll to running; create never blocks
run_command("comfy", "git clone --depth 1 --branch v0.31.0 https://github.com/comfyanonymous/ComfyUI && pip install --break-system-packages -r ComfyUI/requirements.txt requests && pip install --break-system-packages --force-reinstall --no-deps torchaudio --index-url https://download.pytorch.org/whl/cu129", detach: true)
get_command("comfy", "cmd_...") # poll to exit_code 0 (minutes of pip)
run_command("comfy", "python ComfyUI/main.py --listen 0.0.0.0 --port 8188", detach: true)
expose_port("comfy", port: 8188, auth: "bearer")import_data from your connections, or curl or huggingface-cli from inside the node for public weights.~/ComfyUI. git clone refuses a directory an import created.--dest is '~/ComfyUI/models/loras'. A single-file checkpoint repack, such as the official Comfy-Org exports, loads through UNETLoader and belongs in '~/ComfyUI/models/diffusion_models', not checkpoints.~, the volume that persists.stop_node, then snapshot_node (poll list_snapshots to ready), then create_node(from_snapshot). The clone runs the original image unless you name another, so the Python environment works as it did.expose_port again, because endpoints die with the container.Fine-tuning the open robot policy models (openpi, OpenVLA, the LeRobot families) fits one GPU for adapter runs and one 8-GPU node for full fine-tunes. Policy evaluation suites and synthetic data generation fan out as batch jobs. A trained policy serves robots over a WebSocket from an exposed port. Training across several nodes does not fit. There is no cross-node fabric, so work that needs more than 8 GPUs in one process belongs on a cluster product.
These guides run on pytorch-2.13-robotics-cuda12.9. The image section documents simulator and GPU rendering support, including the Isaac Sim limits.
Bring a dataset in once and keep it resident. Pull public data from inside the node (Hugging Face, GCS, S3) onto the volume or the scratch disk, or use a connection for private buckets. A copy on local disk does not tax a shared uplink every epoch. Working sets up to a few hundred GB fit the default volume, and the limit can be raised per organization.
Checkpoints leave the same ways data comes in. Push to your own bucket or Hugging Face repo from inside the node, use a write connection, or use org storage. Batch jobs that load the same base model can set hf_cache: true to share the model cache.
autor node create --name trainer --chip 8xh100 \
--image pytorch-2.13-robotics-cuda12.9 --wait
autor run trainer -- huggingface-cli download my-lab/aloha-episodes --repo-type dataset
autor run trainer -d -- python train.py --config pi0_aloha
autor logs trainer -f # incremental output, any timecreate_node(name: "trainer", chip: "8xh100",
image: "pytorch-2.13-robotics-cuda12.9")
run_command("trainer", "huggingface-cli download ...")
run_command("trainer", "python train.py ...", detach: true)
get_command("trainer", "cmd_...") # poll from anywhereA fine-tune runs as a detached command on a node you hold, where files survive stops, or as a batch job with a checkpoint slot, which is cheaper, preemptible, and resumes automatically. Evaluation suites fan out with sweeps: one build, one job per episode, and one call for the results.
Run the policy server as a detached command and expose its port. The HTTPS URL passes WebSocket traffic, so the standard remote-inference clients work as shipped. Two limits matter. Exposed URLs expire, so a robot that holds a fixed URL needs its agent to refresh it. And the round trip over the public internet suits chunked action prediction at about one request per second, not tight real-time control loops.
On PyTorch 2.8 and newer, torch.linalg.eigh and torch.linalg.eigvalsh on CUDA fail when one call carries more than about 64,000 small matrices. Torch hands a batch of small matrices to cuSOLVER's batched solver. Above that count the call errors with CUSOLVER_STATUS_INVALID_VALUE or CUSOLVER_STATUS_INTERNAL_ERROR from cusolverDnXsyevBatched. The exact ceiling moves a little with matrix size and dtype. Split the batch into calls of 50,000 matrices or fewer and concatenate the results:
vals = torch.cat([torch.linalg.eigvalsh(chunk) for chunk in A.split(50_000)])The limit is in the torch wheel and the cuSOLVER library it bundles. It behaves the same on every image here and on any other machine. pytorch/pytorch#166004 is the upstream report. On torch builds that ship the MAGMA backend, torch.backends.cuda.preferred_linalg_library("magma") also avoids the cap, at lower throughput.
Batched eigensolvers can also fail to converge on matrices with repeated or near-repeated eigenvalues. That is a separate problem from the batch cap. Add a small random jitter to the diagonal before the call. It pulls the repeated eigenvalues apart. A uniform eps * I does not help, because it shifts every eigenvalue by the same amount.
Your account has a personal organization. We make it when you sign up, and its name is your email. Only you use it. Its balance, nodes and missions are yours alone.
A team is an organization for work with other people. Create a team, then invite your teammates to it. A team has its own members, workspaces and balance, and you are its admin. Its balance starts at zero, with no free credit and no card. Add its first credit on the billing page before you start a node in it. The answer to create_org gives the link as fund_url.
autor api post /orgs --data '{"name": "Protein lab"}'create_org(name: "Protein lab")autor api post /org/invites --data '{"email": "student@lab.edu", "role": "member", "org": "Protein lab"}'invite_member(email: "student@lab.edu", role: "member", org: "Protein lab")In the browser, open the organization menu at the top left of any page and choose Create team. The new team becomes your active organization at once. The menu shows a Personal badge on each personal organization, and list_team gives each organization a kind of personal or team.
You can also invite people to your personal organization. They then share an organization that has your email as its name. For shared work, use a team.
The person accepts an invite when they next sign in with the invited address. Gmail ignores dots and plus-sign suffixes, so jane.doe@gmail.com and janedoe@gmail.com are one inbox, and one inbox has one Autoresearch account. An invite to any spelling of a Gmail address goes to the account that already uses that inbox. If you sign in with a second spelling of an inbox that has an account, the sign-in tells you which address to use.
Over MCP and the HTTP API, a new team does not become your active organization. The tools for billing, members, workspaces and caps take an org argument, so pass the team name there. Nodes, jobs and sandboxes go to the organization your connection uses. To start them in the team, make it active and use a connection that you did not open in a specific organization. Over MCP, switch_active_org makes it active. You cannot create a team while you are the admin of 10 or more organizations.
| Role | Can |
|---|---|
admin | Everything |
billing | The money pages |
member | Their own nodes and usage |
The same management is in the browser at /team. That page has the roster, invites, role changes, workspaces and who can reach them, and a stop button for any org node (also stop_org_node over MCP). What each workspace can spend is on /limits. Members of a workspace share its nodes. switch_active_org(org) picks where your next node bills if you belong to several orgs.
Separate agent sessions share a workspace the same way two people do, even under one account. list_nodes answers with every node in the workspace, and each row says who created it.
autor node lslist_nodes()yours, created_by, and the mission the node was created for. While work is live, a row also carries sessions and running_commands.force. A stop kills whatever runs on the node, and a delete erases its disk.If you run several agents yourself, tell each one its mission. A request identifies your account, and your agents share it, so a node your other agent made reads back as yours. Pass mission to list_nodes, stop_node, and delete_node. Each node is then judged against that mission. Rows gain same_mission, and a node from a different mission is treated like a teammate's.
autor node ls --mission osteosarclist_nodes(mission: "osteosarc")
stop_node("train-osteosarc", mission: "osteosarc")For a stronger separation, give each agent its own service token. Every token is its own machine member, so created_by names the token and the checks above work without a mission. A workspace per agent goes further, because then node names cannot collide at all.
Node names are unique per workspace, so the same name in two workspaces is two different nodes. Each workspace carries its own spend cap. Create workspaces at /team, then point each agent at one.
autor workspace ls # the ones your org has
autor whoami # names the workspace you land in
autor --workspace osteosarc node ls # or set GMAN_WORKSPACE
# or mint a token that can only reach one workspace
autor token create --name materials-agent --workspace materialsA workspace bounds what you can see. autor node ls names the workspace it read above the table, and lists nodes from that workspace only. A node your teammate keeps in another workspace is not missing. autor workspace ls shows where else to look.
You decide what we keep about your workloads. Six captures are each a switch at /settings/data. A capture that is off is never written.
| Capture | What it records | Default |
|---|---|---|
| Command output | The last 8 KiB of a detached command, so get_command answers after the node stops. | 90 days |
| Command lines | What ran. Encrypted at rest and never echoed back. Feeds the mission Commands tab and support. | 90 days |
| Declared results | The JSON your code writes to GMN_RESULT_PATH. Mission receipts are built on these. | kept |
| Logs | What query_logs reads. Off, logs still stream to your own volume and any exec can tail them. | 31 days |
| Traces | The spans you send us. | 14 days |
| Job logs and artifacts | Kept, or purged within a minute of the job ending. | 30 days |
autor retention # the effective policy
autor retention --json # the same answer, for a security reviewget_retention() # policy + what is kept regardlessSet it from a script with autor retention set. It changes the same policy the page changes. Only what you name changes, so a line that turns logs off never re-enables anything else. An org admin has to run it.
autor retention set --logs off --content-days 30 --yes
autor retention set --logs-days 7 --yes # keep logs, but only 7 days
autor retention set --zero --yes # every optional capture offset_retention(logs: false, command_content_days: 30)No setting turns these off:
Sandboxes write nothing by construction. A sandbox_exec returns {exit_code, stdout, stderr} and leaves no row and no log line. The same is true of every sync exec not attached to a mission, and of write_file content. Environment variable values are encrypted at rest, echoed back by name only, and redacted from anything we ship.
Changes apply from the moment you make them and cannot be undone. A capture turned back on starts new records. It does not restore old ones. Only an org admin can change these settings.
CI cannot complete a browser login. For CI, and for machines that are rebuilt for every task, an org admin mints a service token. It acts as its own machine member, scoped to the workspaces you grant it. Mint it from /team or the CLI. The token is shown once. Store it as a secret at once.
autor token create --name ci-deploy --workspace protein-lab
# prints GMN_TOKEN=gmnt_... once - copy it into your CI secrets
autor token ls # names, scopes, workspaces, expiry, last used
autor token revoke <id> # immediateGMN_TOKEN to the value. Every autor command and every /preview API call then runs as that token. The CLI also reads the earlier name GMAN_TOKEN, so a pipeline that already sets it keeps working.infra (nodes, sandboxes, exec, jobs) by default, with read-only and org-read variants. A read-only token can list and read. It cannot create, change, or delete anything.--expires-in-days. 0 means it does not expire.autor token revoke, and access stops at once.# .github/workflows/deploy.yml
- run: curl -fsSL https://autoresearch.sfcompute.com/cli.sh | bash
- run: autor job submit --chip h100 --context ./train --wait
env: { GMN_TOKEN: ${{ secrets.GMN_TOKEN }} }On a machine that stays around, such as a dev box or a Codespace, autor mcp install uses your own login and needs no token.
Everything an agent can do over MCP, a script can do over plain HTTP. The two interfaces run the same code and answer with the same wording. Every MCP tool has a route here, and we check that automatically. The one exception is switch_active_org. It has no route, because you name the organization and the workspace on each call.
curl -s -H "Authorization: Bearer $GMN_TOKEN" \
https://autoresearch.sfcompute.com/preview/nodesThe contract is an OpenAPI 3.1 document. You do not need a token to read it. Every operation has a unique operation id, a description, typed parameters, a typed request body, and a typed response.
curl -s https://autoresearch.sfcompute.com/openapi.json | jq '.paths | keys'autor login mints one for you. An org admin mints a service token for a pipeline. A call outside the token's scopes is refused with the scope word to ask for.?cursor= and ?limit=, and answer with {object: "list", items, next_cursor?}. Pass next_cursor back verbatim for the next page.?offset=, not streamed.?workspace=<slug>. There is no way to set a workspace once and have it stick.Every failure from the API comes back in one shape:
{
"error": {
"code": "refused",
"message": "no node named "tulip" in workspace protein-lab"
}
}The code is for your program. The message is for a person, and you can show it without rewriting it. The code vocabulary is open. Treat a code you have not seen as a generic failure of the HTTP status that carried it.
| Status | Code | What to do |
|---|---|---|
| 400 | invalid | The request did not parse. Fix it and retry. |
| 401 | unauthenticated | No token, or one we cannot verify. Run autor login again. |
| 403 | forbidden | The token is real but does not reach this call. The message names the scope to request. |
| 404 | not_found | No such route. Check the path against the OpenAPI document. |
| 422 | refused | The request was well formed and we declined it. The message says what to do instead. |
| 429 | rate_limited | Over your budget. Wait the number of seconds in Retry-After. |
| 500 | internal | Our fault. Retry. If it continues, send us the request-id. |
| 503 | settling | The node changed state under the call (a wake, a stop, or a container exit). Nothing ran and nothing was billed. Wait the seconds in Retry-After and send the same call again. |
Every response from the API carries a request-id header. Quote it when you report a problem. Every response also carries a Server-Timing header. Its door entry is the time we spent on the request, in milliseconds. Over MCP the same clock is timing_ms.server in the answer body.
Some failures come from the network in front of the API. Such an answer has a plain text body and no request-id header. Quote its x-vercel-id header when you report it. The API may not have received the request. You can always send a file write, a file part, or a commit again. Read the current state before you repeat a call that creates or starts something.
Every response says where you stand:
RateLimit-Limit: 60
RateLimit-Remaining: 41
RateLimit-Reset: 19
RateLimit-Policy: 60;w=60RateLimit-Remaining is the calls you have left in the current window. RateLimit-Reset is the seconds until it refills.Retry-After in seconds. Wait that long. A tight retry loop spends the budget it is waiting for.sandbox_exec across fifty sandboxes costs one call.GET /preview/limits.The version is the first segment of the path. /preview is what you call today. It is the pre-launch contract and it can still change. At launch these routes are promoted to /v1 unchanged, and /preview keeps answering as an alias. After that, a breaking change takes a new version segment. An existing version only gains optional fields and optional parameters.
A route on its way out says so on every response for at least 30 days before it stops answering:
Deprecation: true
Sunset: Sat, 29 Aug 2026 00:00:00 GMTIf you write something that must work unattended, check for those two headers and alert on them. The same policy is in the OpenAPI document under x-versioning. The rate limit classes are under x-rate-limits.
Per-minute rates are on the front page. A node bills while it runs or idles in its grace window. A stopped node costs nothing.
create_node and get_node response carries both numbers. rate.node_per_min is your node per minute. rate.per_min is the per-GPU rate.rate.calendar: today's percent off, the schedule in one line, and the next cheaper day with its rate. Batch jobs are already discounted below the interactive rate. A calendar day reaches them only when that day is cheaper still. Today that means Friday moves a job and the weekend does not. submit_job reports which in calendar.moves_jobs_today. A job with nodes has no job discount, so every discounted day moves it, the weekend too.rate.idle_window_minutes and its full cost as rate.idle_window_usd. The older names, rate.minimum_minutes and rate.session_minimum_usd, carry the same values. There is no minimum charge. stop_node, or the in-container POST /v1/stop, ends the window early and closes billing at the stop.create_node, get_node, and cluster quotes carry it in rate.upcoming_increase. submit_job, submit_jobs, and hold_node carry it in upcoming_increase. It gives the time the increase takes effect, a message, the new list rate per GPU and per node, and the new discounts. A job keeps the rate that it started with, so a job that starts after that time pays the new price.autor usage # this month, per node and per batch
autor usage --month 2026-06
autor billing # dunning state, MTD spend, capsget_usage() # this month, per node and per batch
get_usage(month: "2026-06")
get_billing() # dunning state, MTD spend, capsget_usage itemizes the month: a row per node, a jobs block for batch work, and lines for builds, snapshot storage, and sandboxes when they charged. The jobs block groups jobs by sweep label, names the ten priciest jobs, and reports GPU minutes. The sandboxes block splits its charge per tier and per UTC day. See Cost under Sandboxes. Every line adds up to the total at the top, which is the same number /usage shows.in_progress_usd for those, and spend_so_far_usd for the two together. get_billing and your spend cap work from spend so far. On /usage the two figures are labeled Charged and Spend so far./usage.csv. Parse the CSV by header name. New columns are added at the end and existing columns keep their names. Node rows carry kind=node and sandbox rows carry kind=sandbox. /billing is the Stripe portal for invoices, receipts, and the payment method.get_billing() returns the same list as credit_expiries, soonest first. It also returns your ten newest credit grants as credit_grants, each with its kind and its state: active, expired, used, removed, or revoked. An expired grant shows the amount that expired as expired_usd. Admin and billing roles only.get_billing() returns them as amount_due_usd and available_to_spend_usd. Admin and billing roles only.payment_method_ok and card_attached say that a verified card is on file. They do not say that the card paid. get_billing() returns the card's last charge in payment: last_charge_succeeded, last_charge_at, last_charge_failure_code, and consecutive_charge_failures. A card can fail while credit pays your invoices. When the credit runs out, the next unpaid invoice stops new nodes. If the last charge failed, update the card in the billing portal. A new card starts the record again: the fields show only charges on the card you have now. Admin and billing roles only.get_billing() reports the prepaid balance and its expiry in a prepaid block, separate from pooled credits. The same call shows cash received in every shape. Admin and billing roles only.claim_referral_code. Activations and credits earned are listed there and on list_referrals().Compute is paid for from your organization's credit balance. The balance is the limit. Every door that starts compute checks it first: a node, a wake, a job, a sandbox, a research turn, and a snapshot past your free tier. Your card is charged for credit you buy. It is never charged for usage after the fact.
autor billing topup 100buy_credits(amount_usd: 100)A purchase charges your card on file. If your bank must approve each payment, open /billing and use Pay on a card page. The bank shows its approval step there.
With no card on file, a purchase charges nothing. The answer carries payment_url, a Stripe payment page for that amount, and charged: false. Give the link to the person who pays. The credit lands only after they pay. That payment also saves the card, so after it buy_credits charges the card directly and automatic top-up works. A second call for the same amount returns the same open link. A link closes 23 hours after it opens, and an org holds at most 5 open links.
When the balance reaches zero, running work stops at once. Nodes stop and their disks park. Running jobs end like a spend-cap kill, and their checkpoints stay. Queued jobs hold. Sandboxes park. Research missions pause at the turn. Nothing is deleted for non-payment. Buy credit and it all starts again on the next command. On every account, we delete a node that stays stopped with no activity for 60 days, as Stop and wake explains. Usage that ran past zero inside the last minute is paid by your next top-up first.
While the balance is at zero, a request to a sandbox port that you exposed gets a 402 answer. The request does not wake the sandbox. After you add credit, the port answers again within 10 seconds.
Automatic top-up keeps a long run going. When the available balance falls below a threshold, your card on file is charged an amount you choose. The defaults are below $20, add $100. The rule runs at most 25 times per day and never past a daily limit you set. Set it on /billing or from a chat session. The rule needs a card on file. Your first payment at a buy_credits link or on Pay on a card page saves one.
autor billing auto-topup --threshold 20 --amount 100 --daily-limit 300set_auto_topup(threshold_usd: 20, amount_usd: 100, daily_limit_usd: 300)get_billing reports the balance in its balance block: what is available, what is journaled and not yet settled, the burn rate, and the minutes of runway. A refusal for a balance too low names the amount the call needs and how to add credit. Your agent also sees a warning line on node responses 60 and 15 minutes before the balance reaches zero.
Credit that you buy at sign-up, on the billing page, with automatic top-up, or with an agent payment does not expire. Credit that you buy through an agreement with our sales team expires on the date in that agreement. Promotional credit expires 3 weeks after it is granted. Credit that we add because we billed minutes after your node stopped running does not expire. When credit expires, it first pays for usage from before that date that no invoice carries yet, and only the rest expires.
We spend credit earliest expiry first, and credit with no expiry last. We spend credit that we add because we billed minutes after your node stopped running as if it expires when a promotional window ends. That window is 3 weeks long, or 90 days for credit that we added before August 28, 2026.
Accounts that paid by monthly invoice moved to prepaid credit on October 1, 2026. Usage before that date is on your last invoice, paid from credit first and from the card for the rest.
A spend cap counts all usage in the month, including usage that your credit pays for. Credit does not raise a cap, so a cap can stop your nodes while you still have credit. Your credit stays on the balance.
If the card payment for an invoice fails, your org cannot start new nodes until you pay the invoice. If the invoice is still unpaid on the date in the payment-failed email, we suspend the org, and its running nodes stop. An org admin sees the unpaid amount and your credit in the refusal. When your credit covers the invoice, the credit pays it automatically if no card payment for it is in progress. If a card payment waits for approval from your bank, the credit cannot pay the invoice. Then open a ticket with file_ticket.
Your card must pass CVC verification when you add it. A card that does not pass the check stays on file but does not count as a payment method, so automatic top-up cannot use it. Add a different card in the billing portal. A card you add through Apple Pay or Google Pay passes on the wallet's verification. A card you save through Link must pass the check like a typed-in card.
An agent with delegated spend authority can buy prepaid credit for your org, over Stripe's Machine Payments Protocol (MPP). The agent pays with a shared payment token, a single-use grant against a card its principal controls. No card is stored with us, and nobody opens the billing portal.
# post the amount with an org service token; the reply is an
# HTTP 402 carrying a signed payment challenge (-i shows the
# WWW-Authenticate header, which is where the challenge rides)
curl -si -X POST https://autoresearch.sfcompute.com/pay/v1/topups \
-H "Authorization: Bearer $GMN_TOKEN" \
-d '{"amount": "50.00"}'WWW-Authenticate header, mints a shared payment token for the challenged amount, and retries the same request with the payment credential. Stripe publishes these clients for Node and Python.Payment-Receipt header names the Stripe payment. A 202 means the payment is still settling. Check your balance before you pay again.gmnt_ service token with org write scope.Every ceiling in these docs (node counts, object sizes, storage footprint, job knobs) is a limit your agent can read before it hits it:
autor limits # NAME / VALUE / LAYER / SETTABLE / SELF-SERVE / BOUNDS
# BOUNDS names the hard cap when one exists;
# SETTABLE says "lower only" for a row at its cap;
# SELF-SERVE says how far a request raises it at once
autor limits --jsonlist_limits()
→ {limits: [{name: "max_nodes", value: 64, layer: "default",
settable: true, description: "Live GPU nodes per member
(CPU nodes have their own ceiling)."},
{name: "max_cpu_nodes", value: 256, layer: "default",
settable: true, description: "Live CPU nodes per member
(GPU nodes have their own ceiling)."},
{name: "node_scratch_gib", value: 250, layer: "default",
settable: true, hard_max: 550, description: "..."}, ...]}layer that decided it: "default", "org override", "customer override", or "platform".settable: true can be set per org by support, usually raised. An override can also set a limit below the default.hard_max names the platform hard cap. No override can exceed it. Size requests against it.raisable_to names the most an org admin gets at once with request_limit_increase. Members see a raise_note instead.layer: "platform". They are deploy configuration, not per-org levers.An org admin asks for a higher limit with request_limit_increase. Pass the name that list_limits shows, the value in the limit's own unit, and a reason that an operator can read.
autor limits request node_pids_max 16384 \
--reason "three vLLM engines on one node cross 4096 threads"request_limit_increase(limit: "node_pids_max", value: 16384,
reason: "three vLLM engines on one node cross 4096 threads")
→ {outcome: "granted", org: {id: "...", name: "acme"}, limit: "node_pids_max",
requested: 16384, previous: 4096, value: 16384,
note: "Granted. node_pids_max is now 16,384 for acme (...). It applies at
each node's next start or wake. ..."}raisable_to, the org gets exactly the value you ask for, at once. The answer says when the new value applies.get_ticket shows the decision.raisable_to, the call is refused and names the ceiling. To ask an operator for more, pass escalate: true.hard_max, the call is refused, because no one can grant it.org, the org is the one that your workspace bills. By default, an org can make 10 requests per rolling day (limit_requests_per_org_per_day).file_ticket with the limit and the value.A grant program pays for compute that other people run. You fund the program and invite people by email. Each person who accepts gets a one-time compute grant in a separate organization. You see how much of each grant they use. You do not see their files, their nodes, or their work.
A grant always goes to an organization outside yours. To give a teammate a budget inside your organization, use a workspace with its own spend cap.
Org admins run programs. Members with the billing role can read them, and other members cannot see them. You manage programs in the browser only. No MCP tool or CLI command funds a program or sends an invite, so an agent cannot spend your escrow.
An organization can have 3 programs that are not closed. The limit is grant_programs_per_org. To run more, close a program or ask for a higher limit.
If your organization has a priority tier, each grantee organization shares it while its grant has credit left. Your organization and its grantees use one shared pool of queue boosts. The program page shows the tier and how many boosts are left this hour.
Each program pays for its grants from its own escrow. To fund it, use one of the two buttons on the program page. Fund from credit moves credit from your organization's balance into the escrow and charges nothing. If your balance is too low, buy credit first.
Fund with card charges the card on file for the amount. One card payment is $10 to $10,000. An organization that pays by invoice funds from credit.
The program page splits the escrow into funded, reserved, granted, returned, withdrawn, and available amounts. Available is what new invites can still reserve.
Paste email addresses under Invite grantees on the program page, one address per line. Each invite is worth the program's default grant unless you set a different amount. An invite can be worth at most $50,000.
grant_invites_per_program.A group invite puts several people in one shared organization with one shared grant. Use it for a lab or a class.
A grant invite arrives by email with a claim link. Open the link. Then sign in, or create an account, with the address that the invite went to. You do not need a payment method to claim a grant.
The consent page names the sponsor, the amount, and how many days the grant lasts after you accept. It also says what the sponsor can see. Nothing happens until you choose.
The new organization takes its name from the program and your email address, such as Residency 2026 - ada. You are its admin, so you can invite collaborators from /team.
If you signed in to an account that you already had, accepting does not change your active organization. To bill the grant, pick its organization in the switcher at the top of the dashboard, or ask your agent to switch.
# reads your active organization; switch it in the dashboard first
autor billing show --json # the grant blockswitch_active_org(org: "Residency 2026 - ada")
get_billing()
→ {grant: {supported_by: "Acme Research", program: "Residency 2026",
amount_usd: 500, remaining_usd: 312.4,
expires_at: "2027-03-28T17:00:00+00:00", ...}, ...}The grant line on /billing names the sponsor, the amount left, and the expiry date. Admin and billing roles also see the grant in get_billing.
The program page has a row for each grant. The row shows the organization, the invited address or group, and the amount. It also shows how much is used, how much is left, the expiry date, and the status. The used amount includes usage that no invoice has settled yet. Download the same rows as a CSV from the program page.
A sponsor sees how much of each grant is used. It does not see the grantee's files, node names, commands, logs, or work. Some sponsors also see how long the grantee's nodes and jobs wait in the queue, and how many shared boosts they use. When this applies, the consent page and the invite email say so, and so does the grantee's /queue page. Before a person accepts, the sponsor sees only the status of the invite.
file_ticket, and our replies: /tickets. One thread per ticket, with a link you can share.set_retention with logs: false). Each of its replies says that an AI wrote it. A person approves each refund and credit, and each action on your data, before it happens. To reach a person, reply on the ticket.gman://docs/llms.txt, which works from inside a sandbox whose egress policy blocks this site.