Troubleshooting

Start with the stable error code from the router and the read-only operator report:

serve status
serve status --history <release-or-model-id>

Do not retry by switching to another model, release, GPU, or implementation unless you deliberately choose that new identity. The system never does that automatically.

Authentication

401 unauthorized

• Confirm the header is exactly Authorization: Bearer <token>.
• Confirm the token belongs to this serving instance, not a Simpcat API key or legacy FMTTS endpoint.
• Operators rotating tokens should keep old and new values together in SERVING_API_TOKENS until callers move, then redeploy the router secret deliberately.

The router intentionally exposes no unauthenticated health or OpenAPI endpoint.

Target resolution

model_not_found

The path was a syntactically valid model ID with no manifest, or was neither a model ID nor a valid alias. Copy model IDs verbatim, including .json. Use:

curl -H "Authorization: Bearer $SERVING_API_TOKEN" \
  "$SERVING_ROUTER_URL/models?state_filter=all" | jq

When the response also carries "reason": "unpublished", the key names a manifest that exists in the store but was never published: training writes an inference manifest beside every durable step, and those keys differ from a real model ID only by the release-slug suffix. The remedy is serve publish, not a credentials hunt.

alias_unresolved

Inspect the returned chain and query the alias:

curl -H "Authorization: Bearer $SERVING_API_TOKEN" \
  "$SERVING_ROUTER_URL/aliases/<alias-name>" | jq

Typical causes are a missing lower-level alias, wrong level, removed target, or hand-written corrupt record. Repair with an explicit serve alias move; do not add a fallback chain.

model_removed

Unpublication is permanent. Use the recorded replacement if one exists outside the removal marker; do not recreate the old key.

Lifecycle

model_disabled or release_disabled

This is an intentional refusal. Check successor_id or successor, then inspect history before changing state:

serve status --history <target>

Reactivate a model directly only when the withdrawal was a mistake. For a disabled release, first re-verify its deployment, then reactivate:

serve deploy <release>
serve reactivate <release>

Deprecated model is missing from GET /models

That is expected. Deprecated models remain directly servable but are hidden from the default active listing. Query ?state_filter=all.

Publication says release is not active

Candidates and deprecated or disabled releases refuse new models. Deploy a candidate without --no-activate, or publish against a deliberate active successor. Do not reactivate an obsolete release merely to bypass publication policy.

Readiness

not_ready with missing: deployment

The release has no verified deployment record, the record is unhealthy, or its descriptor digest is stale. Run:

serve deploy <release>
serve status

If another identity occupies the app name, ordinary deploy refuses. Confirm the situation and use serve deploy <release> --repair to replace it from the immutable pin.

not_ready with missing: engine_set(H100)

The model is on an artifact-requiring release without a valid exact-architecture engine set:

serve engines build <model-id> --release <release>

Do not deploy Torch as a fallback or build on a different GPU. The artifact must bind the model manifest, release descriptor, builder commit, and declared architecture.

not_ready with missing: state

The router could not read authoritative state. This is a store or credential incident, not a model build. Check router logs, its serving-router secret, RND_SERVING_INSTANCE, storage profile variables, and object-store availability.

Request validation

invalid_params

Use the returned param and rule, then fetch this model’s current spec. Common causes:

• missing required parameter;
• unknown parameter copied from another model;
• invalid enum or range;
• malformed/oversized base64 audio;
• mutually exclusive duration or solver controls;
• solver_atol without solver_rtol;
• Euler without solver_steps;
• output-target count not matching the declared text list;
• insecure or non-Tigris upload URL.

The router validates defaults and enum overrides too. A failure caused by a stored default is a contract bug that requires a new release/model, not client-side message parsing.

Runtime returns invalid_params after router validation

Some checks require decoded audio or model semantics and run inside the runtime. Inspect the message for humans but continue branching on code. Fix the input or the release’s contract/runtime agreement.

Scratch

scratch_not_found

Use the canonical session UUID printed by scratch start. Display names and Modal app names are not accepted as targets.

scratch_expired or scratch_closed with reason: "expired"

Both mean the same expiry; which one you see depends on timing. scratch_expired appears only in the short window (up to ~30 seconds) before the reaper’s next tick closes the session; after that, the same session answers scratch_closed with reason: "expired". Branch on both. Reads do not renew leases. Start a new session; scratch has no reactivation path.

scratch_closed

The session was stopped, promoted, or expired and reaped — the reason field says which, and promoted_to is included when applicable. Start a new session for further changes.

scratch_not_ready

No verified revision or required artifact is current. Check the scratch start/watch terminal for the failed deploy, metadata, snapshot, or engine-build step. A failed revision never becomes current; fix the primary failure and sync again or start a new session.

Watcher does not react to a dependency change

Watch mode follows the declared Python source set. Dependency locks, system packages, image recipes, implementation, and GPU identity require a new session so dependency identity cannot change underneath an existing lease.

Promotion refuses a digest or recomposition mismatch

Promotion is asserting that the release is the code and composition you actually previewed. Typical causes are:

• the release pin omitted or added a serving source change;
• implementation differs, such as Torch scratch to TensorRT release;
• requirements or image recipe differs;
• the selected revision is not the current revision;
• source weights or recipe no longer reproduce the stored draft.

What you can do next depends on when the refusal fired. Digest and identity mismatches are checked before the session’s promotion claim is recorded, so the session is untouched: commit and mint the exact tested source, select the right revision, and promote again. A failure after the claim — a recomposition mismatch or publication error inside the promotion itself — deliberately keeps the claim: retrying the same session, revision, and release resumes idempotently, but promoting that session to a different release (for example one you re-minted to fix the mismatch) is refused as already claimed. In that case, start a fresh session from the same source, or use ordinary publication and treat the result as a new comparison. Do not weaken the equality check.

Capacity and runtime integrity

over_capacity

Retry with backoff using the optional retry_after value in the JSON body. Preserve request identity and avoid multiplying an already large batch.

attestation_mismatch

Treat this as a deployment-integrity incident. The runtime did work under an identity different from the router’s expectation, or a scratch app was replaced. Do not accept or share the returned output. Inspect deployment/app history, then redeploy from pin or start a fresh scratch session.

upstream_failure

The runtime failed before it could return a typed envelope. Common sources are container import/setup failures, missing blobs, image problems, or Modal errors. Inspect the named app’s logs. Fix the failing primary path; there is intentionally no alternate implementation fallback.

blob_missing

A typed runtime check found missing authoritative content. Verify the manifest and its blobs:

cd "$RND_ROOT"
./with_storage_env.sh uv run --project models/fmtts \
  python -m serving.model_store verify <model-id>

Do not rewrite the immutable manifest to point somewhere else. Restore the content under its correct digest from a trusted source or publish a new identity.

upload_failed

The runtime synthesized but could not complete direct delivery. Presigned URLs may have expired, permissions may be wrong, or Tigris may have failed. Because earlier parallel PUTs may already exist even though the batch reports no success, inspect and clean the caller-owned targets before retrying.

Webhooks

A consumer stops receiving events

Delivery is ordered with head-of-line blocking: one failing POST halts that subscription (only) until it succeeds, retried every tick. Check GET /webhooks — a cursor_at far behind the newest event with a stale last_delivered_at means the endpoint is refusing or timing out; the router’s logs name the failing event. Nothing is lost while the consumer is down: delivery resumes from the cursor, and the feed retains ~90 days.

webhooks_unconfigured on registration

The router has no signing key, so it refuses to create a subscription that could only ever receive forgeable unsigned posts. Set SERVING_WEBHOOK_PRIVATE_KEY in the router’s serving-webhook-signing secret and redeploy.

Signature verification fails on the consumer

Verify Ed25519 over {X-Serving-Timestamp}.{raw request body} — the exact bytes received, before any JSON parsing or re-serialization — against the public key from GET /webhooks/signing-key. Clock skew beyond the tolerance window (±300 s recommended) also rejects; check the consumer host’s clock and confirm the configured public key matches the router’s before suspecting anything deeper.

CLI setup

RND_SERVING_INSTANCE is not set

Run through rnd/with_storage_env.sh; RND_SERVING_INSTANCE belongs in the encrypted storage environment beside RND_CHECKPOINT_STORE. Use init-instance only for a genuinely empty, unstamped store.

Modal authentication or lookup fails

Operator commands currently take MODAL_TOKEN_ID and MODAL_TOKEN_SECRET from Simpcat’s encrypted .env.infra. Use the composed serve helper and run npx from the rnd checkout, not inside Simpcat.

Mint rejects dirty serving paths

Commit the implementation source, generic serving source, builder inputs, and installed requirements lock that define the runtime you tested. Unrelated training work does not need to be committed. Mint always pins the current HEAD; to mint a historical commit, check it out first.

Runtime tag is missing or points elsewhere

serve status checks origin, not only local tags. The descriptor remains the identity authority, but the release has lost reachability insurance. Restore the protected runtime/<release> tag to the descriptor’s exact commit; never move it to newer code.