Operate Serving

Operator commands use the model-owned CLI because it supplies the implementation table and model-specific publication logic to the generic serving library. Examples assume the serve helper from Get Started.

Inspect the instance

serve status

The report includes:

• release state, implementation, pinned commit, deployment health, app name, successor, and remote runtime-tag problems;
• model state, readiness failures, removal markers, and successors;
• inference manifests without lifecycle records, normally legacy auto-saves rather than published models;
• active scratch sessions and lease expiry.

Inspect one complete lifecycle history:

serve status --history fmtts-r7
serve status --history <model-id>

The first audit entry is creation (null → candidate for a release, null → active for a model). Every later entry records endpoints, operation, actor, time, and optional reason.

Manage aliases

Create a level-0 alias over a model:

serve alias set fmtts/best --level 0 <model-id>

Build a higher-level product alias over an existing alias. Alias targets above level 0 use the literal alias: prefix:

serve alias set fmtts/prod --level 1 alias:fmtts/best

Move an alias without changing its level:

serve alias move fmtts/best <new-model-id> --why 'listening review passed'

The router validates every descending hop and returns the full chain. It never follows a broken chain to a fallback target.

Deprecate

Deprecation hides a target from the default model listing but preserves existing direct and alias-based traffic:

serve deprecate <model-id> --successor <replacement-model-id> \
  --why 'superseded'

serve deprecate fmtts-r6 --successor fmtts-r7 \
  --why 'fleet moved to r7'

deprecate accepts multiple targets in one command. A deprecated release continues to serve its existing models but refuses new publication.

Disable

Disable is the deliberate, fail-closed traffic control:

serve disable <model-id> --successor <replacement-model-id> \
  --why 'bad checkpoint'

serve disable fmtts-r6 --successor fmtts-r7 \
  --why 'incorrect inference behavior'

Disabling one model leaves sibling models on its release untouched. Disabling a release makes every bound model unroutable and marks the deployment record unhealthy before changing lifecycle state, so an interrupted operation cannot later reactivate on stale verification.

The command does not currently promise to stop the provider app. Router lifecycle/readiness enforcement is authoritative; stop old Modal apps separately when operational cleanup requires it.

Reactivate

A mistakenly disabled or deprecated model can return directly to active:

serve reactivate <model-id>

A deprecated release can reactivate while its verified healthy deployment record remains current:

serve reactivate fmtts-r7

A disabled release’s deployment record is intentionally unhealthy. Re-verify it first, then reactivate:

serve deploy fmtts-r7
serve reactivate fmtts-r7

Redeploying a non-candidate writes a healthy record but does not itself change lifecycle state. This two-command recovery is intentional: code identity is re-verified before the explicit lifecycle reversal.

Unpublish

Unpublication is irreversible for the model ID and requires disabled state:

serve disable <model-id> --why 'retiring model'
serve unpublish <model-id> --why 'retention decision'

The second command writes a permanent removal marker. Routers and aliases refuse the target, listings omit it, and the ID can never be reused even if someone deletes the original manifest out of band.

Unpublication does not reclaim bytes: the manifest object remains in the store and stays a GC root, so its blobs remain pinned. Actual reclamation is the blob GC’s job — a daily Modal cron (serving.gc_app) that marks live references from every root manifest, publishes the condemned set, fences against concurrent writers via pins, and sweeps. It runs dry (reporting what it would delete) unless the serving-blob-gc secret sets BLOB_GC_DELETE=1.

Trim training checkpoints

Training cleanup uses the generic model-store CLI. It refuses to delete a published model key:

cd "$RND_ROOT"
./with_storage_env.sh uv run --project models/fmtts \
  python -m serving.model_store trim <run> --keep 3

./with_storage_env.sh uv run --project models/fmtts \
  python -m serving.model_store verify <manifest-key>

./with_storage_env.sh uv run --project models/fmtts \
  python -m serving.model_store delete-step <run> <step>

trim works on training manifests. Disable and unpublish hosted models separately; do not use training cleanup as lifecycle control.

Build or import TensorRT engines

Build from the release’s pinned code on its exact declared architecture:

serve engines build <model-id> --release fmtts-r7-trt

Use --force to bypass the builder Volume cache. The resulting bytes must still agree with an existing immutable engine-set manifest; nondeterministic different output is a different artifact identity, not an in-place replacement.

Import an already-built local engine directory only when you can name the exact commit that built it:

serve engines import /path/to/engine-set <model-id> \
  --release fmtts-r7-trt \
  --source-commit <40-hex-sha> \
  --gpu-arch H100

The importer checks --source-commit against the release pin first — a wrong commit fails before any multi-gigabyte upload. It then parses and verifies the directory with the runtime’s own engine-manifest parser, uploads it, and checks model weights and assets, the required component set, the release binding, and the actual build GPU before minting the engine-set manifest.

Deploy the router

The router is the instance’s one standing service and its only public HTTP surface. It deploys separately from release apps:

cd "$RND_ROOT/models/fmtts"
npx --yes @dotenvx/dotenvx run \
  -f "$SIMPCAT_ROOT/.env.infra" -fk "$SIMPCAT_ROOT/.env.keys" -- \
  ../../with_infra_env.sh \
  uv run --extra serve modal deploy -m serving.router_app

It reads two Modal secrets, one pair per instance:

• serving-router: RND_CHECKPOINT_STORE plus its AWS credentials (read/write — the router CASes scratch leases and webhook records), RND_SERVING_INSTANCE (the router refuses to start without it and verifies it against the store’s instance.json), and SERVING_API_TOKENS (comma-separated shared bearer tokens — keep old and new together during a rotation, then redeploy).
• serving-webhook-signing: SERVING_WEBHOOK_PRIVATE_KEY, the Ed25519 seed signing outbound webhook posts (also in rnd/.env.infra). Consumers verify with the public key from GET /webhooks/signing-key, so onboarding never hands over a secret. Webhook registration refuses on a router without it.

Rotating either secret’s contents requires a redeploy to take effect. The deploy itself fails if a named secret does not exist, so the router can never come up half-configured.

Replay drills

Replay drills are how “retained outputs are the durable evidence” becomes an operation. Bless a reference answer, then re-check it later — after a redeploy, a platform update, or on a schedule:

serve replay bless <model-id> --request request.json
serve replay run                  # every blessed pair on the instance
serve replay run --model <model-id>

bless sends the request through the live router, stores the request verbatim with digests of the returned audio under replays/, and records the attestation. run re-sends each blessed request and classifies the outcome:

Classification	Meaning
match	Byte-identical results.
platform_drift	Same identity attested, different bytes — expected across driver or hardware changes.
model_change	The target resolves to a different model than was blessed.
identity_failure	Attestation mismatch — a deployment-integrity incident.
missing_artifact	The blessed pair or its target is gone.

Both commands need SERVING_ROUTER_URL and SERVING_API_TOKENS in the environment (the serve helper supplies them). replay run exits non-zero on any finding, including platform_drift — wire it into a scheduled job only with that in mind, or a routine driver update will page you.

Operate from CI

Every lifecycle operation above (mint, deploy, publish, deprecate, disable, reactivate, unpublish, status) is also exposed as a workflow_dispatch in rnd’s serving-operations.yml, for operators who prefer not to run from a workstation. Scratch deliberately has no CI mirror — it exists to serve a local worktree.

Redeploy and repair

An idempotent redeploy is safe when the existing app reports the same identity:

serve deploy fmtts-r7

If an app name is occupied by a different identity, ordinary deploy refuses. Replace it from the immutable pin with the explicit audited repair flag:

serve deploy fmtts-r7 --repair

--repair does not weaken verification. It only authorizes replacement and adds a same-state repair audit entry after the new app verifies.

Subscribe to changes

The router delivers the event feed to webhook subscriptions in order, at-least-once, from a durable per-subscription cursor: a failed post retries from the same position every tick, a router restart resumes instead of replaying or skipping, and a consumer can be down for anything up to the feed’s ~90-day retention without losing events. Posts are Ed25519-signed (verify key at GET /webhooks/signing-key); register with backfill: true to replay the retained feed as an initial sync. Consumers deduplicate on operation_id and keep a slow reconciliation poll against GET /models as the backstop. See HTTP API.