Hosted Serving

Hosted Serving turns a training checkpoint and inference code into an immutable, attributable model that can be called through one stable HTTP API. It also provides temporary scratch deployments for the edit–listen–debug loop before code is ready to release.

The system is deliberately split into durable identity and replaceable compute:

caller ─ bearer token ─ router
                           ├── released runtime app
                           ├── scratch revision app
                           └── model store
                               (identity and state authority)

• The model store holds immutable manifests, content-addressed blobs, release descriptors, engine artifacts, and audited mutable state.
• The router is the only public serving endpoint. It resolves aliases, enforces lifecycle and readiness, validates requests, dispatches to Modal, and verifies the returned attestation.
• A runtime app executes one pinned release. The app can be destroyed and rebuilt from its descriptor and the model store.
• A scratch revision app executes source from a worktree. It is temporary, separately identified, and can never masquerade as a released model.

The four identities to know

Model

An immutable manifest containing exact weights, resolved configuration, assets, a generated API contract, lineage, and the runtime release that executes it. The manifest’s object-store key is the model ID:

runs/app-simpcat-ai/<run-id>/checkpoints/step-000200000.inference-fmtts-r2.json

Named models use named/<name>/<label>.json. A model ID never changes meaning and is never reused after unpublication.

Runtime release

An immutable serving program such as fmtts-r2: implementation name, exact git commit, image settings, GPU identity where relevant, and artifact requirements. Minting also pushes runtime/<release> so the commit remains reachable. A new code or identity-relevant configuration means a new release name.

Alias

A mutable, audited name such as fmtts/first-light that resolves to a model ID. Callers normally use aliases; every response still records the resolved immutable model ID and the full alias chain.

Scratch session

A temporary handle over verified worktree revisions. Each successful source update gets a new revision ID, source digest, spec digest, app version, and generation number. Scratch responses carry scratch provenance, not a model ID or release identity.

What the system guarantees

• A released response identifies the exact model manifest, release, source commit, descriptor, and engine set used.
• A scratch response identifies the exact session, revision snapshot, source, dependency recipe, spec, draft manifest, and app version used.
• The router refuses disabled, removed, unready, malformed, or incorrectly attested targets before reporting success.
• Models, releases, engine sets, revision snapshots, and removal markers are write-once. Mutable lifecycle, aliases, deployments, and leases use compare-and-swap records with audit history where applicable.
• Publishing is pull-based and normally reuses existing blobs. Saving a checkpoint does not automatically make it a hosted model.

The guarantee is identity and provenance, not bitwise-identical audio forever. GPU nondeterminism, drivers, and hosted-platform changes can affect numerical output or eventually make old hardware unavailable. Retained outputs are the durable evidence — serve replay bless captures them and serve replay run re-checks them (see Replay drills); the recorded inputs and identities make those outputs attributable.

Current product boundary

The serving router, FMTTS lifecycle CLI, Torch and TensorRT runtimes, scratch workflow, publication, aliases, lifecycle controls, and webhook feed are implemented. Today, use the router HTTP API and python -m fmtts.serving CLI described here.

Caution

Automatic Simpcat model-store mirroring, the Scratch picker, non-shareable scratch previews, and post-promotion recompute UI are not yet on Simpcat’s main branch. The serving side of mirroring is live — signed webhooks plus the discovery API (see HTTP API) — and Simpcat’s consumer is in progress. Publication itself notifies no one and never depends on Simpcat being reachable. The docs describe the working serving system and explicitly mark the integration boundary; they do not treat the planned UI as available.

Choose your path

• Get Started — check the live instance and make an authenticated request.
• Models and Releases — learn the durable object model, lifecycle, aliases, and readiness.
• Scratch Sessions — understand temporary worktree deployments and promotion.
• Preview and Release — take code from a worktree to a published model.
• Operate Serving — inspect status, move aliases, manage lifecycle, and build TensorRT artifacts.
• HTTP API — call released models, scratch sessions, discovery endpoints, and webhooks.
• CLI Reference — look up every user-facing command and option.