Scratch Sessions

A scratch session is the fast path for trying inference code on Modal before it is committed or released. It binds one immutable weight source and optional composition recipe to a sequence of verified worktree-source revisions.

Scratch is not a draft release and not a mutable model. Its namespace, identity block, lifecycle, and app names are deliberately separate from released serving.

Start a session

Use a training selection or an existing model ID:

serve scratch start <run>/120000 --name "$USER/solver-test" --watch

serve scratch start \
  runs/<scope>/<run>/checkpoints/step-000120000.inference.json \
  --name "$USER/normalization-test" --watch

Selections such as <run>/last, <run>/best, <run>/step-120000, and <run>/120000 are resolved once when the command starts. The session stores the exact source manifest key and digest and never consults the mutable name again.

Useful options:

--implementation torch                # default, intended fast iteration path
--implementation tensorrt             # builds revision-scoped engines
--gpu H100                             # exact architecture label
--recipe averaging=ema_h1pct,dtype=bf16
--name grant/experiment                # display only; UUID stays canonical
--watch                                # continue polling source changes

Caution

A TensorRT scratch session rebuilds its entire engine set on every revision — roughly 25 minutes of H100 per source save, because the builder cache is partitioned by source digest and any watched edit invalidates it. Iterate with Torch unless the change under test is TensorRT-specific, and think twice before combining --implementation tensorrt with --watch.

TensorRT scratch also builds engines from a stored manifest, so it cannot be combined with --recipe when starting from a run/step: a recipe composition has no stored manifest. Start from a published model of the composition you want, or scratch the default recipe.

The command prints the canonical session UUID, resolved source, generation, and revision UUID. Scripts and API calls must use the session UUID, never the display label or Modal app name.

What activation verifies

Starting or syncing a revision follows one ordered path:

1. Re-read and digest-verify the recorded weight source, and construct a draft model manifest and generated spec from it.
2. Hash the implementation’s source set.
3. Deploy a fresh, revision-specific Modal app.
4. Verify the app’s baked identity metadata.
5. Write an immutable revision snapshot.
6. Build and verify revision-scoped artifacts when the implementation requires them.
7. Compare-and-swap the session’s current pointer.

The session never points at an app that skipped verification. Requests already dispatched to the previous revision may finish there; their attestations continue to identify that older revision correctly.

Watch, detach, and reconnect

With --watch, changes in the FMTTS and generic serving Python source sets deploy a new generation:

verified g2 revision <uuid>
verified g3 revision <uuid>

Pressing Ctrl-C detaches the watcher. It does not close the session or stop its current verified app. Reconnect later with:

serve scratch watch <session-uuid>

The watcher tracks the implementation’s declared source files. Dependency locks, image recipes, system packages, implementation choice, and GPU identity are not an in-place session switch; start a new session after changing them.

Lease and cleanup

An invocation or successful source update extends the session’s sliding lease to one hour. Reading or listing the session does not. Modal may scale GPU containers to zero much sooner; lease lifetime and GPU warmth are independent.

After expiry, the router refuses new calls — as scratch_expired in the short window before the background reaper’s next tick, then as scratch_closed with reason: "expired" once the reaper closes the session; branch on both. The reaper also stops revision apps asynchronously. Starting a new session is the recovery path—scratch has no reactivation state.

List or explicitly close sessions:

serve scratch list
serve scratch stop <session-uuid>

Stopping is terminal. Ctrl-C on a watcher is only a detach; scratch stop closes the authoritative record and eagerly attempts to stop every revision app.

Invoke a scratch revision

Fetch the current spec:

GET /scratch-sessions/<session-uuid>

Send the same request envelope used for released models:

POST /scratch-sessions/<session-uuid>/invoke

The response includes identity.provenance.kind = "scratch", the session’s descriptive based_on source, and a verified scratch attestation. It intentionally has no released model_id, release, or release commit. See HTTP API.

Caution

The current Simpcat UI does not yet provide its planned Scratch picker or preview/recompute workflow. Use the router API directly and treat scratch outputs as development artifacts, not shareable released-model results.

Promotion

Promotion publishes an ordinary immutable model only when the selected scratch revision matches the release pin’s:

• implementation;
• serving source digest;
• dependency-recipe digest;
• recorded weight source and recipe;
• recomposed model weights.

serve publish --from-scratch <session-uuid> --runtime fmtts-r7

Use --revision <revision-uuid> to select a non-current revision and --named NAME/LABEL when the source is a named model.

Promotion does not publish mutable code and does not relabel scratch audio. It publishes from the immutable recorded source against a real release, records the revision-to-model mapping, and closes the session.

If you iterated with Torch scratch but intend to serve a TensorRT release, the implementations intentionally do not compare equal. Publish the same weight source normally against the TensorRT release and treat results as a new comparison, not as promotion of the Torch preview.