Resuming and Forking Runs

A training job that stops does not have to become a new run. SimpCat gives you two ways to continue from existing state, with different semantics:

• Resume continues the same run: same run ID, same charts, same config record. Use it when the process died (preemption, crash, node failure) and you are picking up exactly where it left off.
• Fork starts a new run that begins at a checkpoint of a parent run. Use it when you are changing something — a learning rate, a data mix, any hyperparameter — and want the divergence recorded as its own experiment.

The rule of thumb: same config → resume; changed config → fork.

Resuming

Resume is on by default. simpcat.init() records the run ID in run.json under the run-data directory (./.simpcat unless you pass dir=), and the next init() in the same directory continues that run:

run = simpcat.init(project="llm", config=config)
# ...crash, requeue, restart...
run = simpcat.init(project="llm", config=config)  # same run, steps continue

The resume argument controls the behavior explicitly (the mode strings match W&B):

Mode	Behavior
(omitted) / "auto"	Resume the run recorded in run.json if present, else create a new run.
True / "allow"	Same as auto.
"must"	Fail unless there is a run to resume.
"never"	Fail if run.json already exists.
False	Always start a fresh run.

On resume the SDK restores the step counter, config, summary values, and metric definitions from the server, so implicit steps continue where they stopped.

Resuming from a training checkpoint

The run directory is convenient local state, but it does not need to be the only copy of the run identity. Save run.id in each training checkpoint. After loading a checkpoint, pass that ID with resume_from:

checkpoint = load_checkpoint()
run = simpcat.init(
    project="llm",
    config=config,
    dir=run_directory,
    resume_from=checkpoint.get("simpcat_run_id"),
)

When the checkpoint contains an ID, SimpCat resumes that run even if run_directory is new or its local state was lost. When the value is None, the normal automatic behavior applies: SimpCat uses the run recorded in the directory if one exists, or creates a new run otherwise.

resume_from requires online mode because the reporter must claim the existing run and restore its server-side step and generation before accepting new metrics. Offline mode rejects it instead of starting a disconnected local attempt under the checkpoint’s run ID.

Config on resume

The stored config is the base layer and the config you pass to init() is overlaid on top: new keys are added, and keys you pass with a different value overwrite the stored value in place.

Because that rewrites the recorded provenance of every step logged before the resume, SimpCat warns (and records a run diagnostic) when a resume changes an existing config value:

UserWarning: simpcat: resume changed stored config value(s) (lr: 0.001 -> 0.01); ...

The new value still wins — the warning never blocks a recovering job. If the change is intentional and you want the same run anyway, pass allow_val_change=True. If you are actually starting a new experiment from a checkpoint, fork instead.

Forking

fork_from starts a new run whose step counter begins at a checkpoint of a parent run. The parent stays untouched, and the new run records parentRunId and forkStep:

run = simpcat.init(
    project="llm",
    config={**config, "lr": 1e-4},   # the change you're experimenting with
    fork_from="<parent run ID>",      # canonical run ID, shown on the run page
    fork_step=50_000,                 # optional — defaults to the parent's latest step
)

• fork_from takes the canonical SimpCat run ID (copy it from the parent’s run page).
• fork_step is optional. Omit it to fork from the parent’s latest step; pass it to fork from an earlier checkpoint. It must not exceed the parent’s current step.
• The forked run inherits the parent’s config as a base; the config you pass overrides it — the same merge as resume, but recorded on a fresh run so both sides of the comparison stay faithful.
• fork_from cannot be combined with resume=True; a forked run is a fresh continuation, not a resumption.

Choosing between them

Situation	Use
Job preempted/crashed, restarting identically	resume (the default)
Continuing training with more epochs, same config	resume
Lowering the LR after a divergence, comparing against the original	fork
Sweeping variations from a common warm-start checkpoint	fork (one per variation)
Re-running from scratch	resume=False