Skip to content

fix(megatron): restore saved scheduler progress without accumulation - #2214

Draft
hershg wants to merge 1 commit into
NovaSky-AI:mainfrom
hershg:codex/glm53-matched-20260916-01a0a846
Draft

hershg wants to merge 1 commit into
NovaSky-AI:mainfrom
hershg:codex/glm53-matched-20260916-01a0a846

Conversation

@hershg

@hershg hershg commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Restore the saved Megatron scheduler counter when loading a checkpoint into an already-used worker. Core loads progress by adding the saved step count; reset the existing counter first so repeated restores do not accumulate steps.

The runtime change is two lines. A regression exercises the production loader with the real Core scheduler, advances it past the saved state, and restores twice. The equipped Megatron CI lane runs that test.

Validation

uv run --isolated --no-project ../cpu-env/bin/python -m pytest -q tests/backends/skyrl_train/distributed/test_scheduler_restore.py

1 passed. Ruff, Black, secret checks, and independent source review passed. The production fix and test match the deployed GLM image source 6538c46a; this PR makes no GLM GPU qualification claim.

Signed-off-by: Hersh Godse <hersh@trajectory.ai>
@hershg
hershg force-pushed the codex/glm53-matched-20260916-01a0a846 branch from 37d8217 to e28e6e3 Compare September 17, 2026 04:50
@hershg hershg changed the title Qualify matched GLM file and native LoRA numerics fix(megatron): restore saved scheduler progress without accumulation Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant