ci(changelog): draft flox/flox release copy with Muse Spark

[djsauble]

Summary

The Changelog release stubs workflow opens a PR for each new flox/flox release with a placeholder entry, and a human rewrites it every time. With this change, scripts/changelog-draft.py drafts the real entry from the release notes with Muse Spark, through one direct request to the Meta Model API. When every automated check passes, the PR opens ready for review, so the maintainer only has to review, approve, and merge.

This replaces the earlier version of this PR, which used anthropics/claude-code-action and needed a CLAUDE_CODE_OAUTH_TOKEN secret. No one needs to run /install-github-app now.

Needs

META_MUSE_CI_API_KEY, which already exists in this repo (it was added for #105). This PR doesn't depend on #105 merging or on MUSE_CI_ENABLED. Without the secret, releases get today's placeholder, and the PR body says The copy was not drafted automatically: the META_MUSE_CI_API_KEY secret is not set.

Design

One new step, Draft release copy, runs scripts/changelog-draft.py (Python stdlib only). A failure never fails the job. It falls back to the placeholder, or, when the API is unavailable, waits for the next daily run.

1. Build the request.
   • Checks the release tag, date, and URL against the shapes a release actually has.
   • The instructions (house rules and the three newest merged flox/flox entries) go in the developer message. They are built only from this repo and the validated tag, date, and URL.
   • The release notes are untrusted, so they travel JSON-encoded in the user message, next to an index of docs pages with their Mintlify anchors and every man page.
   • response_format is a strict JSON schema: {body, reviewer_notes}.
2. Send it to https://api.meta.ai/v1/chat/completions with model muse-spark-1.3-contributor, at the API's default reasoning effort.
   • The model has no tools and runs nothing on the runner. It gets text and returns text.
   • A failed request is retried up to six times within 15 minutes, except for 400, 401, and 403. Redirects are never followed with the key.
   • If the API is still unavailable after the retries and the release is under three days old, the run opens no PR. The next daily run finds the release still new and drafts again. After three days, the release gets the placeholder.
   • The API's error message is logged, except for 401 and 403.
3. Wrap the lines. The model writes one line per paragraph and bullet, and the script wraps them at 72 characters without breaking inline code or links.
4. Check the answer.
   • Rejects (placeholder, with a reason) unsafe MDX and anything shaped like a credential.
   • Flags (PR stays a draft) links whose page or anchor doesn't exist, links outside flox.dev and github.com/flox, and a missing link to the release notes. Anchors are checked with Mintlify's own slug rules.
   • Writes the PR body: it says the copy was machine-drafted, names the model the API ran, and includes the reviewer checklist, the check results, and the model's notes. @ in the notes is neutralized so they can't ping anyone.
5. Insert (changelog-stub.sh, unchanged except for an optional CHANGELOG_BODY_FILE). It still writes the <Update> wrapper and the changelog-id marker, so dedupe is untouched. Placeholder output is byte-identical to before.

Scope stays flox/flox only. Private floxhub release notes never reach a public PR unreviewed, and they never reach the model.

Ready vs. draft. A clean draft opens ready for review; a placeholder, or a draft with flagged links, opens as a draft.

Testing input. Running the workflow by hand with draft_tag set to a flox/flox release tag drafts that release again, whether or not it has an entry, and shows the entry and PR body in the run summary. It opens no PR, and it fails when nothing was drafted.

Testing

Live runs of this workflow on this branch. 64 draft_tag runs across six releases (v1.13.0 to v1.17.0), 57 of which produced an entry. The last three ran on the final commit, rebased onto #105, and all three drafted an entry.

Question	Result
Does it work end to end?	Yes. The key authenticates, the model answers in the schema, and the checks and the insert step pass.
Typos	1 in the first 8 entries, where the model wrapped its own lines ("a Nix sexpression build", exactly on a line break). 0 in the 49 entries since the script took over wrapping. The sample is too small to prove that was the cause.
Formatting	6 of the first 8 entries had unindented bullet continuation lines. 0 of 49 since.
Reasoning effort	No visible effect on quality. high used the same reasoning tokens as the default (about 9,200 per draft). xhigh used about 14,500 and took longer, with the same kinds of small mistakes. Left at the default.
Invented examples	11 of 12 drafts of v1.15.0 suggested a use for hook.on-deactivate that the notes never mention. After the prompt told it to use only the notes' own examples: 0 of 4.
Time and cost	54 s to 2 min 51 s per draft on the final settings. About 59,000 input and 6,000 to 17,500 output tokens, or about one cent at the published Contributor prices.

The API is unreliable. About half of all requests in these runs (50 of 107) came back as HTTP 404: The requested model was not found, for a model that answered other requests in the same minute. It didn't depend on the release, the reasoning effort, the runner's region, or how many runs were in flight, and it tended to repeat within a run. With one attempt, 1 of 9 runs failed; with three, 3 of 19; with six over 15 minutes, 3 of 36. That is why an unavailable API now puts drafting off to the next daily run instead of settling for the placeholder.

I read about half of the entries against their release notes. Claims trace to the notes, with occasional loose wording a reviewer would tighten, such as saying flox publish no longer sends credentials from ssh remotes when only the password is dropped.

Two paths haven't run live, because no release is waiting: the step that opens a real PR, and the deferral to the next daily run. The first is the existing peter-evans/create-pull-request step with three inputs changed (draft, body-path, and the commit message). The second is covered by the offline tests of the script's defer output, plus one added condition on the insert and PR steps.

Other checks:

• Offline tests of the script (87/87 pass) against a local fake of the API: the request's shape, a successful draft, a missing or malformed key, retries on 404, 429, 500 and 504, the retry window, deferral by release age and failure type, 400, 401, a redirect, a refused connection, eight malformed answers, eleven unsafe-MDX and credential cases, six link cases, eight wrapping cases, and six input-validation cases. The key never appears in the log, the outputs, or the PR body.
• Wrapping. On the 41 plain existing changelog entries, unwrapping and rewrapping keeps every word, code span, and link intact, and reproduces the hand wrapping exactly in 29.
• Build checks. With two live entries inserted by changelog-stub.sh, mint validate passes and mint broken-links --check-anchors reports exactly what it reports on main.
• Lint. actionlint is clean apart from one existing SC2129 style hint in the untouched "Determine mode" step.

Sample output (v1.17.0)

An entry from the final batch (run 36946127744), verbatim:

## Debug Nix expression builds in a development shell

You can now open a development shell for a Nix expression build with
`flox develop [<package>]`, which puts the package's build inputs on
`PATH` and loads the stdenv build phases. You drive a failing build by
hand in that shell instead of running a full `flox build`. Pass
`-c/--command` to run one command in the shell instead of entering it.
See [`flox develop`](/man/flox-develop).

## Describe environments with Markdown

You can now document an environment with the toplevel `description`
manifest field. Interactive `flox activate` renders the value as
Markdown. `flox envs` shows its first line. See
[`manifest.toml`](/man/manifest.toml#description-2).

## Also in this release

- `flox init -D` creates a `<current_user>/default` environment on
  FloxHub, equivalent to `flox init -r <current_user>/default`.
- `flox upgrade --dry-run` only reports which packages have upgrades
  available without building or downloading them, so a successful dry
  run no longer guarantees that the upgraded environment builds.
- `flox auth status` exits non-zero and prints a warning when your
  FloxHub token has expired.
- `flox activate --start-services` and the `flox services` commands
  recover automatically when a previous service manager was killed
  without cleaning up its socket.

See the
[v1.17.0 release notes](https://github.com/flox/flox/releases/tag/v1.17.0)
for the full list of fixes.

The model's notes, verbatim as they appear in the PR body:

• Chose flox develop and description as the two headline features; kept flox upgrade --dry-run behavior change in Also list to stay within length.
• Omitted GC protection detail from flox develop paragraph for brevity; omitted minor items on purpose: single keychain prompt, catalog retry delay, flox config -l explanation, and flox install RC guidance text.
• flox init -D is not documented in the provided flox-init man page; trusted release notes that it is equivalent to flox init -r <current_user>/default.

Risks

• A changelog PR can arrive up to three days late. When the API is unavailable, the PR waits for a later daily run. If the API is still unavailable on the third day, the release gets a placeholder PR, and the workflow never drafts that release again, because it never regenerates an existing PR. A maintainer can then run the workflow with draft_tag and paste the entry in. The same applies when the model's answer is rejected, which gets the placeholder at once.
• Meta may train on the request. The Contributor tier is the cheap one because it grants that right. Everything sent is already public: flox/flox release notes and pages from this repo.
• Copy quality. Muse Spark makes small mistakes that the automated checks can't see. The human review of each PR is what catches them. The repo's Vale spell check doesn't help today: it reports nothing for changelog.mdx, because a BlockIgnores rule in .vale.ini skips everything between the page's first { and last }.
• Prompt injection. The model has no tools, and the request holds no secrets. The worst a hostile release note can do is produce bad copy, which still has to pass the checks and a human review.
• Pinned model. muse-spark-1.3-contributor is named in the script. When Meta retires it, drafting fails back to the placeholder until the constant is bumped. The PR body names the model that ran.
• Man pages lag. New man pages arrive through the 07:00 sync PR, so a draft may be unable to link a page that hasn't merged. The model says so in its notes.
• Mintlify slug changes. If Mintlify changes its slug rules, valid anchors could be flagged. That keeps the PR a draft; it can't merge a broken link.
• Muse reviewing Muse. Once ci: add Muse PR reviews with OpenCode #105 is merged and enabled, its PR review will run on these PRs when they open ready, because FloxBot is a user account with write access.

🤖 Generated with Claude Code

https://claude.ai/code/session_0119mtrNDJ1mgkQortTt9JJb

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
flox	🟢 Ready	View Preview	Oct 2, 2026, 1:29 AM

💡 Tip: Enable Automations to automatically generate PRs for you.