Skip to content

test: every external field → fields= (al.MassField); one galaxy-attached legacy regression [draft until release] - #322

Merged
Jammy2211 merged 2 commits into
mainfrom
feature/mass-field-workspace-sweep
Sep 19, 2026
Merged

Jammy2211 merged 2 commits into
mainfrom
feature/mass-field-workspace-sweep

Conversation

@Jammy2211

Copy link
Copy Markdown
Collaborator

Summary

DO NOT MERGE until release — this PR needs PyAutoGalaxy #621 (ag.MassField) and PyAutoLens #742 (Tracer(fields=), analysis fields slot, model_util.mass_field_from, LOS fields_from, COOLEST 1:1) on PyPI. The workspace follows the released stack, not main; CI here is green only because the smoke workflow clones the library main branches. Opened as a draft for that reason; gh pr ready + /prm after the release.

Phase 3 of the MassField epic (PyAutoMind draft/feature/autogalaxy/mass_field_epic.md), re-scoped on the human's 2026-09-17 ruling: the user-facing API from here on is fields in the model as a separate thing. Every ExternalShear / MassSheet / ExternalPotential that was attached to a Galaxy in autolens_workspace_test now lives in an al.MassField in its own fields= slot of the model (fields=af.Collection(field=af.Model(al.MassField, redshift=…, shear=…))) and of the tracer (al.Tracer(galaxies=[…], fields=[field])). SLaM stages chain fields=<result>.model|instance.fields; results are read as result.instance.fields.field.shear; aggregator paths are fields.field.shear.<param>. Numerically identical (deflections sum over planes — parity asserted, see Test Plan); committed datasets untouched. tracer.galaxies never holds a field, so positional galaxy access is unaffected (the multi_galaxy/ shear_galaxy galaxy leaving the list is the one place indices moved; re-checked). A migrated example composes a different model and therefore gets a new result identifier; old output/ folders are not resumed. The library keeps al.Galaxy(shear=…) working and unwarned for users' own scripts.

Heart readiness at ship: RED — "install verification FAILED (testpypi; checks F)"; "release validation FAILED (stage integrate)" (+ YELLOW manifest drift). Neither concerns this repo; the human acknowledged these exact reasons on 2026-09-18 for commit/push/draft PR-open only. Merge stays human and release-gated.

The one script allowed to attach a field to a Galaxy is the new scripts/misc/mass/galaxy_attached_legacy.py (the human's "maybe an autolens_workspace_test integration test"): tracer parity vs its MassField twin (deflections 4.4e-16, convergence 0, potential 1.8e-15) and the workspace-side identifier pin 35ebe9353118bcc0c7b2d577ce2639ee for the galaxy-attached lens+shear/source model (computed from this repo's config/; the library's own pin of the same model under its shipped config is fef2697b5c32ba56bb18a7baecb7b0f6 — identifiers depend on the active config). coolest_round_trip.py keeps its legacy galaxy-attached case as phase 2's legacy-peel regression. Two judgement calls worth a glance: composition_mge.py's identifier constant is re-anchored because its model now composes the shear as a field (a different model by design), and its unique_prior_paths round-trip is compared as a set because from_dict re-orders prior ids for any model with a second top-level collection (reproduced with extra_galaxies=; pre-existing PyAutoFit behaviour, identifier unaffected). Three files (multi_dataset/simulator.py, multi_galaxy/model_fit.py, multi_galaxy/jax_likelihood/lp.py) were already non-black at HEAD and were not reformatted. Substructure tests build per-plane MassFields; substructure_util.galaxies_to_halo_arrays accepts them unchanged.

Scripts Changed

  • scripts/imaging/jax_grad/delaunay.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_grad/knn.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_grad/lp.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_grad/mge.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_grad/pixelization.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_grad/regularization.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/delaunay.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/delaunay_mge.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/delaunay_near_caustic.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/lp.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/mge.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/mge_group.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/multipole.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/rectangular.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/rectangular_dspl.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/rectangular_dspl_rtu.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/rectangular_mge.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/rectangular_mge_rtu.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/rectangular_rtu.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/jax_likelihood/smbh.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/simulator/dspl.py — al.MassField(redshift, shear=...) passed as Tracer(fields=[field]) (dataset bit-identical)
  • scripts/imaging/simulator/simple.py — al.MassField(redshift, shear=...) passed as Tracer(fields=[field]) (dataset bit-identical)
  • scripts/imaging/substructure/subhalo.py — per-plane mass_sheet galaxies → per-plane MassFields in fields=
  • scripts/imaging/substructure/test_batched_simulate.py — per-plane mass_sheet galaxies → per-plane MassFields in fields=
  • scripts/imaging/substructure/test_scan_multiplane.py — per-plane mass_sheet galaxies → per-plane MassFields in fields=
  • scripts/imaging/substructure/test_simulate_e2e.py — per-plane mass_sheet galaxies → per-plane MassFields in fields=
  • scripts/imaging/visualization/modeling_visualization_delaunay_jit.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/visualization/modeling_visualization_jit.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/imaging/visualization/modeling_visualization_rectangular_jit.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/datacube/delaunay.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/datacube/rectangular.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/datacube/shared_preloads.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/jax_grad/gradient.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/jax_likelihood/delaunay.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/jax_likelihood/delaunay_mge.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/jax_likelihood/lp.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/jax_likelihood/mge.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/jax_likelihood/mge_group.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/jax_likelihood/rectangular.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/jax_likelihood/rectangular_dspl.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/jax_likelihood/rectangular_mge.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/jax_likelihood/rectangular_sparse.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/nufft.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/simulator/dspl.py — al.MassField(redshift, shear=...) passed as Tracer(fields=[field]) (dataset bit-identical)
  • scripts/interferometer/simulator/simple.py — al.MassField(redshift, shear=...) passed as Tracer(fields=[field]) (dataset bit-identical)
  • scripts/interferometer/visualization/modeling_visualization_jit.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/interferometer/visualization/visualization_jax.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/misc/database/scrape/general.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/misc/database/scrape/multi_analysis_jax.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/misc/database/scrape/scaling_relation.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/misc/database/scrape/slam_general_jax.py — SLaM stage chains fields=<result>.model|instance.fields instead of a galaxy shear= kwarg
  • scripts/misc/database/scrape/slam_multi_one_by_one_jax.py — SLaM stage chains fields=<result>.model|instance.fields instead of a galaxy shear= kwarg
  • scripts/misc/database/scrape/slam_pix_jax.py — SLaM stage chains fields=<result>.model|instance.fields instead of a galaxy shear= kwarg
  • scripts/misc/interop/coolest_round_trip.py — legacy galaxy-attached case kept (phase 2 legacy-peel regression, post-import checks follow the shear into tracer_back.fields); new MassField case
  • scripts/misc/jax_assertions/delaunay_nn_caps.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/misc/jax_assertions/delaunay_walk.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/misc/latent/latent_integration_smoke.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/misc/latent/latent_integration_smoke_jax.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/misc/mass/galaxy_attached_legacy.py — NEW — the one script allowed to attach a field to a Galaxy: tracer parity vs MassField twin + workspace-side identifier pin 35ebe9353118bcc0c7b2d577ce2639ee + both models fit under test mode
  • scripts/misc/mass/sheets.py — new __MassField__ section: field tracer equals the direct profile sum
  • scripts/multi_dataset/jax_likelihood/dataset_model.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/jax_likelihood/delaunay.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/jax_likelihood/delaunay_mge.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/jax_likelihood/lp.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/jax_likelihood/mge.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/jax_likelihood/mge_group.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/jax_likelihood/rectangular.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/jax_likelihood/rectangular_mge.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/jax_likelihood/rectangular_mge_rtu.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/jax_likelihood/rectangular_rtu.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/jax_likelihood/shared_preloads.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/simulator.py — al.MassField(redshift, shear=...) passed as Tracer(fields=[field]) (dataset bit-identical)
  • scripts/multi_dataset/visualization/imaging.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_dataset/visualization/interferometer.py — galaxy-attached shear → fields=af.Collection(field=af.Model(al.MassField, ...)); prose names the MassField
  • scripts/multi_galaxy/composition_mge.py — shear → fields=; identifier pin re-anchored 5a3c480d… → b99831e6… (model changed by design); unique_prior_paths round-trip compared as a set (PyAutoFit from_dict id-ordering, not MassField-specific)
  • scripts/multi_galaxy/jax_likelihood/lp.py — shear → fields= (JAX pytree leg; ENV: jax full_datasets kept; literal likelihood unchanged)
  • scripts/multi_galaxy/model_fit.py — shear → fields=; prior-count assertions now require equal lens counts and a 2-prior fields.field.shear

Upstream PR

Test Plan

  • Local smoke subset against library main (PyAutoGalaxy 33714b80 / PyAutoLens 71973806), run_python.py --list with the smoke profile (PYAUTO_TEST_MODE=2), from the task worktree:

autolens_workspace_test subset — 13/13 PASS, 0 FAIL, 0 TIMEOUT (the two jax_likelihood/lp.py entries declare ENV: jax and ran with JAX: NumPy == JIT likelihood 626.2235846074775 and 1206.80145178, the pre-migration literals):

script result s
imaging/jax_likelihood/lp.py PASSED 32.2
imaging/jax_likelihood/delaunay.py PASSED 62.7
imaging/jax_likelihood/mge.py PASSED 38.0
interferometer/jax_likelihood/rectangular.py PASSED 48.0
multi_dataset/jax_likelihood/mge.py PASSED 22.2
multi_galaxy/composition_mge.py PASSED 3.6
multi_galaxy/model_fit.py PASSED 6.8
multi_galaxy/jax_likelihood/lp.py PASSED 22.7
misc/jax_assertions/delaunay_walk.py PASSED 27.7
misc/latent/latent_integration_smoke.py PASSED 18.8
misc/mass/sheets.py PASSED 3.1
misc/mass/galaxy_attached_legacy.py PASSED 5.2
misc/interop/coolest_round_trip.py PASSED 4.2
  • Tracer parity (pre-change galaxy-attached vs post-change fields=): imaging/simulator/simple.py tracer max abs diff 0.0; galaxy_attached_legacy.py and misc/mass/sheets.py assert field-vs-galaxy parity to ~1e-15.
  • grep -rnE "al\.Galaxy\(.*(shear|mass_sheet|potential)=" and grep -rn shear_galaxy over scripts/ return exactly the sanctioned sites: misc/mass/galaxy_attached_legacy.py (×2) and the legacy case in misc/interop/coolest_round_trip.py (AST walk over every Galaxy(...) / af.Model(al.Galaxy, ...) call).
  • No notebooks/ in this repo; black --check clean on 74 of 77 changed scripts (3 pre-existing non-black files untouched).
  • CI smoke (runs against library main)
  • Merge only after the PyAutoGalaxy + PyAutoLens release is on PyPI

Closes PyAutoLabs/autolens_workspace#559 (both PRs of the task reference it; the issue closes at /prm).

Generated by the PyAutoLabs agent workflow.

🤖 Generated with Claude Code

…y-attached legacy regression kept

Phase 3 of the MassField epic (autolens_workspace#559), re-scoped 2026-09-17:
parity, JAX-likelihood, multi-dataset, substructure and interop scripts compose
the shear / sheets as an al.MassField in `fields=`. misc/mass/galaxy_attached_legacy.py
is the single script allowed to attach a field to a Galaxy (tracer parity +
workspace-side identifier pin); coolest_round_trip.py keeps its legacy-peel case
and gains a MassField case. Needs the PyAutoGalaxy #621 + PyAutoLens #742
release — draft PR, pending-release.

Refs PyAutoLabs/autolens_workspace#559

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…pace

PyAutoLens #744 made the model's `fields=` slot accept a bare `al.MassField`
alongside a collection. This adopts the flat form everywhere in this repo, so a
model now reads

    fields=field

instead of `fields=af.Collection(field=field)`, and prior paths read
`fields.shear.gamma_1` instead of `fields.field.shear.gamma_1`.

Sites. An AST walk of the branch tip (63c3323) found 66 model-side
`fields=af.Collection(field=...)` sites across 64 files. 65 are migrated here:
63 in the bulk pass (61 files) and 2 in the delicate files
(`multi_galaxy/composition_mge.py`, `multi_galaxy/model_fit.py`). The 66th is
deliberately retained -- see below. A re-walk of the tree confirms the census is
conserved at 103 `fields=` sites overall (66 -> 65 flat + 1 retained, 19
attr-chain, 16 tracer list-literals, 2 tracer `all_fields`), that no file mixes
the two forms, and that `fields.field` no longer appears anywhere in the repo,
docstrings included.

`multi_galaxy/composition_mge.py`. Three coupled edits: the model-composition
site, the path-grouping assertions at :159-:170 (the second path element under
`fields` is now the mass profile `shear`, with no intervening collection key),
and the identifier pin at :263. The pin was RECOMPUTED, not adjusted:

    OLD  b99831e66dd27eee314113e8e58235b6   (collection form)
    NEW  29f82bd3de24984b94657c328b64c3be   (flat form)

computed with the cwd at this repository's root (`autolens_workspace_test/`, the
same cwd every script here runs from), because an identifier is a function of the
model AND the active PyAutoFit configuration. The docstring at :237 now records
that. This script is on the smoke gate, so a wrong pin would go red immediately;
it passes (3.0 s) in a 32/32 local smoke run.

`misc/mass/galaxy_attached_legacy.py`. Its galaxy-attached pin
`35ebe9353118bcc0c7b2d577ce2639ee` is UNCHANGED -- that is the backwards-
compatibility regression and must not move. Its second model (:189) deliberately
KEEPS the `fields=af.Collection(field=...)` form: with every other model in
`autolens_workspace` and `autolens_workspace_test` now flat, it is the
workspaces' last regression witness that the library still accepts a collection
in that slot. Only a docstring saying so was added.

Left untouched. The 19 chained `fields=<result>.{model,instance}.fields` stages
in the four `misc/database/scrape/*_jax` SLaM scripts (chaining is shape-
transparent and carries the flat form unchanged); the 16 tracer-side
`al.Tracer(fields=[field])` list literals; the 2 genuine multi-field tracer sites
(`all_fields`); and `misc/interop/coolest_round_trip.py`, whose `.fields[0]`
reads are tracer-side list indexing.

Identifiers. Flat models compose differently, so they write to new `output/`
directories. Existing trees keyed on the old identifiers are orphaned and are not
resumed; no pin is aliased to an old value and no directory is renamed.

Phase 3 of the epic (the galaxy-attached -> collection migration) is also
unmerged on this branch, so this branch takes the repository from the
galaxy-attached form straight to the flat form and never ships the intermediate
collection form from here.

Verification (local stack: PyAutoLens 478213e78, PyAutoGalaxy 90e757d3,
PyAutoFit 7c0e79aab):
- Smoke: 32/32 PASS (979 s, `profile_smoke.yaml`), including `composition_mge.py`
  and both latent integration smokes.
- `misc/latent/latent_integration_smoke_jax.py` -- the script whose CI leg was red
  on this PR before PyAutoGalaxy 90e757d3 -- PASSES under the flat form (56.4 s),
  writing exactly one `files/latent/latent_summary.json` with all 8 latents finite
  and non-zero.
- Chained stages: `misc/database/scrape/slam_general_jax.py` hand-run (these four
  scripts are SLOW-skipped in `no_run.yaml` and have no automated coverage).
  All five stages carry a top-level `fields` under the flat form, with the
  free/fixed split intact: `source_lp[1]` (seed), `source_pix[1]`
  (`.model.fields`) and `mass_total[1]` (`.model.fields`) show
  `fields  MassField (N=2)` with the seed's own priors propagated;
  `source_pix[2]` and `light[1]` (`.instance.fields`) show
  `fields  MassField (N=0)`.
- Equivalence: `prior_count` unchanged (21 == 21) and prior-path sets equal after
  normalising `fields.field.` -> `fields.`, cwd at this repository's root.
- Every changed file byte-compiles.

No generated artifacts changed: this repo has no notebooks
(`config/build/visualise_notebooks.yaml` is `[]`), and the only generator
(`gallery/gallery_run.sh`) writes solely to `output/` and `scripts/*/images/`,
both gitignored with nothing tracked under either.

Issue: #561
Lands on PR #322 (draft), alongside the phase 3 commit 63c3323.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Jammy2211

Copy link
Copy Markdown
Collaborator Author

The flat fields= form folds into this PR

Commit 8c1a82a has been pushed onto feature/mass-field-workspace-sweep (fast-forward from 63c3323, 64 files, +110/-103). This PR now carries the workspace_test half of the flat-form sweep on top of the phase-3 collection migration, so the branch takes this repository from the galaxy-attached form straight to the flat form and never ships the intermediate collection form from here.

Companion PR: autolens_workspace #562 (draft) — same sweep, user-facing workspace. Issue: autolens_workspace#561.

What changed

Following PyAutoLens #744 (478213e78), the model's fields= slot accepts a bare al.MassField:

# before                                  # after
fields=af.Collection(field=field)         fields=field

Prior paths become fields.shear.gamma_1 instead of fields.field.shear.gamma_1.

  • This repo: an AST walk of 63c3323 found 66 model-side fields=af.Collection(field=...) sites across 64 files. 65 are migrated (63 in the bulk pass across 61 files, 2 in the delicate files multi_galaxy/composition_mge.py and multi_galaxy/model_fit.py); 1 is deliberately retained (below).
  • autolens_workspace (#562): 143 sites / 120 files, plus readers, prose and 356 regenerated notebooks.

What was deliberately NOT changed

  • Chained fields=<result>.{model,instance}.fields stages — 19 here (the four misc/database/scrape/*_jax SLaM scripts), 170 in autolens_workspace. Chaining is shape-transparent, so these work unchanged; the free-vs-fixed distinction rides on .model vs .instance.
  • Tracer-side al.Tracer(fields=[field]) list literals — 16 here, 112 in autolens_workspace. Tracers take a list, and it stays a list, mirroring the merged euclid pipeline (#90, 9cdee7b).
  • The 2 genuine multi-field tracer sites (all_fields).
  • The 13 intentionally field-less light-only stages — none gained a fields=.
  • misc/interop/coolest_round_trip.py — its .fields[0] reads are tracer-side list indexing.
  • misc/mass/galaxy_attached_legacy.py's second model (:189) keeps the fields=af.Collection(field=...) form on purpose. With every other model in both workspaces now flat, this is the workspaces' last regression witness that the library still accepts a collection in that slot. Only a docstring saying so was added.

A re-walk confirms the census is conserved at 103 fields= sites overall (66 → 65 flat + 1 retained, 19 attr-chain, 16 tracer list-literals, 2 all_fields), that no file mixes the two forms, and that fields.field no longer appears anywhere in the repo, docstrings included.

⚠️ IDENTIFIER MIGRATION NOTE — read before merging

The flat form intentionally changes prior paths (fields.field.shear.* → fields.shear.*) and therefore PyAutoFit result identifiers. Migrated scripts write to NEW output/ directories; existing result trees keyed on the old identifiers are ORPHANED and are NOT resumed. No identifier was aliased to an old value and no result directory was renamed to fake continuity.

Script OLD NEW
scripts/multi_galaxy/composition_mge.py (:263) b99831e66dd27eee314113e8e58235b6 29f82bd3de24984b94657c328b64c3be
scripts/misc/mass/galaxy_attached_legacy.py (:159) 35ebe9353118bcc0c7b2d577ce2639ee UNCHANGED — the backwards-compatibility regression, must not move

The composition_mge.py pin was recomputed, not adjusted: the baseline was reproduced first (old pin green pre-edit), then the new value read from the genuinely edited script's own assertion output. It was computed with the cwd at this repository's root, the same cwd every script here runs from, because an identifier is a function of the model and the active PyAutoFit configuration. The docstring at :237 now records that. composition_mge.py is on the smoke gate, so a wrong pin would go red immediately — it passes in 3.0 s.

That file took four coupled edits: the model-composition site, the path-grouping assertions at :159-:170 (the second path element under fields is now the mass profile shear, with no intervening collection key), the identifier pin, and the docstring/prose.

Evidence

  • Smoke: 32/32 PASS (979 s, profile_smoke.yaml), including composition_mge.py and both latent integration smokes. autolens_workspace: 38/38 scripts + 2/2 notebooks.
  • Equivalence 8/8: prior_count unchanged, model.identifier changed, prior-path sets equal after normalising fields.field. → fields.. (composition_mge: 17/17, symmetric difference empty. Repo-wide witness: 21 == 21.)
  • Log-likelihood bit-identical for the same instance under both forms — -37796.54513586828.
  • Chained stages, misc/database/scrape/slam_general_jax.py hand-run (these four are SLOW-skipped in no_run.yaml and have no automated coverage): all five stages carry a top-level fields under the flat form with the free/fixed split intact — source_lp[1] (seed), source_pix[1] and mass_total[1] (.model.fields) show fields MassField (N=2) with the seed's own prior ids 25/26 propagated; source_pix[2] and light[1] (.instance.fields) show fields MassField (N=0), gamma 0.0. slam_start_here.py hand-run PASS with all 5 stages carrying fields.shear.
  • Every changed file byte-compiles.
  • No generated artifacts changed: this repo has no notebooks (config/build/visualise_notebooks.yaml is []), and the only generator (gallery/gallery_run.sh) writes solely to output/ and scripts/*/images/, both gitignored with nothing tracked under either.

Local stack: PyAutoLens 478213e78, PyAutoGalaxy 90e757d3, PyAutoFit 7c0e79aab.

The subhalo sensitivity fix (in #562)

Both hasattr guards in base_model_narrow_priors_from() were rewritten. Under the flat form the OLD guard silently skipped a deliberate shear prior re-centring — base_model.fields.field raises AttributeError inside hasattr, which returns False: no error, no log, the re-centring never ran. Proven to fire now: gamma_1 UniformPrior(-0.3, 0.3) → UniformPrior(0.0734, 0.1734), gamma_2 UniformPrior(-0.3, 0.3) → UniformPrior(-0.1067, -0.0067). Those scripts remain parked in no_run.yaml and out of smoke_tests.txt.

SEPARATELY — a pre-existing bug fix, not part of the migration (in #562)

scripts/multi_dataset/features/one_by_one/modeling.py re-collected af.Collection(dataset_model=..., galaxies=result.instance.galaxies) and dropped fields entirely, so its "Second Dataset Offset" stage — documented as keeping all lens model parameters fixed — silently fitted with no external shear. The author's own commented-out block above it already uses the correct fields=result.instance.fields. One line added. Pre-existing on main, approved by the human.

This PR's red CI leg is NOT this sweep

Bisected to a JAX 0.11.2 × unfixed-PyAutoGalaxy regression: TypeError: functools.partial ... is not a valid JAX type, raised in the zero-contour solver via the effective_einstein_radius latent. The unmodified main version of the script fails identically in a CI-matching venv. It is fixed by PyAutoGalaxy 90e757d3, which merged roughly 10.5 hours after that CI run, so a CI re-run should clear it.

The flat form passes under JAX 0.11.2 on current mains: misc/latent/latent_integration_smoke_jax.py — the exact script whose leg was red — PASSES under the flat form (56.4 s), writing exactly one files/latent/latent_summary.json with all 8 latents finite and non-zero.

PyAutoHeart ack

Heart was RED at ship time. Verdict, verbatim:

RELEASE READINESS  ✗ RED  score 60
  ✗ release validation FAILED (stage integrate)
  ! manifest drift: remote-session blocks (generated) — 2 mismatch(es) vs PyAutoMind/repos.yaml

Both conditions are pre-existing and unrelated to these two workspace repos. The human was asked:

"Heart is RED (release validation FAILED, stage integrate) plus a YELLOW manifest drift — both pre-existing and unrelated to these two workspace repos. Do you authorize pushing the two commits and opening/updating the PRs despite the RED?"

and answered:

"Yes — push and open/update PRs"

That authorization covers push and PR-open/update only. Merge and release remain human-only and are NOT authorized by it. This PR stays draft with pending-release until PyAutoLens #744 (478213e78) is on PyPI.

Cross-references

Issue autolens_workspace#561 · phase-3 issue #559 · merged autolens_workspace#560 (c79c8d3) · PyAutoLens #743/#744 (478213e78) · euclid pipeline #90 (9cdee7b)

🤖 Generated with Claude Code

@Jammy2211
Jammy2211 marked this pull request as ready for review September 19, 2026 09:34
@Jammy2211
Jammy2211 merged commit 279a69d into main Sep 19, 2026
3 checks passed
@Jammy2211
Jammy2211 deleted the feature/mass-field-workspace-sweep branch September 19, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release PR queued for the next release build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: every external field → fields= (al.MassField), both workspaces

1 participant