One ShapePipe workflow for data and image sims

[cailmdaley]

The image-sims ShapePipe run is a separate orchestrator (example/cfis_image_sims plus the *_canfar_v2.0.bash dispatcher, driven from sp_validation's image_sims.smk), and the config cleanup that came with the data workflow (#847, #852) has broken it: its default.* symlinks dangle and it selects mask configs that no longer exist. Meanwhile the Snakemake workflow (#852) does everything the sims need — chunked ngmix, node-local vignette staging, completeness checks, resume, the run report — and the sims get none of it.

Target: sims are a branch of the one workflow, not a sibling.

• rules/image_sims.smk in the main tree. Two config keys select the branch: kind: data|sims (where the pixels come from and how they're named) and psf_model: psfex|mccd|fake. They stay separate so sims can also run the real PSF chain once the star-injected grids exist.
• The sims delta is small: a few _sims ini variants under workflow/config/cfis/, one new rule (tile_fake_psf), and tile_uncompress/exp_psf skipped for sims. Everything else is the data DAG unchanged.
• Each shear branch (1z2z, 1p2z, …) is one campaign: five sp runs over the same sky with different input roots. sp_validation's sims workflow drives ShapePipe — its m-bias DAG calls sp run per branch, then reads only the campaign products (Persist PSF products and add the two campaign-level merges (exp_persist, star_cat_merge, final_cat_merge) #879's final_cat_<campaign>.hdf5 plus a run manifest), never ShapePipe's run tree.
• The workflow runs on both nibi and candide.
• Gone: example/cfis_image_sims, the bash dispatcher (after a deprecation release — they're packaged entry points), and sp_validation's im_init/im_pipeline/im_merge with their ShapePipe-internal config keys.

Steps — each becomes a sub-issue when we're ready to pick it up:

1. sp run --config-file and a candide profile (shapepipe). No dependencies; this is the two-machine work.
2. Persist PSF products and add the two campaign-level merges (exp_persist, star_cat_merge, final_cat_merge) #879 (in progress). The sims branch stacks on its final_cat_merge.
3. The sims branch (shapepipe): kind/psf_model keys, _sims ini variants, tile_fake_psf, the two elided rules. Acceptance: one tile of grid_3 through sp run, compared against the existing catalogue.
4. The boundary (sp_validation): run manifest, im_shapepipe calling sp run per branch, cat_config reading the manifest; im_init/im_pipeline/im_merge and their keys removed. The comprehensive catalogue comes from the v2 hdf5 readers (Read ShapePipe v2 campaign products: hdf5 catalogues, MASK_n* cut, patches retired sp_validation#343) rather than extract_info.py (Config cleanup: one source of truth + fiducial-plus-overrides philosophy sp_validation#294).
5. Deprecate example/cfis_image_sims and the bash dispatcher (shapepipe), one release later removal.

Supersedes the pipeline half of CosmoStat/sp_validation#284 / #285 / #287; related: #870, #885.

Rule table (data vs sims)

Rule	data	sims	Parameterisation
tile_get_images	✓	✓	config_tile_Git_{kind}.ini (INPUT_PATH, INPUT_FILE_PATTERN, INPUT_NUMBERING \d{3}\.\d{3} vs \d{3}-\d{3}, INPUT_FILE_EXT, OUTPUT_FILE_PATTERN). Plus a rule-side change: get_images_runner matches tile_numbers.txt, which unit_pre writes in dot format; sims need the dashed id, so unit_pre branches on kind (or exports a third var). Not ini-only.
tile_uncompress	✓	—	Elided for sims (weights ship uncompressed); sims' Sx reads the weight from Git directly. Works because A's inis use fixed absolute INPUT_DIRs. Kills bit 2.
tile_find_exposures	✓	✓	config_tile_Fe_{kind}.ini — not one ini. Verified diff: FILE_PATTERN CFIS_image→CFIS_simu_image, COLNUM 3→2, EXP_PREFIX p→simu_image-. Those two are FITS-HISTORY parsing facts, they define the tile↔exposure edge build_index.py consumes, and getting them wrong yields a wrong or empty exposure list silently.
exp_get_images	✓	✓	config_exp_Gie_{kind}.ini (patterns + roots).
exp_split	✓	✓	One ini; FILE_EXT → $SP_IMG_EXT. N_HDU = 40 identical.
exp_psf	✓	only if psf_model != fake	Unchanged for data (post-#888). A sims × psfex variant is a fourth ini — see §2.1.
tile_fake_psf (new)	—	only if psf_model == fake	Tile-level, between tile_detect and tile_vignets. Must join the tile_shape group — fake_psf_runner writes galaxy_psf*.sqlite, the store ngmix reads at random per object per epoch, which is exactly what node-local NVMe staging exists for; outside the group it lands on NFS. Joining means inheriting the group's resource contract, and NGMIX_VIGNET_DIR hardcodes run_sp_tile_PiViVi, so any sims PiViVi variant keeps that RUN_NAME.
tile_exp_forest, tile_merge_headers	✓	✓	Unchanged (config_tile_Mh_exp_{kind}.ini for WARN_MISSING_EXP/INPUT_DIR).
tile_detect	✓	✓	config_tile_Sx_{kind}.ini (weight source, image pattern).
tile_vignets	✓	✓	config_tile_PiViVi_{psf_model}.ini gains a fake variant (vignetmakers only).
tile_ngmix	✓	✓	Both keys: PSF-source dir on psf_model, background inputs + BKG_SUB on kind. Chunking, ngmix_ranges.json, the tile_shape group apply unchanged — sims gain parallel ngmix, their compute bottleneck, for free.
tile_merge_cats, tile_make_cat	✓	✓	config_tile_Mc.ini carries no psf suffix on the data side; sims' config_tile_Mc_psfex.ini folds into it.
clean_exposure, clean_tile	✓	✓	Unchanged.
final_cat_merge (#879)	✓	✓	Already the campaign-merge rule; sims inherit it.

— Fable on behalf of Cail

[cailmdaley]

@martinkilbinger I tried to take stock of where we are in terms of getting the image sims snakemake workflow integrated with the nibi work, please take a look at this before starting on image sims!