Cherry pick AddFiles: Add dry_run (#40167)

[claudevdm]

• AddFiles: dry_run reports what the schema pre-pass would do without committing or registering

SchemaEvolutionConfig.setDryRun(true) (provider key dry_run) turns AddFiles into a report-only transform: the read side runs as usual (footers, distinct schemas), then DryRunReport emits one Row per distinct schema plus one summary row on a new output, dry_run_report.

Why: before enabling evolution on a large import, a user wants to know what the options would do to the table and which files would be refused, without touching anything. Running the real transform under FAIL_PIPELINE answers only "would it fail", and only for the first failure.

The verdicts come from CommitSchemaUnion.plan, the one step that decides everything a commit does before writing anything. A Plan says which distinct file schemas are merged (SchemaToMerge), which are refused and why (IncompatibleSchema), and the schema the table ends with. It is an EvolutionPlan against an existing table (base schema snapshot, name-mapping repair) or a CreationPlan when the table is missing (partition spec and sort order resolved against the union, or the problem that blocks creation). commitOnce dispatches to evolve or create, which fail or warn under the handling mode and then write; the dry run turns the same plan into rows, so a check added to the plan reaches both and the report cannot drift from the commit. A schema that is fine against the table but conflicts with another schema of the input is therefore reported with the blame a real run assigns.

Real-run changes that come with planning first: partition or sort fields that do not fit the union fail with a message naming them, before any catalog write and under either handling (the per-file fallback creation throws the same error, so it could never be routed); problems are reported before the transaction is opened; every transaction is checked against the one base snapshot; planning retries like committing; a window without schemas plans nothing. Settings (config, the handling resolved for the mode, NewTableSettings) replaces the three loose arguments both DoFns carried.

Report rows (REPORT_SCHEMA), told apart by row_type:
row_type schema | create | unreadable | unchecked | summary
schema_key short murmur3 key of the schema JSON on schema rows
schema the canonical schema JSON; on the create row, the
union the table would be created with
num_files files the row covers
changes ARRAY: the SchemaDelta descriptions on schema
rows; "create <optional|required> " per
column on the create row (pins shown required); the
totals line and table-level changes on the summary
allowed whether a real run would accept it
reason why not, else ""; the consequence on the summary
would_create_table false whenever a real run would not create the
table, including when it would fail first

Provider: the dry_run_report output only exists when dry_run is set, so existing YAML pipelines that enumerate outputs are unaffected.

• improve retry

• comments

• add untracked file

• move tests

• track file

• comments

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[claudevdm]

R: @Amar3tto

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers