DPD partial failures can be fatal

[cfzimmerman]

Summary

"fatal" feels like the wrong word because I associate it with panic. But basically the rng txn chaos test sometimes gets dpd into a state where a repeated port_settings_* operation will always fail due to soft state / SDE mismatches. This pointed me toward a broader pattern of interrupted writes causing unrecoverable states. The general solution is replacing cases of "success if the desired state transition occurred" with "success if the final state matches the request". The latter behaves batter among reconcilers and absorbs previously-interrupted writes.

Background

test_port_settings_txn_sweep fails about 1/10 runs locally. I believe I've also seen this test fail in CI. The failures are a class of bugs related to partial config creating an unexpected state for future config. Reproduced in the tests on this branch: accounts.google.com/gsi/transform

Here's an example of one such failure:

• port_settings_clear
• unplumb_link
  • uplink_clear
  • clear_mac_config
    • table op fails due to chaos
    • unplumb exits in error
• unplumb_link (tries again)
  • uplink_clear
    • table op fails because the uplink is already cleared

In this case, the uplink is already in the state we want, but the code errs because the expected state transition didn't occur. And now we'll get this failure perpetually.

The PR identifies and fixes some more cases. But the issue is that various create/delete actions expect state transitions, and reconciling through partial failures sometimes loses a state transition to some other error. So the state doesn't transition during the next reconciliation attempt, and that's interpreted as an error indefinitely.

port_settings_apply

Here's an example from the repro test failing indefinitely at port_settings_apply.

{"msg":"failed to add ipv4 address","v":0,"name":"dpd","level":50,"time":"2026-09-05T13:41:36.242997652Z","hostname":"helios","pid":22379,"error":"ASIC error: Exists","port":33,"addr":"211.33.123.210"}
{"msg":"failed to add ipv4 address","v":0,"name":"dpd","level":50,"time":"2026-09-05T13:41:36.747271693Z","hostname":"helios","pid":22379,"error":"ASIC error: Exists","port":33,"addr":"211.33.123.210"}
{"msg":"failed to add ipv4 address","v":0,"name":"dpd","level":50,"time":"2026-09-05T13:41:37.250982862Z","hostname":"helios","pid":22379,"error":"ASIC error: Exists","port":33,"addr":"211.33.123.210"}
cory@helios:~/dev/dendrite/dpd-client$ cat /tmp/settings_eventually_reconcile-dpd.stdout | rg "ASIC error: Exists" | wc -l
     124
cory@helios:~/dev/dendrite/dpd-client$

DPD reconciler

A partial write to the mac tables can cause the same. This is from another test.

{"msg":"Failed to program mac and port mapping: Switch(Exists)","v":0,"name":"dpd","level":50,"time":"2026-09-06T11:20:38.839686537Z","hostname":"helios","pid":4419,"link":"qsfp0/0","unit":"reconciler"}
{"msg":"failed to add entry to PortMacAddress: Switch(Exists)","v":0,"name":"dpd","level":20,"time":"2026-09-06T11:20:39.090990028Z","hostname":"helios","pid":4419}
{"msg":"set mac on 33 in table Ingress.mac_rewrite.mac_rewrite: a8:40:25:4a:1c:c2 failed: Switch(Exists)","v":0,"name":"dpd","level":50,"time":"2026-09-06T11:20:39.091017025Z","hostname":"helios","pid":4419}
{"msg":"Failed to program mac and port mapping: Switch(Exists)","v":0,"name":"dpd","level":50,"time":"2026-09-06T11:20:39.0910418Z","hostname":"helios","pid":4419,"link":"qsfp0/0","unit":"reconciler"}