Dropped webhooks leave QA and CR stamps off the board

[probablyian]

Some QA and CR comments never show up on the board. Pulldasher only hears about a new comment, push, or close when GitHub sends it a webhook, and some of those webhooks are not getting through. GitHub does not resend a failed delivery, and the only thing that re-reads a pull later is a restart. A restart only covers the repos in the config, so stamps in any other repo stay missing until someone clicks refresh on the card.

Reported on ifixit#64761: the QA comment came after the last push and never counted.

What the board missed on 2026-09-29

• ifixit#64895: the board stopped updating it at 10:28Z and still shows the old head as a draft. It missed a push at 12:18Z, ready for review, and a CR comment at 12:51Z.
• Pushes: #64893, #64521, #64894 and #64885 still show their old head after pushes between 11:01Z and 12:58Z. #64886, pushed three seconds after #64885, came through, so single events are being dropped.
• Repos missing from the config: 8 open pulls in fixbot, DocHarvestor, fixhub-app, ci-actions and expo have a QA comment after their last push that the board ignores.
• Why they stay missing: the startup refresh only walks config.repos (utils.js#L79, from app.js#L98). Other repos reach the board only through webhooks.

Why deliveries fail is still open. It needs the webhook's delivery log or the production container logs.

It isn't the QA pattern, the push check, or comment edits

• The production config uses the same regex as config.example.js#L5-L8, and it matches every missed comment when run in Node.
• Every missed stamp is newer than its pull's head commit, so the check that drops stamps older than the last push doesn't apply.
• The refresh on edit (githubHooks.js#L158) works: edited QA stamps on ifixit#64779, #64780 and #64782 through #64785 registered.

Ways to fix it

A re-read costs 8 to 15 REST calls per pull (#467). The board holds 151 open pulls from configured repos, 316 in all.

#	Approach	Fixes missed events	Missing repos	Delay	GitHub calls	New code
1	Refresh the card, or restart	When someone notices	Refresh only	Until noticed	1,200 to 2,300 per restart	None
2	Add every repo to config.repos	At restart	Yes	Next restart	2,500 to 4,700 per restart	Config only, a line per new repo
3	Startup refresh every hour	Yes	No	Up to 1 hour	1,200 to 2,300 an hour	A timer
4	Per-repo list of recently updated pulls every 10 min	Yes	No	Up to 10 min	156 an hour, plus re-reads	A timer and a list call per repo
5	Option 4 with conditional requests	Yes	No	Up to 10 min	Re-reads only	Option 4 plus an ETag store
6	One org-wide search every 10 min	Yes	Yes	Up to 10 min	6 search calls an hour, plus re-reads	A timer and one search call
7	Resend failed deliveries on a schedule	Only failed deliveries	Yes	Next run	A few per run	Script, workflow, org owner token
8	Fix why deliveries fail	That cause only	n/a	n/a	None	Unknown

I'd go with 6. It needs the least new code of 4 to 6, covers every repo, and search returns closed pulls, so it covers #501's case too. The open question is 5 or 6. Option 5 avoids search, but it only covers configured repos and has to store each list's ETag.

Notes on the options

• Re-reads: 111 org pulls changed in the 24 hours before this was written, so options 4 to 6 spend up to 900 to 1,700 calls a day on re-reads. Comment edits count as updates: fixhub-app#21, ci-actions#15 and expo#1332 each show an updated_at equal to their QA edit.
• Options 2 and 3: both make the startup refresh heavier or more frequent, and Reduce GitHub API consumption to avoid rate-limit outages #467 traced a drained hourly quota to that refresh. Option 3 also lists open pulls only (git-manager.js#L165-L171), so a pull whose close was dropped stays stuck, the case Fix pulls stuck open after a missed close webhook #501 fixes.
• Option 5: an unchanged list comes back 304, which doesn't count against the rate limit.
• Option 6: search has its own limit of 30 calls a minute. Its indexing delay isn't documented, so each search should overlap the previous one by a few minutes.
• Option 7: GitHub's redelivery script needs an org owner token. It misses events Pulldasher answered 200 to but failed to save, and only deliveries from the last 3 days can be resent.
• Option 8: deploys and restarts still drop events, so it needs one of 4 to 6 next to it.

QA comments the board ignores in repos missing from the config

All posted after the pull's last push and matching the QA pattern, as of 2026-09-29. Clicking refresh on each card should clear them.

Pull	QA comment
fixbot#3194	2026-08-17 20:13Z
fixbot#3215	2026-08-17 19:52Z
fixbot#3121	2026-09-01 18:38Z
DocHarvestor#236	2026-08-17 18:30Z
DocHarvestor#242	2026-08-28 03:52Z
fixhub-app#21	edited 2026-09-24 19:11Z
ci-actions#15	edited 2026-09-22 16:47Z
expo#1322	edited 2026-09-24 20:06Z

Found by comparing signoffs.qa.by from /api/v1/pulls with each pull's comments and head commit date from the GitHub API.

[probablyian]

@jarstelfox any input/preference on how to handle this?

[jarstelfox]

Nope!

[probablyian]

The dropped webhooks weren't random. All 71 stuck rows on the board come from three windows when webhooks weren't getting through. Two of those were outages of the server that runs Pulldasher.

Window (UTC)	Stuck rows	Cause
2026-08-17 18:31 to 20:13	3	Probably GitHub's 08-17 incident
2026-08-28 03:38 to 2026-09-02 21:52	55	The server was rebuilt and booted at 16:46 on 08-28. The 5 days after that are unexplained, and their logs have expired.
2026-09-29 10:50 to 14:55	13	The server lost its network

One case doesn't fit. The QA comment on iFixit/ifixit#64761 went up at 03:53 on 09-29, 9 minutes after its last push and 7 hours before the outage. The org webhook's delivery log keeps 3 days, so until about 10-02 an org owner can see what Pulldasher answered. @jarstelfox, could you check that delivery, and redeliver the failed ones from 10:50 to 14:55? That also clears the 13 rows from that window.

#503 already recovers from an outage Pulldasher stays running through, because a failed search keeps its window for the next sweep. db5e626 adds restarts: the first sweep after boot reaches back to the newest pull update in the DB, up to 24 hours.

How the windows were found

• Stuck rows: every open row from /api/v1/pulls that GitHub shows closed, or with a different head commit. A lost close is dated by its close time, a lost push by when CI started on the new head.
• 08-17: GitHub's incident ran 13:28 to 21:15 and listed Webhooks as degraded, but it called webhooks mitigated at 16:59, so this match isn't proven.
• 08-31 to 09-02: the same server answered 840 to 35,132 web requests an hour, so the tail of the second window wasn't a server outage.
• 09-29: monitoring lost every check on the server for 4 hours while six other servers kept reporting. It didn't reboot, and its CPU sat 99% idle through the gap. The newest update the board holds from before it is iFixit/ifixit#64895 at 10:28:56. The stuck rows' events run from 12:17 (a force push to iFixit/fixbot#3367) to 14:42 (iFixit/ops#1101 closed).
• iFixit/ifixit#64886: the issue says it came through 3 seconds after iFixit/ifixit#64885 was dropped. That's wrong: its 12:58 push was inside the outage, and the board picked it up later from a re-read.
• Ruled out: the GitHub IP allowlist in front of Pulldasher matches GitHub's current webhook ranges exactly.