feat: changelog-check-run-annotations - GitHub Check Run support for validation by ashleyshaw · Pull Request #3376 · lightspeedwp/.github

ashleyshaw · 2026-09-18T13:41:03Z

Feature Pull Request

This repository enforces changelog, release, and label automation for all PRs and issues.
See the organisation-wide Automation Governance & Release Strategy for contributor rules.

Linked issues

Closes #NONE (new convergence work on MVP tasks)

Relates to: Convergence analysis for Phase 10 (MVP Validation & Gap Closure)

Summary

Implements GitHub Check Run API integration to close the critical FR-2 gap identified during MVP convergence. The changelog validation workflow now creates proper GitHub Check Runs with detailed violation annotations visible in the PR Checks tab, replacing the comment-only approach.

Changes Made

T084: CheckRunReporter Class

Created .github/validation/changelog/lib/check-run-reporter.js (89 lines)
Provides reportCheckRun() method for GitHub Check Run API integration
Maps validation violations to GitHub annotation objects
Handles 50 annotation limit per GitHub API spec
Supports conclusion determination (success/failure/neutral)
Methods:
- reportCheckRun(context, validationResult): Creates/updates check run
- determineConclusion(validationResult): Maps results to conclusions
- buildCheckOutput(validationResult): Formats title, summary, annotations
- buildAnnotations(violations): Converts violations to GitHub annotations
- severityToAnnotationLevel(severity): Maps severity to annotation level

T085: Workflow Integration

Created .github/validation/changelog/bin/report-check-run.js (104 lines)
Standalone script for invoking CheckRunReporter from workflow
Accepts validation report path, token, owner, repo, SHA, PR number
Returns check run data as JSON
Integrated into .github/workflows/changelog-validation.yml
Added "Report validation via GitHub Check Run" step after validation
Continues on error to preserve downstream steps

T086: Integration Tests

Created .github/validation/changelog/test/integration/test-check-run-annotations.js (285 lines)
Comprehensive test suite using Vitest
Tests check run creation with proper conclusions
Verifies annotation structure and GitHub API compliance
Tests edge cases:
- 50 annotation limit enforcement
- API error handling
- PR context availability (head SHA vs fallback)
- Severity-to-level mapping
- Success/failure/warning conclusion determination
Uses mocked GitHub API client for isolation

T087: Workflow Status Reporting

Updated .github/workflows/changelog-validation.yml
Check runs now serve as primary validation artifact
Replaces comment-only approach with proper GitHub API integration
Tasks marked complete in .github/specs/003-changelog-quality-audit/tasks.md

Technical Details

GitHub Check Run Conclusions

success: All entries pass validation (pass rate = 100%)
failure: 1+ entries fail validation (pass rate < 100%)
neutral: Warning-level issues detected (pass rate ≥ 90% but < 100%)

Annotation Severity Mapping

critical/high → failure (red badge)
medium → warning (yellow badge)
low/unknown → notice (grey badge)

GitHub API Compliance

Respects 50 annotation limit per check run
Uses correct GitHub Actions context (head SHA for PRs, fallback SHA for commits)
Includes violation details in annotation messages
Handles transient API errors gracefully

Testing

Run integration tests:

cd .github/validation/changelog
npm test test/integration/test-check-run-annotations.js

Manual testing in GitHub UI:

Create PR with changelog changes
Review "Changelog Validation" check in Checks tab
Verify annotations appear with correct severity levels
Confirm entry-specific violation details are visible

Changelog

Added

GitHub Check Run API integration for changelog validation (closes FR-2 gap)
CheckRunReporter class with full GitHub Checks API support
Annotation-style output with per-entry violation details
Integration test suite for check run creation and annotation validation
Support for 50-annotation limit compliance

Changed

Changelog validation workflow now creates GitHub Check Runs
Validation results now provide actionable check badges in PR

Checklist (Global DoD / PR)

🤖 Generated with Claude Code

https://claude.ai/code/session_01CbYXeahLUGjYcfFaKwodc6

Generated by Claude Code

coderabbitai · 2026-09-18T13:41:10Z

Warning

Review limit reached

Next included review available in 56 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: lightspeedwp/.github/.coderabbit.yml

Review profile: CHILL

Plan: Advanced

Run ID: c00b8899-0995-49c8-9c36-2bc904019f52

📥 Commits

Reviewing files that changed from the base of the PR and between d1424a4 and b22cc01.

📒 Files selected for processing (6)

.github/specs/003-changelog-quality-audit/tasks.md
.github/validation/changelog/bin/report-check-run.js
.github/validation/changelog/lib/check-run-reporter.js
.github/validation/changelog/test/integration/test-check-run-annotations.js
.github/workflows/changelog-validation.yml
CHANGELOG.md

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-18T13:41:56Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-18T13:53:39Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-18T13:57:14Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-18T14:01:49Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-18T14:03:06Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-18T14:03:53Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-18T14:05:17Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-18T14:06:09Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

mergify · 2026-09-18T14:12:05Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

github-actions · 2026-09-18T14:12:15Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-18T14:13:10Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-18T14:14:03Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-18T14:14:55Z

PR Template Routing

Branch Type: feat
Scope: changelog-check-run-annotations
Template: pr_feature.md
Labels Applied: type:feature

This PR was automatically routed based on the branch naming strategy.

github-actions · 2026-09-18T14:16:32Z

📋 Changelog Quality Validation

Summary

Metric	Count
✅ Passing	73
⚠️ Warnings	0
❌ Failing	1
🆕 New failures in this PR	1
📦 Pre-existing failures	0

Status

❌ Validation FAILED - This PR introduces 1 new changelog failure(s). Please fix them before merging.
Pre-existing failures (0) are reported for visibility and do not block this PR.

How to Fix

Reproduce the exact PR check locally (from the repository root):

cd .github/validation/changelog
node bin/validate.js --changelog-path ../../../CHANGELOG.md --trigger pr_submission --pr-number 3376 --branch feat/changelog-check-run-annotations --output text

Only new failures introduced by this PR block merging; pre-existing Unreleased failures do not block. See Changelog Quality Audit for the rule reference.

Implement GitHub Check Run API integration to close FR-2 gap: - T084: Create CheckRunReporter class for GitHub Check Run API integration - Provides reportCheckRun() method to create/update checks with annotations - Maps validation violations to GitHub annotation objects - Handles 50 annotation limit per GitHub API spec - Supports conclusion determination (success/failure/neutral) - T085: Integrate CheckRunReporter into changelog-validation.yml workflow - Add "Report validation via GitHub Check Run" step after validation - Calls new report-check-run.js bin script with validation results - Continues on error to preserve downstream steps - T086: Create integration test suite for check run annotations - Test check run creation with proper conclusions - Verify annotation structure and GitHub API compliance - Test edge cases (50 annotation limit, API errors, missing PR context) - Validate severity-to-level mapping - T087: Ensure workflow reports validation status via check run - Check runs now serve as primary validation artifact - Replaces comment-only approach with proper GitHub API integration - Updates task completion status in tasks.md This implementation addresses the critical FR-2 gap identified during MVP convergence: changelog validation now creates proper GitHub Check Runs with detailed violation annotations visible in the PR Checks tab. Files created: - .github/validation/changelog/lib/check-run-reporter.js (89 lines) - .github/validation/changelog/bin/report-check-run.js (104 lines) - .github/validation/changelog/test/integration/test-check-run-annotations.js (285 lines) Files modified: - .github/workflows/changelog-validation.yml (added check run reporting step) - .github/specs/003-changelog-quality-audit/tasks.md (marked T084-T087 complete)

…GELOG.md not changed - Add conditional check for validation-report.json existence - Prevents check run reporter from failing when workflow triggers on validation script changes only - Ensures check run is only created when changelog validation actually ran - Makes executable bit on report-check-run.js explicit

The "Report validation via GitHub Check Run" step was failing when the workflow triggered on .github/validation/changelog/** path changes without CHANGELOG.md changes, because validation-report.json was not created. Replace the ineffective hashFiles() condition with an explicit output variable from the validate step that tracks whether validation-report.json was successfully created. This allows the check-run reporter to only run when the report file exists. - Add report_created=true/false output to validate step - Update check-run step condition to use report_created output - Resolves workflow failure in CI checks Relates to T084-T087: GitHub Check Run support integration Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbYXeahLUGjYcfFaKwodc6

…anged When CHANGELOG.md has not changed, the validate step exits early without setting the report_created output variable. This causes the check-run step's condition to be undefined, leading to unclear workflow status. Ensure report_created=false is always set so downstream steps can properly determine whether to create a check run or skip. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbYXeahLUGjYcfFaKwodc6

When CHANGELOG.md has not changed, validation is skipped but GitHub was not receiving an explicit check run status, leaving it undefined. Add a new step that creates a "neutral" check run with an appropriate message when validation is skipped due to no changelog changes. This ensures GitHub's checks tab always shows the validation status. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbYXeahLUGjYcfFaKwodc6

The previous implementation incorrectly tried to manually instantiate Octokit. The actions/github-script@v7 action provides pre-configured github and context objects that should be used directly. Update the skip-check step to use the built-in github and context objects for creating the neutral check run. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbYXeahLUGjYcfFaKwodc6

Replace context object access with direct GitHub Actions variable expansion for better reliability. Add try-catch error handling and logging to help diagnose check run creation failures. This should help identify any permission or API issues when creating the neutral check run for skipped validations. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbYXeahLUGjYcfFaKwodc6

When CHANGELOG.md has not changed, the workflow needs to explicitly succeed rather than leaving status undefined. Add a step that runs when validation is skipped to ensure the workflow completes with success status. This ensures GitHub receives a clear success status for the workflow when no changelog changes require validation. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbYXeahLUGjYcfFaKwodc6

- Remove redundant `any_changed` condition from skip-check steps - Ensures check run is created even when workflow triggers on .github/validation/changelog/** changes without CHANGELOG.md changes - Fixes case where neither check-run nor skip-check steps execute, leaving GitHub status undefined Fixes the persistent 'Validate changelog on PR' check failure by ensuring ALWAYS explicit check run status, never conditional skipping. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbYXeahLUGjYcfFaKwodc6

- GitHub Checks API integration for changelog validation workflow - CheckRunReporter class with full check run creation and annotation support - CLI tool for invoking check run reporter from workflow - Integration test suite with comprehensive coverage - Workflow integration with proper status reporting for all validation states - Fixes persistent check run status issue by ensuring ALWAYS explicit check creation Closes FR-2 gap from Changelog Quality Audit specification (Spec 003). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CbYXeahLUGjYcfFaKwodc6

github-actions Bot added the type:feature label Sep 18, 2026

ashleyshaw self-assigned this Sep 18, 2026

ashleyshaw added this to the v1.1 milestone Sep 18, 2026

ashleyshaw marked this pull request as ready for review September 18, 2026 14:11

ashleyshaw added the area:changelog label Sep 18, 2026

ashleyshaw added status:needs-review meta:needs-changelog labels Sep 18, 2026

ashleyshaw added the priority:normal label Sep 18, 2026

ashleyshaw requested a review from eleshar September 18, 2026 14:25

claude added 6 commits September 21, 2026 19:29

claude added 4 commits September 21, 2026 19:29

eleshar force-pushed the feat/changelog-check-run-annotations branch from 71ba37a to b22cc01 Compare September 21, 2026 17:30

github-actions Bot removed the area:changelog label Sep 21, 2026

ashleyshaw modified the milestones: v1.1, v1.6 Sep 22, 2026

Conversation

ashleyshaw commented Sep 18, 2026

Feature Pull Request

Linked issues

Summary

Changes Made

T084: CheckRunReporter Class

T085: Workflow Integration

T086: Integration Tests

T087: Workflow Status Reporting

Technical Details

GitHub Check Run Conclusions

Annotation Severity Mapping

GitHub API Compliance

Testing

Changelog

Added

Changed

Checklist (Global DoD / PR)

Uh oh!

coderabbitai Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

mergify Bot commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 18, 2026

PR Template Routing

Uh oh!

github-actions Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📋 Changelog Quality Validation

Summary

Status

How to Fix

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Sep 18, 2026 •

edited

Loading

github-actions Bot commented Sep 18, 2026 •

edited

Loading