Skip to content

fix: allow all egress from RHDH pods in CI - #168

Merged
subhashkhileri merged 13 commits into
redhat-developer:mainfrom
rm3l:fix/allow-http-egress-for-keycloak
Sep 29, 2026
Merged

subhashkhileri merged 13 commits into
redhat-developer:mainfrom
rm3l:fix/allow-http-egress-for-keycloak

Conversation

@rm3l

@rm3l rm3l commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Summary

The RHDH chart 2.y ships default-deny NetworkPolicies that only allow egress on specific ports (443, 53/5353, 5432, 6379). E2E tests talk to services on many other ports — Keycloak HTTP routes, ArgoCD, external APIs — causing failures like RPError: outgoing request timed out when the chart's NPs block the traffic.

This PR allows all egress from RHDH pods in CI by applying a permissive NetworkPolicy after deployment. A TODO is kept to narrow this down to explicit ports once all required destinations are mapped.

Changes

  • Added NetworkingV1Api to KubernetesClientHelper with an applyNetworkPolicy method (same create-or-replace pattern as ConfigMap/Secret)
  • Applied a rhdh-allow-all-egress NetworkPolicy after RHDH deployment
  • Used the correct pod selector labels per deployment method (Helm: app.kubernetes.io/name + instance + component; Operator: rhdh.redhat.com/app)

Why allow-all instead of specific ports

Keycloak in CI uses a plain HTTP Route (port 80) accessed via the external Route URL, which resolves to a router/LB IP — not a pod IP. The chart's in-cluster port 80 NP uses namespaceSelector: {}, which only matches pod IPs in namespaces, so it doesn't cover Route-based access. Beyond Keycloak, other E2E services also use non-standard ports. Rather than chasing individual port gaps, we allow all egress in CI and plan to narrow it down later.

Test plan

  • Verify workspaces using Keycloak auth no longer fail with OIDC timeout
  • Verify the NetworkPolicy uses correct pod selector labels for the deployment method
  • Baseline with allow-all-egress confirmed ~40 fewer failures vs port-80-only

The RHDH chart 2.y ships default-deny NetworkPolicies that only allow
egress on ports 443, 53/5353, 5432, and 6379. Keycloak in CI uses a
plain HTTP route (port 80), causing OIDC discovery to time out.

Apply an external NetworkPolicy after deployment that allows TCP port 80
egress to in-cluster destinations (namespaceSelector: {}), so RHDH pods
can reach Keycloak through the OpenShift router.

Assisted-by: Claude
Assisted-by: Claude
@rm3l
rm3l requested a review from subhashkhileri September 24, 2026 09:47
Match the chart's selectorLabels (name + instance + component) for Helm
deployments and the operator's rhdh.redhat.com/app label for operator
deployments, instead of only matching app.kubernetes.io/component.

Assisted-by: Claude
Comment thread src/deployment/rhdh/deployment.ts Outdated
Keycloak is accessed via an OpenShift Route (plain HTTP), which
resolves to a router/LB IP. namespaceSelector only matches pod IPs,
so it would not cover Route-based traffic. It would only work if
RHDH accessed Keycloak via the internal Service DNS name.

Assisted-by: Claude
The chart's OOTB port 80 NP uses namespaceSelector (in-cluster only),
but Keycloak is accessed via the Route URL which resolves to a
router/LB IP. This NP is not temporary — it remains necessary.

Assisted-by: Claude
rm3l added a commit to rm3l/rhdh-plugin-export-overlays that referenced this pull request Sep 24, 2026
Point E2E_TEST_UTILS_GIT_REF to the branch that adds a NetworkPolicy
allowing port 80 egress for Keycloak HTTP Route access.

TODO: revert once redhat-developer/rhdh-e2e-test-utils#168 is merged
and published.

Assisted-by: Claude
Comment thread docs/changelog.md Outdated
@rm3l rm3l changed the title fix: allow in-cluster HTTP egress for Keycloak in CI [WIP] fix: allow in-cluster HTTP egress for Keycloak in CI Sep 24, 2026
The chart's default-deny NPs only allow egress on a few specific ports.
E2E tests need access to many other services (Keycloak HTTP routes,
ArgoCD, external APIs), so allow all egress in CI rather than chasing
individual port gaps.

Assisted-by: Claude
@rm3l rm3l changed the title [WIP] fix: allow in-cluster HTTP egress for Keycloak in CI fix: allow all egress from RHDH pods in CI Sep 24, 2026
@rm3l rm3l changed the title fix: allow all egress from RHDH pods in CI [WIP] fix: allow all egress from RHDH pods in CI Sep 24, 2026
@rm3l rm3l changed the title [WIP] fix: allow all egress from RHDH pods in CI fix: allow all egress from RHDH pods in CI Sep 29, 2026
@subhashkhileri
subhashkhileri merged commit 8afead3 into redhat-developer:main Sep 29, 2026
3 checks passed
@rm3l
rm3l deleted the fix/allow-http-egress-for-keycloak branch September 29, 2026 09:02
jrichter1 pushed a commit to redhat-developer/rhdh-plugin-export-overlays that referenced this pull request Sep 29, 2026
* feat: migrate e2e values files to new RHDH chart 2.y format

Update all workspace e2e test value files from the old bitnami-wrapper
chart structure (upstream.backstage.*, global.dynamic.*) to the new
standalone chart format with flattened root-level keys.

Key changes:
- global.dynamic → dynamicPlugins
- upstream.backstage.* → root-level (appConfig, extraEnv, etc.)
- global.lightspeed → intelligentAssistant
- Remove system volumes/mounts now auto-managed by the chart
- Remove auto-injected env vars (BACKEND_SECRET, POSTGRESQL_ADMIN_PASSWORD)
- extraEnvVarsSecrets → extraEnvFrom with secretRef
- Drop duplicated install-dynamic-plugins initContainers

Assisted-by: Claude

* chore: bump @red-hat-developer-hub/e2e-test-utils to 2.1.19

Major version bump required for the RHDH chart 2.y migration in
e2e-test-utils, which changed deployment logic and default values
structure.

Assisted-by: Claude

* chore: update yarn lockfiles for e2e-test-utils 2.1.19

Assisted-by: Claude

* test(backstage): add port-80 egress NP via extraDeploy for Keycloak HTTP

The RHDH chart 2.y ships default-deny NetworkPolicies that only allow
egress on ports 443, 53, 5432, and 6379. Keycloak in CI uses a plain
HTTP route (port 80), causing OIDC discovery to time out.

Add a NetworkPolicy via the chart's new `extraDeploy` field to allow
TCP port 80 egress from RHDH pods. This is a targeted test to verify
the fix; follow-up work will integrate this into e2e-test-utils so it
applies to all workspaces automatically.

Assisted-by: Claude

* Revert "test(backstage): add port-80 egress NP via extraDeploy for Keycloak HTTP"

This reverts commit 6ab3f8c.

* test: use e2e-test-utils branch with port 80 egress NP

Point E2E_TEST_UTILS_GIT_REF to the branch that adds a NetworkPolicy
allowing port 80 egress for Keycloak HTTP Route access.

TODO: revert once redhat-developer/rhdh-e2e-test-utils#168 is merged
and published.

Assisted-by: Claude

* fix: bump RHDH_VERSION to 2.1 across all workspaces

2.0 only has CI-only chart tags (2.0-91-CI), while 2.1 has proper
release tags (2.1, 2.1-92). Also fixes the intelligent-assistant
workspace which was still defaulting to 1.11 (chart 1.y), causing
chart 2.y value keys to be silently ignored.

Assisted-by: Claude

* Revert "fix: bump RHDH_VERSION to 2.1 across all workspaces"

This reverts commit 2a2ead9.

* fix(fullsend): align Yarn registry sandbox policy (#3932)

fix(fullsend): align yarn registry sandbox policy

* fix(fullsend): use env.runner for code target allowlist (#3935)

fix(fullsend): pass code target branches through env.runner

* fix(fullsend): auto-trigger code agent for ready-to-code issues (#3936)

* fix(fullsend): trigger coder on ready-to-code label

* fix(fullsend): scope ready-to-code dispatch to E2E issues

* Revert "Pin Helm chart version for root E2E runs (#3944)"

This reverts commit 4737d60.

* fix(e2e): pin RHDH_VERSION to 2.0-91-CI

Work around a pagination bug in the chart version lookup when using
just "2.0" — pin to the exact CI build tag instead.

Assisted-by: Claude

* fix(e2e): pin RHDH_VERSION to 2.1-99-CI

Assisted-by: Claude

* fix(e2e): pin RHDH_VERSION back to 2.0-91-CI

2.1-99-CI fails to deploy due to a wrong digest.

Assisted-by: Claude

* chore: bump e2e-test-utils to 2.1.20 and revert fork pin

e2e-test-utils PR #168 has been merged and published. Revert the
temporary E2E_TEST_UTILS_GIT_REF pin and bump all workspaces to
the published 2.1.20 release.

Assisted-by: Claude

---------

Co-authored-by: Subhash Khileri <skhileri@redhat.com>
jrichter1 pushed a commit to redhat-developer/rhdh-plugin-export-overlays that referenced this pull request Sep 29, 2026
…mat (#3984)

fix: migrate e2e values files to new RHDH chart 2.y format (#3907)

* feat: migrate e2e values files to new RHDH chart 2.y format

Update all workspace e2e test value files from the old bitnami-wrapper
chart structure (upstream.backstage.*, global.dynamic.*) to the new
standalone chart format with flattened root-level keys.

Key changes:
- global.dynamic → dynamicPlugins
- upstream.backstage.* → root-level (appConfig, extraEnv, etc.)
- global.lightspeed → intelligentAssistant
- Remove system volumes/mounts now auto-managed by the chart
- Remove auto-injected env vars (BACKEND_SECRET, POSTGRESQL_ADMIN_PASSWORD)
- extraEnvVarsSecrets → extraEnvFrom with secretRef
- Drop duplicated install-dynamic-plugins initContainers

Assisted-by: Claude

* chore: bump @red-hat-developer-hub/e2e-test-utils to 2.1.19

Major version bump required for the RHDH chart 2.y migration in
e2e-test-utils, which changed deployment logic and default values
structure.

Assisted-by: Claude

* chore: update yarn lockfiles for e2e-test-utils 2.1.19

Assisted-by: Claude

* test(backstage): add port-80 egress NP via extraDeploy for Keycloak HTTP

The RHDH chart 2.y ships default-deny NetworkPolicies that only allow
egress on ports 443, 53, 5432, and 6379. Keycloak in CI uses a plain
HTTP route (port 80), causing OIDC discovery to time out.

Add a NetworkPolicy via the chart's new `extraDeploy` field to allow
TCP port 80 egress from RHDH pods. This is a targeted test to verify
the fix; follow-up work will integrate this into e2e-test-utils so it
applies to all workspaces automatically.

Assisted-by: Claude

* Revert "test(backstage): add port-80 egress NP via extraDeploy for Keycloak HTTP"

This reverts commit 6ab3f8c.

* test: use e2e-test-utils branch with port 80 egress NP

Point E2E_TEST_UTILS_GIT_REF to the branch that adds a NetworkPolicy
allowing port 80 egress for Keycloak HTTP Route access.

TODO: revert once redhat-developer/rhdh-e2e-test-utils#168 is merged
and published.

Assisted-by: Claude

* fix: bump RHDH_VERSION to 2.1 across all workspaces

2.0 only has CI-only chart tags (2.0-91-CI), while 2.1 has proper
release tags (2.1, 2.1-92). Also fixes the intelligent-assistant
workspace which was still defaulting to 1.11 (chart 1.y), causing
chart 2.y value keys to be silently ignored.

Assisted-by: Claude

* Revert "fix: bump RHDH_VERSION to 2.1 across all workspaces"

This reverts commit 2a2ead9.

* fix(fullsend): align Yarn registry sandbox policy (#3932)

fix(fullsend): align yarn registry sandbox policy

* fix(fullsend): use env.runner for code target allowlist (#3935)

fix(fullsend): pass code target branches through env.runner

* fix(fullsend): auto-trigger code agent for ready-to-code issues (#3936)

* fix(fullsend): trigger coder on ready-to-code label

* fix(fullsend): scope ready-to-code dispatch to E2E issues

* Revert "Pin Helm chart version for root E2E runs (#3944)"

This reverts commit 4737d60.

* fix(e2e): pin RHDH_VERSION to 2.0-91-CI

Work around a pagination bug in the chart version lookup when using
just "2.0" — pin to the exact CI build tag instead.

Assisted-by: Claude

* fix(e2e): pin RHDH_VERSION to 2.1-99-CI

Assisted-by: Claude

* fix(e2e): pin RHDH_VERSION back to 2.0-91-CI

2.1-99-CI fails to deploy due to a wrong digest.

Assisted-by: Claude

* chore: bump e2e-test-utils to 2.1.20 and revert fork pin

e2e-test-utils PR #168 has been merged and published. Revert the
temporary E2E_TEST_UTILS_GIT_REF pin and bump all workspaces to
the published 2.1.20 release.

Assisted-by: Claude

---------

Co-authored-by: Subhash Khileri <skhileri@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants