[WIP] Language agnostic Kafka comms

[aliok]

Changes

🚧 Work in progress — not ready to merge. This lands the func-side deploy
integration for the language-agnostic Kafka runtime, but it depends on several
external pieces that are not yet in place. See Why this is WIP below.

• 🎁 Inject a language-agnostic Kafka runtime as a sidecar container when a
  function declares run.kafka. The function stays a plain CloudEvents-over-HTTP
  server; the sidecar consumes Kafka and delivers each record to the function over
  localhost (http://127.0.0.1:8080/), committing the offset on a 2xx response.
  This replaces the retired in-process path (FUNC_TRANSPORT=kafka on the function
  container).
• 🎁 Drop the Go-only gate in validateKafka: because the sidecar owns all Kafka
  handling, run.kafka now works for any runtime (it still requires
  invoke: cloudevent).
• 🧹 Carry the Kafka config (brokers/topic/group + TLS/SASL) on the sidecar
  container only; TLS certs and SASL credentials mount into the runtime, never into
  the function container.
• 🐛 Reject run.kafka on the knative (Serving) deployer. The sidecar is only
  injected by the k8s and keda deployers, and the in-process path is gone, so a Kafka
  function deployed on knative would come up healthy yet never receive a record. Fail
  fast with an error pointing at the keda deployer. (Injecting the sidecar into the
  Knative Service is left to a follow-up.)

Why this is WIP

• The runtime image lives in a personal repo. The sidecar is built from
  https://github.com/aliok/func-kafka-adapter, currently under a personal GitHub
  account rather than a Knative / knative-extensions org. Its permanent home is TBD.
• The image is not published. The deploy default points at
  ghcr.io/aliok/func-kafka-adapter:latest, which is not an official, versioned,
  published image, and there is no release/pinning story yet. It is overridable via
  FUNC_KAFKA_RUNTIME_IMAGE.
• The adapter is not a dependency of this repo. func and the runtime are coupled
  only by the CloudEvents-over-HTTP wire contract plus a set of KAFKA_* /
  FUNCTION_TARGET env vars — there is no Go module dependency — so nothing here pins
  or verifies the runtime version.
• The in-process path is not fully retired. func-go still ships the in-process
  kafka package and the CloudEvents scaffolding still imports it. That code must be
  removed (dropping Sarama from every Go function's dependency tree) before this is
  complete.
• Depends on an unreleased func-go fix. Reliable concurrent delivery needs the
  CloudEvents receiver-wedge fix (fix(cloudevents): stop receiver stalling under concurrent request cancel knative-extensions/func-go#190, upstream
  fix(http): stop receiver stalling under concurrent request cancellation cloudevents/sdk-go#1333), which is not yet merged/released.
• Only validated on some runtimes. End-to-end proven for Go and Python; other
  runtimes are expected to work over the same HTTP contract but are untested.
• Stacked on unmerged work. This branch is currently based on the KEDA
  consumer-lag scaling work (scale.keda), which is not yet in main; those commits
  appear in this diff and will drop out once that lands.
• Known gaps not yet designed. Poison-pill / DLQ handling (a permanently-failing
  record blocks its partition) and the local func run sub-process topology are on
  the roadmap but not built.

/kind enhancement

Relates to SRVOCF-976

Release Note

Docs

[knative-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aliok
Once this PR has been reviewed and has the lgtm label, please assign lkingland for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment