Skip to content

Unbreak the build and image, fix the restart loop, harden providers (1.5.0) - #6

Open
rockfordlhotka wants to merge 3 commits into
mainfrom
fix/build-security-and-provider-hardening
Open

rockfordlhotka wants to merge 3 commits into
mainfrom
fix/build-security-and-provider-hardening

Conversation

@rockfordlhotka

Copy link
Copy Markdown
Member

main did not build, the container image could not be produced, and the deployed pod was
restarting on transient network errors. This fixes those, then works through the review findings
behind them.

Verified against a real PostgreSQL 17 server and a built image, not just unit tests — that testing
caught three defects that unit tests alone had passed.

The blockers

The build failed. Microsoft.EntityFrameworkCore.Sqlite 10.0.5 pulls SQLitePCLRaw.lib.e_sqlite3
2.1.11, which carries a high-severity advisory (GHSA-2m69-gcr7-jv3q).
With TreatWarningsAsErrors, NU1903 broke dotnet build for four projects. EF Core is now 10.0.12,
which brings SQLitePCLRaw 2.1.12.

The image could not be built. The Dockerfile never listed SocialAgent.Providers.Threads.csproj
in its restore layer, added in 1.4.0, so restore skipped it and dotnet publish --no-restore failed
with NETSDK1004. This is why the cluster still runs socialagent:1.3.4 while the manifest claimed
1.4.0 — the Threads release was never actually deployable.

Nothing verified this repository. .github/workflows/ held only the squad-* automation, which
is how both of the above reached main. CI now builds, tests, and builds the container image on
every push and pull request.

The restart loop

The deployed pod had 60 restarts, last exiting Completed. Its log:

at SocialAgent.Providers.Mastodon.MastodonProvider.GetProfileAsync
at SocialAgent.Host.Services.SocialMediaPollingService.ExecuteAsync
at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync
[INF] Application is shutting down...

An HttpClient timeout surfaces as TaskCanceledException, which is an OperationCanceledException.
Every background service filtered its handler with when (ex is not OperationCanceledException)
meant to avoid swallowing shutdown, but it means a transient blip talking to Mastodon was
deliberately not caught, escaped ExecuteAsync, and the host's default StopHost behaviour killed
the process. The handlers now key off the stopping token, which actually distinguishes shutdown from
an inner operation timing out. The same filter is corrected in the providers, where it defeated the
graceful-degradation paths it was written for.

Covered by regression tests; the timeout test was confirmed to fail against the old filter.

Other runtime fixes

  • Bluesky died about two hours after every pod start. The provider cached its access JWT forever
    and never used the refresh JWT it had already parsed, so every call returned 401 until a restart.
    Sessions now refresh via com.atproto.server.refreshSession on a 401, fall back to a full login,
    and retry the failed request.
  • Providers pinned one HttpClient for the process lifetime. AddHttpClient<T> registers T as
    transient, so resolving it from a singleton factory captured one instance and its message handler
    permanently — defeating handler rotation and sharing a mutable DefaultRequestHeaders between the
    polling loop and A2A request threads, which HttpHeaders does not support. Providers now take
    IHttpClientFactory and set Authorization per request.
  • /health/ready reported healthy with the database downAddHealthChecks() had no checks
    registered. Readiness now runs AddDbContextCheck; liveness stays a process-only check so a
    database blip does not restart the pod.
  • Polling silently dropped data. Each provider fetched one fixed page and filtered since
    client-side. All three now page to the cutoff (Mastodon max_id, Bluesky cursor, Threads after).
  • The agent card reported 1.4.0.0; it now reports the three-part informational version.
  • Bluesky recorded the relay's indexedAt as CreatedAt instead of the record's authoring timestamp.

Security

  • API keys compare with CryptographicOperations.FixedTimeEquals rather than string.Equals, so
    response latency no longer leaks a prefix of the key. Repeated X-Api-Key headers are rejected.
  • The Threads token moves from the query string to an Authorization header, keeping it out of
    url.full on OpenTelemetry spans and out of exception messages. The documented query-parameter
    form remains as a fallback if Meta rejects the header.
  • A missing Authentication:ApiKey outside Development fails at startup instead of leaving the agent
    running and rejecting every request.
  • The container runs non-root with a read-only root filesystem, dropped capabilities and
    RuntimeDefault seccomp.

EF Core migrations

Migrations replace EnsureCreatedAsync plus hand-written CREATE TABLE IF NOT EXISTS patching, in
SocialAgent.Data.Migrations.Sqlite and SocialAgent.Data.Migrations.Npgsql — EF cannot resolve two
providers' migrations from one assembly. DatabaseMigrationService adopts a pre-1.5.0 database
(schema present, no __EFMigrationsHistory) by recording the baseline as already applied.

One trap worth knowing: on Npgsql, IHistoryRepository.ExistsAsync() returns true even when no
history table exists
. Gating adoption on it meant adoption never ran on PostgreSQL and MigrateAsync
hit 42P07: relation "Notifications" already exists — a 1.4.0 database would have crash-looped on
upgrade. SQLite does not exhibit this, so SQLite-only tests passed. Detection now queries the
catalogue directly.

Quality

Resilience handlers and 30s timeouts on provider HTTP clients; ValidateOnStart on provider options;
batched repository upserts instead of a SELECT per row; analytics aggregated in SQL rather than
materialising the retention window; a bounded SkillRouter timeout; HTML stripped from Mastodon
content; a cached Mastodon account id.

Testing

78 passing, 0 failing (was: build broken, and 12 tests failing without credentials — integration tests
now self-skip). New coverage for the host, which had none, plus the provider HTTP behaviour.

Verified end to end against PostgreSQL 17 and a built image:

  • fresh database provisions from migrations; a genuine 1.4.0-shaped database with seeded rows is
    adopted with posts and provider tokens intact; both idempotent across restarts
  • container runs as uid 1654 with read-only rootfs and all capabilities dropped
  • agent card reports 1.5.0; /a2a returns 401 without a key and dispatches skills with one,
    reading real aggregates out of the adopted database
  • /health/ready returns 503 while PostgreSQL is stopped and recovers to 200, while /health/live
    stays 200 throughout

Deployment notes

  • Back up the database before the first rollout. Baseline adoption is tested against SQLite and a
    real PostgreSQL server, but never against your live data.
  • The Threads ConfigMap and Secret references are now optional: true. Threads is disabled and those
    keys are absent from the deployed ConfigMap and Secret, so the manifest as it stood would have left
    the pod in CreateContainerConfigError.
  • The manifest moves to 1.5.0; the cluster currently runs 1.3.4.

Not verified

Whether Meta accepts Authorization: Bearer on the Threads refresh endpoint — that needs a live
token, and Threads is not in use. The query-parameter fallback covers a rejection.

🤖 Generated with Claude Code

rockfordlhotka and others added 3 commits September 18, 2026 12:04
…1.5.0)

`main` did not build and the container image could not be produced:

- EF Core 10.0.5 pulled SQLitePCLRaw 2.1.11, which carries a high-severity
  advisory (GHSA-2m69-gcr7-jv3q). With TreatWarningsAsErrors, NU1903 failed
  `dotnet build` for four projects. EF Core is now 10.0.12 (SQLitePCLRaw 2.1.12).
- The Dockerfile never listed SocialAgent.Providers.Threads.csproj in its
  restore layer, added in 1.4.0, so `publish --no-restore` failed NETSDK1004.
- Nothing in CI built or tested this repo, which is why both reached main.
  Adds .github/workflows/ci.yml covering build, test and image build.

Runtime fixes:

- Bluesky cached its access JWT forever and never used the refresh JWT it
  already parsed, so every call 401'd about two hours after pod start until a
  restart. Sessions now refresh on 401 and fall back to a full login.
- Providers captured a transient typed HttpClient in a singleton, pinning the
  message handler for the process lifetime and sharing mutable
  DefaultRequestHeaders across the polling loop and A2A request threads. They
  now take IHttpClientFactory and set Authorization per request.
- /health/ready had no checks registered and reported healthy with the database
  down. It now runs AddDbContextCheck; liveness stays a process-only check.
- Providers fetched one fixed page and filtered `since` client-side, dropping
  anything beyond it. All three now page to the cutoff.

Security:

- API keys compare with CryptographicOperations.FixedTimeEquals.
- The Threads token moves from the query string to an Authorization header, so
  it no longer lands in OpenTelemetry spans; the documented query form remains
  as a fallback.
- A missing Authentication:ApiKey outside Development fails at startup.
- The pod runs non-root with a read-only root filesystem.

EF Core migrations replace EnsureCreated plus hand-written CREATE TABLE
patching, in per-dialect assemblies. DatabaseMigrationService adopts a
pre-1.5.0 database by recording the baseline as applied; covered by tests on
SQLite and opt-in PostgreSQL tests. Not dry-run against live Postgres — back up
before the first production rollout.

Also batches repository upserts, aggregates analytics in SQL, bounds
SkillRouter with a timeout, strips HTML from Mastodon content, and adds host
and provider test coverage where there was none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…file

Real Docker and PostgreSQL testing surfaced two defects in the previous commit
that SQLite-only tests could not see.

Migration adoption never ran on PostgreSQL. The legacy-database check gated on
IHistoryRepository.ExistsAsync(), which on Npgsql returns true even when no
__EFMigrationsHistory table exists. Adoption was therefore skipped and
MigrateAsync ran the baseline against an existing schema, failing with
42P07 "relation Notifications already exists" — i.e. a 1.4.0 database would
have crash-looped on upgrade. Detection now queries the database catalogue
directly for the history and Posts tables, which reads the same on both
providers and is also quiet: EF logs its own probe of a missing history table
at Error level, which is alarming on a first run.

The Dockerfile restore layer did not list the two new migration projects, so
`dotnet publish --no-restore` failed with NETSDK1004. The earlier verification
simulated the intended project list rather than reading the actual Dockerfile,
so it passed while the real `docker build` did not.

Verified against PostgreSQL 17 and a built image:
- fresh database provisions from migrations; legacy 1.4.0-shaped database is
  adopted with posts and provider tokens intact; both idempotent across restarts
- container runs non-root (uid 1654) with read-only rootfs and all caps dropped
- agent card reports 1.5.0; /a2a returns 401 without a key and dispatches skills
  with one, reading real aggregates out of the adopted database
- /health/ready returns 503 while PostgreSQL is stopped and recovers to 200,
  while /health/live stays 200 throughout

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… optional

Inspecting the running cluster explains the deployment's 60 restarts. The
previous container exited "Completed" with this in its log:

  at SocialAgent.Providers.Mastodon.MastodonProvider.GetProfileAsync
  at SocialAgent.Host.Services.SocialMediaPollingService.ExecuteAsync
  at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync
  [INF] Application is shutting down...

An HttpClient timeout surfaces as TaskCanceledException, which is an
OperationCanceledException. Every background service filtered its handler with
`when (ex is not OperationCanceledException)` to avoid swallowing shutdown, so a
transient network blip talking to Mastodon was deliberately not caught, escaped
ExecuteAsync, and the host's default StopHost behaviour terminated the process.
Kubernetes then restarted the pod. The handlers now key off the stopping token,
which distinguishes real shutdown from an inner operation timing out. The same
filter is corrected in the providers, where it defeated the graceful-degradation
paths it was written for — a slow endpoint propagated instead of being reported
as "not connected" or skipped.

Adds regression coverage: SocialMediaPollingServiceTests asserts the service
survives a timeout, keeps polling other providers past one failure, and still
shuts down cleanly. Verified the timeout test fails against the old filter.

Threads ConfigMap and Secret references in deploy/k8s/deployment.yaml are now
optional. Threads is disabled and those keys are absent from the deployed
ConfigMap and Secret, so applying the manifest as-is would leave the pod in
CreateContainerConfigError.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant