Conversation
A typed stream append validated each event with validateWriteEvent, which re-read the entity type from Postgres for every event, so an append of N events cost 1 + N sequential registry round trips before it was forwarded to the durable-streams server. Resolve the effective schemas once per append and validate every event in memory against that snapshot: the same checks and error codes in the same order, the first invalid event reported, for two round trips instead of 1 + N. validateWriteEvent keeps its single-event signature for writeCollection.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4820
Problem
The stream-append route validates a typed append one event at a time (
src/routing/stream-append.ts:120-135), andEntityManager.validateWriteEventre-reads the entity type from Postgres on every call (getEffectiveSchemas->registry.getEntityType,src/entity-manager.ts:3716and:4035). An append of N events to a typed entity therefore costs 1 + N sequential registry round trips before it is forwarded, all N type reads returning the same row. With a local Postgres this is invisible; with Postgres ~60 ms away, a batch of ~10 events spends about 660 ms in the append path instead of about 120 ms. It also means the events of one append can be validated against different type revisions if the type is re-registered mid-batch.Fix
Resolve the effective schemas once per append and validate every event in memory against that snapshot.
EntityManager.validateWriteEvents(entity, events)does the singlegetEffectiveSchemaslookup and returns the first invalid event's error, in event order, ornull.validateEventAgainstStateSchemas(stateSchemas, event): unknown type ->422 UNKNOWN_EVENT_TYPE; delete withold_valuevalidatesold_value, otherwisevalue; an undefined payload passes; a validator failure ->422 SCHEMA_VALIDATION_FAILED.validateWriteEvent(entity, event)becomesreturn this.validateWriteEvents(entity, [event]), so its signature and behaviour are unchanged forwriteCollectionand the existing test.validateWriteEventscall. The write-token, fork-lock and stopped checks still run before validation; forwarding and the fire-and-forget side effects are untouched.Round trips per typed append go from 1 + N to 2. Untyped entities and appends with no events still cost one round trip. No SQL, drizzle schema, registry API, or error code changes.
Trade-offs
getEntityByStreamandgetEffectiveSchemasas they are (both have other callers), so the diff is small and the behaviour is easy to compare line by line.entities LEFT JOIN entity_typesregistry query would reach a single round trip, but it needs a new registry method duplicatinggetEntityByStream's/mainparsing andgetEffectiveSchemas' merge. Left as a possible follow-up.writeCollection(src/entity-manager.ts:2466-2467,:2530) still reads the type twice (getEffectiveSchemasthenvalidateWriteEvent). Deliberately not changed here to keep the diff minimal; it could call the new helper with the schemas it already holds.Tests
New
test/stream-append-route.test.tsdrives the realelectricAgentsStreamAppendRouterwith a realEntityManagerover a mocked registry (no Docker):resolves the entity type once for a multi-event append: a 10-event typed append is forwarded once andregistry.getEntityTypeis called once. Fails onmain(called 10 times), passes with this change.rejects the first invalid event in order without forwarding: event 2 has an unknown type and event 3 fails its schema; the response is422 UNKNOWN_EVENT_TYPEfor event 2 andforwardis not called.reports a schema failure at its position in the batch: event 3 fails its schema ->422 SCHEMA_VALIDATION_FAILED, not forwarded.skips validation for an untyped entity: forwarded,getEntityTypenever called.validates every event of an append against one schema snapshot: the registry returns a different type revision on the second read; bothobserved_itemevents are accepted. Fails onmain(the second event is checked against the second revision and rejected), passes with this change.test/electric-agents-manager-write-validation.test.tsgainsElectricAgentsManager.validateWriteEvents/validates a batch against the entity's own schemas when the type row is missing(type lookup returnsnull, the entity's ownstate_schemasare used, one lookup). The existingvalidateWriteEventdelete/old_valuetest is unchanged and still passes.Verification
pnpm install --frozen-lockfile pnpm -r --filter "@electric-ax/agents-server^..." buildRed: on
origin/main(140a5f4e7) with only the test files added:Green: on this branch:
The three red cases are the two route tests that encode the fix (lookup count, single snapshot) and the new manager test, whose method does not exist on
main; the other 18 cases pass on both sides, which is the "semantics unchanged" evidence. The fullagents-serversuite needs the docker-compose Postgres + Electric backend and was not run here.Files changed
packages/agents-server/src/entity-manager.ts: addvalidateWriteEventsand the privatevalidateEventAgainstStateSchemashelper;validateWriteEventdelegates to the batch method.packages/agents-server/src/routing/stream-append.ts: validate the whole append with onevalidateWriteEventscall instead of a per-event loop.packages/agents-server/test/stream-append-route.test.ts: new route-level tests for the lookup count, error order, untyped entities and the single schema snapshot.packages/agents-server/test/electric-agents-manager-write-validation.test.ts: unit test for the missing-type-row fallback ofvalidateWriteEvents..changeset/agents-server-append-schema-lookup.md: patch changeset for@electric-ax/agents-server.