Conversation
jwils
force-pushed
the
joshuaw/proto-integration
branch
4 times, most recently
from
September 15, 2026 14:20
49058c8 to
0ad7f71
Compare
jwils
force-pushed
the
joshuaw/proto-integration
branch
4 times, most recently
from
September 24, 2026 04:02
73e7853 to
628c847
Compare
jwils
changed the title
jwils
changed the base branch from
main
to
joshuaw/proto-decoding-validation
September 24, 2026 04:02
jwils
added this pull request to stack #1407
September 24, 2026 04:03
jwils
force-pushed
the
joshuaw/proto-integration
branch
from
September 24, 2026 04:13
628c847 to
4f6417c
Compare
jwils
removed this pull request from stack #1407
September 24, 2026 04:14
jwils
added this pull request to stack #1409
September 24, 2026 04:14
jwils
force-pushed
the
joshuaw/proto-integration
branch
2 times, most recently
from
September 24, 2026 13:55
8fc3d16 to
86f7895
Compare
jwils
force-pushed
the
joshuaw/proto-integration
branch
from
September 28, 2026 14:34
86f7895 to
e6831fe
Compare
jwils
force-pushed
the
joshuaw/proto-integration
branch
from
September 28, 2026 15:00
e6831fe to
f07ca01
Compare
jwils
force-pushed
the
joshuaw/proto-integration
branch
from
September 28, 2026 15:22
f07ca01 to
8b1b253
Compare
jwils
force-pushed
the
joshuaw/proto-integration
branch
2 times, most recently
from
September 29, 2026 20:22
56764a8 to
d088ad5
Compare
jwils
force-pushed
the
joshuaw/proto-integration
branch
from
September 29, 2026 21:55
d088ad5 to
e725d0e
Compare
jwils
force-pushed
the
joshuaw/proto-integration
branch
2 times, most recently
from
September 30, 2026 01:23
d621534 to
595ec8c
Compare
jwils
added a commit
that referenced
this pull request
Oct 1, 2026
…ormat (#1412) ## Why Protobuf ingestion should support everything JSON ingestion does. The indexer acceptance and integration specs are the best record of what JSON ingestion supports, but they only run against JSON. Running them against protobuf too would show each gap as the protobuf stack closes it. ## What Run all indexer acceptance and integration specs once per ingestion format, in `with json ingestion` and `with proto ingestion` contexts. The protobuf examples are `pending` because the indexer cannot ingest protobuf events yet. RSpec still runs pending examples and fails any that pass. As protobuf ingestion lands, the blanket `pending` gives way to `pending` metadata on the examples that still fail. RSpec then flags each of those examples as its gap closes. ## How A new `spec_support/.../ingestion_formats.rb` provides `for_each_ingestion_format`, which defines a context per format. Examples keep building events with the JSON factories. They process them through `process_indexing_events`, which calls `round_trip_indexing_events` (and `index_records` for the derived-type spec). For JSON, the round trip returns the events unchanged. For protobuf, it raises `NotImplementedError` for now; the protobuf stack will replace that with publishing to and decoding from protobuf. The names match the harness in #1388 so the stack can rebase onto this. The protobuf contexts build indexers from the stock schema defined with the protobuf extension, cached once per process. `schema_evolution_spec` dumps protobuf-only artifacts and leaves out `json_schema_version`. The malformed-event examples in `processor_spec` run only for JSON. Their events have wrong primitive types (e.g. `name: 17`), which a protobuf publisher can't encode. This also fixes the VCR hook. It skipped recording a cassette only when `ex.exception` was set. RSpec stores the expected failure of a pending example in `execution_result.pending_exception` instead, so failing pending examples got recorded. After a gap closed, replaying that cassette would fail the example with a VCR error, so it stayed pending and RSpec never reported it fixed. I checked this locally by making the protobuf round trip return events unchanged: before the fix, examples stayed pending; after it, RSpec reports them `FIXED`. ## Risk Low. Only specs and spec support change. The JSON examples are the same 52 as before. ## Testing No manual testing. On main, the proto variants already show real gaps beyond the stubbed round trip. Protobuf-only artifacts give "No ingestion adapters are available". Removing a field or type doesn't abort the dump the way the JSON schema version check does. ## Bigger picture This follows up on Myron's suggestion to run the indexer specs against both formats and use `pending` for the gaps. The protobuf ingestion stack (#1405 → #1408 → #1406 → #1388) can rebase onto this, implement `round_trip_indexing_events` for protobuf, and replace the blanket `pending` with per-example metadata. Schema evolution for protobuf will also need to encode old events with the old compiled descriptors. --------- Co-authored-by: myronmarston-toast <myron.marston@toasttab.com>
Define the unversioned wire contract and its tradeoffs. Correct the historical index destination and enum behavior assumptions, and explain safe renames, deletions, integer changes, and explicit field rotation.
Retain wire identities and retired contracts in the field-number sidecar. Map old publisher bytes to current public names and index destinations, and keep range checks active when record validation is sampled out. Decode and validate protobuf events through the shared indexer. Exercise historical publishers with separate descriptors under proto2 and proto3.
Expose a protobuf indexer wrapper that decodes a batch, converts valid events through the registered adapter, and sends them to the shared processor. Exercise protobuf and JSON ingestion through the shared indexing path.
jwils
force-pushed
the
joshuaw/proto-integration
branch
from
October 1, 2026 17:04
595ec8c to
f0d025e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The protobuf decoder and adapter need a public entry point that sends binary payloads through the shared indexer.
What
Expose a protobuf indexer wrapper and exercise the full ingestion path under proto2 and proto3.
How
The wrapper decodes batches or raw messages, returns individual failures, and indexes valid events through the shared processor. Integration tests check indexed values, source-only updates, out-of-order replay, warehouse output, and boot from dumped artifacts.
Risk
The wrapper is new and adds no transport-specific consumers. It uses the versionless validation and schema-evolution rules from #1406.
Testing
No manual testing.
Bigger picture
Final PR in the protobuf ingestion stack. This layer contains the wrapper and end-to-end tests; #1406 owns decoding, validation, and schema evolution.