[rust] Dispatch independent table writes concurrently - #4478
Open
naivedogger wants to merge 1 commit into
Open
naivedogger wants to merge 1 commit into
naivedogger wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes #4476.
Remove cross-table head-of-line blocking in the Rust write sender. Requests for different tables on the same tablet server should not have to wait for one another's responses, matching Java's asynchronous per-table dispatch.
For two tables drained together to the same tablet server, a slow response from A currently delays even sending B:
Each table's request can now progress independently, so B can finish while A is still waiting. This changes cross-table scheduling only; existing per-bucket ordering and in-flight limits remain unchanged.
Brief change log
(destination, table)in the existingFuturesUnordered, instead of awaiting each table inside one node future.Tests
Local validation of
c301fe78b:cargo test --offline -p fluss-rs --libfromfluss-rust: 795 passed.cargo clippy --offline -p fluss-rs --lib --tests -- -D warnings, andgit diff --checkpassed../mvnw -N -o verify spotless:checkpassed. The full Java module suite was not run.Historical local A/B: the C++ benchmark used callback development base
cb7258a3ewith only this sender change toggled. The sender implementation matches this PR; the benchmark was not rerun on this standalone PR head.One Connection, four producers writing four tables, four buckets per table, 256 MiB buffer, ACK all and idempotence enabled. Each run wrote 33,554,432 rows through Flush. Medians of three interleaved repetitions, in million rows/s:
Server offsets matched the submitted row counts; callback completion counts also matched. This was a small, highly compressible workload on an Apple M5 Pro with one local Docker tablet server. Single-table control ranges overlapped, so no single-table speedup is claimed. These short runs do not establish long-duration or failover performance.
API and Format
No public API, configuration, protocol or storage-format changes. The change removes incidental cross-table serialization, not the existing per-bucket ordering constraints.
Documentation
No new user-facing feature or configuration.
Generative AI disclosure
OpenAI Codex assisted with the implementation, tests and description. Human maintainer review is requested.