Skip to content

[rust] Dispatch independent table writes concurrently - #4478

Open
naivedogger wants to merge 1 commit into
apache:mainfrom
naivedogger:perf/rust-independent-table-sends
Open

naivedogger wants to merge 1 commit into
apache:mainfrom
naivedogger:perf/rust-independent-table-sends

Conversation

@naivedogger

Copy link
Copy Markdown
Contributor

Purpose

Closes #4476.

Remove cross-table head-of-line blocking in the Rust write sender. Requests for different tables on the same tablet server should not have to wait for one another's responses, matching Java's asynchronous per-table dispatch.

For two tables drained together to the same tablet server, a slow response from A currently delays even sending B:

Before                                              time ->
A: send ----------- wait for response ----------- complete
B:                                               send -- complete

After                                               time ->
A: send ----------- wait for response ----------- complete
B: send -- complete

Each table's request can now progress independently, so B can finish while A is still waiting. This changes cross-table scheduling only; existing per-bucket ordering and in-flight limits remain unchanged.

Brief change log

  • Group drained batches by table and enqueue one future per (destination, table) in the existing FuturesUnordered, instead of awaiting each table inside one node future.
  • Give each future ownership of its batches and response state. A table's terminal error does not cancel unrelated table futures.
  • Keep the existing per-bucket drain admission, sequence assignment, in-flight limits, retry/backoff and completion handling. No new worker threads, detached tasks or callback-executor changes.

Tests

Local validation of c301fe78b:

  • cargo test --offline -p fluss-rs --lib from fluss-rust: 795 passed.
  • Four new deterministic tests cover independent dispatch/completion over one established connection, table-scoped errors and permit cleanup, per-bucket sequencing/in-flight limits, and retry/backoff isolation. The dispatch regression also failed as expected against the original sender in a negative-control run.
  • Rustfmt, cargo clippy --offline -p fluss-rs --lib --tests -- -D warnings, and git diff --check passed.
  • Root-only ./mvnw -N -o verify spotless:check passed. The full Java module suite was not run.

Historical local A/B: the C++ benchmark used callback development base cb7258a3e with only this sender change toggled. The sender implementation matches this PR; the benchmark was not rerun on this standalone PR head.

One Connection, four producers writing four tables, four buckets per table, 256 MiB buffer, ACK all and idempotence enabled. Each run wrote 33,554,432 rows through Flush. Medians of three interleaved repetitions, in million rows/s:

Mode Original sender Per-table futures
Row append, no callback 6.303 6.610
Row append, lightweight callback 3.874 4.592
Arrow, 1024 rows/call, no callback 1.340 6.799
Arrow, 1024 rows/call, lightweight callback 1.373 6.156

Server offsets matched the submitted row counts; callback completion counts also matched. This was a small, highly compressible workload on an Apple M5 Pro with one local Docker tablet server. Single-table control ranges overlapped, so no single-table speedup is claimed. These short runs do not establish long-duration or failover performance.

API and Format

No public API, configuration, protocol or storage-format changes. The change removes incidental cross-table serialization, not the existing per-bucket ordering constraints.

Documentation

No new user-facing feature or configuration.

Generative AI disclosure

OpenAI Codex assisted with the implementation, tests and description. Human maintainer review is requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[rust] Writes to different tables wait for unrelated responses

1 participant