Skip to content

[trino] Add initial connector build and plugin skeleton - #4207

Open
yangshangqing95 wants to merge 5 commits into
apache:mainfrom
yangshangqing95:trino/init
Open

yangshangqing95 wants to merge 5 commits into
apache:mainfrom
yangshangqing95:trino/init

Conversation

@yangshangqing95

@yangshangqing95 yangshangqing95 commented Sep 3, 2026 •

Copy link
Copy Markdown

This PR introduces a native, read-only Fluss Trino connector proposed in #4197.

The connector supports metadata discovery and bounded reads from partitioned and non-partitioned Log Tables and Primary Key Tables.

Changes

  • Add a standalone fluss-trino Maven build
  • Support schema, table and column discovery, DESCRIBE, SHOW CREATE TABLE, and the "table$columns" system table.
  • Implement bounded Log Table reads using offset ranges determined during split planning.
  • Implement Primary Key Table reads using per-bucket snapshots.
  • Support scalar and nested type conversion, including ARRAY, MAP and ROW.
  • Add unit tests and integration tests covering metadata, distributed reads, partitioned tables, snapshot behavior and resource cleanup.
  • Apply Fluss Checkstyle, Spotless and Apache RAT checks.
  • Document installation, configuration, read semantics and known limitations.

Scope and limitations

Log reads use fixed per-bucket offset boundaries. Primary Key Table snapshots open independently when each bucket scan starts. Neither provides a globally consistent snapshot across buckets or partitions.

Predicates, aggregations, sorting and limits are evaluated by Trino. Pushdown, writes, DDL, Lakehouse reads, lookup, Union Read and task retries are outside this PR's scope.

Client-side scanner memory is not fully exposed by the Fluss client, so memory accounting remains incomplete.

The module stays outside the default Fluss Maven reactor to accommodate Trino's JDK requirements without changing those of the existing Fluss build. Connector source follows Fluss code style and Java 8 syntax conventions; the plugin itself requires JDK 25.

For Arrow-backed reads, add the following option to Trino's etc/jvm.config on every coordinator and worker, then restart the servers:

--add-opens=java.base/java.nio=ALL-UNNAMED

@polyzos

polyzos commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

@yangshangqing95, Once more, thank you for driving this. One thought here: Fluss has recently created a new organization to support its ecosystem
https://github.com/fluss-contrib

I think the fluss-trino-connector would be a great fit for that repo initially.
When it is mature, we can consider moving it into the main repo, but for now it will help give the connector a faster release cycle.

WDYT?

@yangshangqing95

Copy link
Copy Markdown
Author

Hi @polyzos,
Thanks again for continuing to drive this discussion. I think fluss-contrib is a very interesting direction, especially considering the faster release cycle and the ability to validate the connector independently.

However, especially after looking through some of the previous Trino-related PRs and issues, If we first develop the connector to feature completeness in fluss-contrib and then import the whole implementation into the main repository, we may end up with exactly the kind of large integration change that I have been trying to avoid with the incremental development model from the beginning.

That said, I do agree with an important part of your suggestion: fluss-contrib could be a very good place for fast prototyping, functional releases, and early validation. It would allow us to iterate quickly, validate the Trino integration against real usage, and evolve the implementation without coupling its release cycle to that of the main Fluss repository.

Ideally, I would see fluss-contrib more as an incubation and validation stage — a transitional state where we can stabilize the core architecture and semantics, quickly validate the end-to-end functionality, and use what we learn there to support incremental contributions to the main repository. Perhaps this could be a middle ground that is easier for everyone to agree on.

How we define "Mature" is also a question. I’ve also seen many promising ideas gradually lose momentum when there wasn’t a clear path to keep them moving forward. That is always a pity, and it is one of the reasons I think an incremental development model is valuable here. For me, "mature enough" would therefore be less about having every feature or optimization implemented, and more about having stable read semantics, solid integration tests, a reasonably stable public/configuration surface, and a clear compatibility and maintenance model. Even after reaching that point, there could still be plenty of incremental work to do. At the same time, I think the functional scope covered by the current design is already relatively clear, complete, and manageable. Given that scope, reaching a solid initial implementation within a release cycle appears quite achievable.

This is also why I still think incremental development in the main repository has value. . As long as each change is well-scoped, independently reviewable, properly tested, and does not negatively affect the existing Fluss build or runtime, we can continue evolving it through small PRs.

The isolated build environment I proposed for the Trino module is intended to preserve exactly this boundary.

I also think this could be an interesting model beyond the Trino connector itself. If it works well, fluss-contrib could potentially provide a lightweight incubation path for future POCs or experimental integrations: allowing larger ideas to be prototyped and validated quickly, while keeping contributions to the main repository incremental and reviewable.

So I really like the idea of using fluss-contrib for fast iteration and validation. I would just prefer to treat it as an incubation path rather than the canonical home.

I’d be very happy to discuss this further, and I’m looking forward to hearing your thoughts and continuing the discussion with the community.

Best,
Shangqing

@yangshangqing95
yangshangqing95 marked this pull request as draft September 22, 2026 00:42
@yangshangqing95
yangshangqing95 force-pushed the trino/init branch 2 times, most recently from 0e0cfe6 to c44ed03 Compare September 22, 2026 21:13
@yangshangqing95
yangshangqing95 marked this pull request as ready for review September 28, 2026 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants