fix(projects): paginate and filter the project list server-side - #57
Merged
Merged
Conversation
`jh project list` sent the Projects GraphQL query with only `ownerId` set: no limit, offset or filter. Hasura therefore computed every project on the instance (57k on nightly.juliahub.dev) and the CLI filtered by owner client-side. The query costs the server roughly a second per row for an admin user because the `resources` field re-materialises the hasura.resources / access_control_resources views for every project, so the request always hit the 30 s client timeout — and Hasura does not cancel the Postgres statement when the client disconnects. Each nightly e2e run left two statements pinning a vCPU each for many hours. - fetch in pages of 100 with limit/offset and order_by created_at desc; --limit caps the total (default 100, 0 = all) - filter by owner in the GraphQL where clause (owner_id for --user, owner.username _ilike for --user <name>) instead of client-side - drop the resources / users / groups fields: resources only ever yields the constant Files/Datasets pseudo-resources and users/groups were never printed - reuse executeGraphQL instead of a hand-rolled HTTP client Measured on a nightly-shaped local instance (57.5k projects, admin caller, limit 100): 87.3 s -> 0.23 s. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
hey @pfitzseb and @thelonewolf1603 can you please take a look at this PR? |
Contributor
|
@claude review |
pfitzseb
reviewed
Sep 9, 2026
Review feedback: expose pagination as `--page=N` rather than a `--limit` that the CLI satisfies by looping over pages internally. `jh project list [--page N]` now issues exactly one GraphQL request per invocation: `limit: 100, offset: (N-1)*100`. The header still prints the server-side total and, when there is more than one page, which page is being shown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pfitzseb
reviewed
Sep 9, 2026
pfitzseb
left a comment
pfitzseb
left a comment
Member
There was a problem hiding this comment.
LGTM but would be good to get a review from @thelonewolf1603 too.
tanmaykm
approved these changes
Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
jh project listsent theProjectsGraphQL query with onlyownerIdas a variable — nolimit,offsetorfilter— and filtered by owner client-side. Hasura therefore computed every project on the instance (57,508 on nightly.juliahub.dev).That query is very expensive per row for an admin caller: the
resourcesfield re-materialises thehasura.resourcesview (UNION over all projects with a volatile plpgsql function) and theaccess_control_resourcesview for every project row — roughly 1 s/row. The request always hit the CLI's 30 shttp.Clienttimeout (nginx 499), but Hasura does not cancel the Postgres statement when the client disconnects, and nightly hasstatement_timeout = 0. Every nightlyTestProjectList+TestProjectListByUserrun left two statements each pinning a vCPU for hours (observed: two backends at 5h+ running exactly this SQL, started 30 s apart, matching the two 30.0 s 499s fromGo-http-client/2.0asgithub-ci@juliahub.com).Fix
limit/offset,order_by: created_at desc. New--limitflag caps the total (default 100,0= all); output says when it is showing a subset.--user→owner_id: {_eq: <me>};--user <name>→owner: {username: {_ilike: <escaped name>}}(same case-insensitive exact-match semantics as the oldEqualFold, LIKE metacharacters escaped).resources,users,groupsfrom the query.resourcesonly ever yields the constantFiles/Datasetspseudo-resources for every project (it's a view overprojects), andusers/groupswere fetched but never printed.executeGraphQLhelper instead of a duplicated HTTP client.buildProjectsFilter/escapeLikePattern; CLAUDE.md updated.Measurements
Nightly-shaped local instance (pg16 + JuliaHub
full_schema.sql+ Hasura v2.48.16 with the platform's metadata; 57.5k projects, 146 users, caller in the admin group), end-to-end/v1/graphql:Unbounded before-shape over all 57.5k rows extrapolates to ~17 h, which is what nightly's Postgres was doing.
make check(fmt, vet incl. e2e tag, unit tests, e2e compile) passes. Output format is unchanged soe2e/project_test.go'sFound N project(s)matcher still holds.Server-side follow-ups (JuliaHub repo, separate): a
statement_timeoutfor the Hasura DB role so abandoned queries die, and restructuringhasura.resources/access_control_resourcesso theresourcesfield stops costing ~1 s/row.🤖 Generated with Claude Code