Speed up hosted and vendored scans: concurrent API requests, parallel crawl, single-pass rewriters

[Mikola Lysenko (mikolalysenko)]

Summary

Hosted and vendored scans spent most of their time waiting on HTTPS requests sent one at a time, and in per-ecosystem work that re-read, re-parsed and re-wrote the same files. This PR:

• runs the network phases concurrently, with results still folded in input order;
• makes the crawlers and lock rewriters single-pass;
• memoizes per-run parses and prefetches and pre-stages vendored downloads from an exact plan;
• ships a small set of approved behavior changes, listed below.

Everything outside those changes is byte-identical to main: stdout, stderr, lockfiles, ledgers, vendored artifacts and exit codes. Every work package was A/B-gated for this, and every commit went through adversarial review (see Review).

Results

These are replay-proxy runs: recorded API responses served locally, with a fixed per-request latency. Release builds; wall time with instructions retired in parentheses; median of 2 interleaved reps. "Original" is the binary before this campaign, "main" is current main (3efdc31d, including #256), and "this PR" is the head (98d4b1d5).

depscan (100 ms/request)

Run	Original	main	This PR	vs main
hosted --dry-run	17.0 s (18.5 G)	17.2 s (18.6 G)	2.8 s (10.4 G)	6.2×
vendored, fresh	47.3 s (29.0 G)	47.4 s (28.9 G)	3.9 s (10.6 G)	12.2×
vendored, idempotent re-run	16.3 s (29.1 G)	16.3 s (29.1 G)	1.2 s (3.7 G)	13.6×

Polyglot monorepo (0 ms and 100 ms/request; original and main at 100 ms ran once, at 2-5 min per step)

Run	Original	main	This PR	vs main
hosted dry-run, 0 ms	27.5 s (290 G)	7.2 s (95 G)	1.7 s (26 G)	4.3×
hosted real run, 0 ms	27.2 s (290 G)	7.4 s (95 G)	1.7 s (27 G)	4.3×
vendored fresh, 0 ms	52.0 s (328 G)	50.3 s (369 G)	7.1 s (110 G)	7.1×
vendored re-run, 0 ms	16.5 s (216 G)	18.6 s (247 G)	2.8 s (34 G)	6.7×
hosted dry-run, 100 ms	104.1 s	85.5 s	4.7 s	18.3×
hosted real run, 100 ms	148.0 s	122.8 s	6.0 s	20.4×
vendored fresh, 100 ms	314.1 s	318.3 s	14.6 s	21.8×
vendored re-run, 100 ms	116.0 s	117.7 s	5.5 s	21.4×

The polyglot monorepo has ~158k files, 8 ecosystems and 560 patched packages, and exercises every service-backed backend at once. Per-ecosystem fixtures (cargo zed, maven, nuget, composer, uv, poetry, go, gem) were A/B'd in the batch comments, with the same byte-equality checks. For example, cargo hosted dry-run went from 204 G to 2.7 G instructions, and maven and nuget vendored re-runs dropped 72–93 %.

Output equality against main (this run; stdout canonicalized with timestamps masked and purl/path arrays compared as sets):

• depscan hosted, and polyglot hosted dry and real runs: stdout, stderr and the full work-tree digest are identical.
• Vendored runs: stderr, locks, manifests, the file list and every vendored artifact are identical. Markers are identical with vendoredAt masked. The polyglot state.json differs because of ledger v2 (X11).
• Vendored stdout differs only by documented changes:
  • refuse-locally-first moves records to the download phase: 3 on depscan and 80 on polyglot, each with the same code and text as main's vendor failed event;
  • the deferred fetch (X1b) drops vendor_fetched_missing events.
• No undocumented difference was found.
• Request counts: batch POSTs go 56 → 12 (depscan) and 147 → 30 (polyglot); by-package GETs are unchanged. Every run had 0 replay misses and 0 unknown purls, and exit codes match main.
• Every commit before the approved changes was A/B'd against the previous head. Stdout, stderr, tree digest, file list and request counts were identical across 11 hosted/vendored fixtures in 9 ecosystems.

What changed

Phase 2: network, crawl, rewriters

• utils/concurrent.rs is an ordered-concurrency helper (buffered, never buffer_unordered). Results, warnings and errors are folded in input order, and --debug lines are held back so they print in serial order.
• These run concurrently: batch POSTs, by-package GETs, hosted GET /patches/view, download views, VEX record fetches and lockfile-only pristine fetches. The batch loop reproduces the 401/403 proxy fallback exactly, at any chunk.
• The npm crawl does a parallel gather and then an ordered merge, on a 4-thread walk pool (SOCKET_WALK_THREADS). The nine crawlers run concurrently and merge in a fixed order. Directory listings, gem env, composer home and the site-packages query are asked once.
• Each lock is parsed once, indexed and spliced in one pass. This covers pnpm, npm, yarn classic/berry, bun, composer, uv/pylock/PEP 723, poetry/pdm, go.mod/go.sum and Cargo.lock. The old rewriters are kept as test oracles.
• Telemetry is sent in the background and flushed before the next stdout write.

Phase 3: per-ecosystem work

• Per-run parse memos for Cargo.lock/config, composer.lock, packages.lock.json, the pypi locks, the JS lockfiles and the hatch ledger. The cargo in-sync re-run and the pnpm lock split are memoized too, and the group-committed ledger is edited in place.
• Lazy vendored sources: a fetched package is extracted only where something reads it. Zip members are inflated on a bounded pool, and jars, nupkgs and wheels are rebuilt from memory. sha2 uses the aarch64 SHA-256 instructions.
• The cargo hosted rewriter searches only up to the next [[package]] block (zed: 204 G → 2.7 G). Maven coordinates come from the ~/.m2 path.
• Hosted redirect confirmation makes one Aho-Corasick pass per lock text. Independent hosted rewriter groups run concurrently, and hosted scan --vex reuses scan's npm crawl.

Exact grant plan and vendored prefetch

• The vendor prefetch plans downloads through each backend's own pre-flight gates. It requests exactly what the serial loop consumes: depscan fresh sends main's 71 grants, not 74. Parity tests pin the plan against the loop, refusal for refusal.
• The plan covers every service-backed ecosystem (cargo, composer, gem, golang, maven, nuget, pypi, npm). Its window slow-starts at 4 and grows to the API cap; a 128 MiB budget stops new downloads from starting.
• Verified archives are pre-staged next to each backend's stage and claimed with one rename. Stale pre-stage trees are swept at the start of the next wet loop.
• Writes and wiring stay serial in record order, and the outage circuit breaker is replayed at consumption time.

Polyglot monorepo work (pp1–pp3)

• pp1: vendored CPU memos (vendored re-run −50 % instructions).
• pp2: presence probe, parallel rewriter groups, walk pool, adaptive API windows (hosted wet at 100 ms: 18.4 s → 5.8 s).
• pp3: cross-ecosystem prefetch, pre-staging and early lock refusals (vendored fresh at 100 ms: 107 s → 15 s).

Tooling: scripts/perf/ holds the record/replay benchmark harness.

Behavior changes (approved)

Details and measurements are in the batch comments (X-series, pp2, pp3, batch 500). Each change is in CHANGELOG and CLI_CONTRACT.

• Deferred pristine fetch (X1b): a purl the ledger already covers, or a cargo crate the patch service serves, is not downloaded up front.
  • An in-sync --offline or network-less re-run now succeeds with already_vendored and exit 0.
  • Re-runs no longer report vendor_fetched_missing.
  • cargo and uv send fewer requests.
• fsync policy (X12): vendored artifacts are written without a per-file fsync. One durability barrier runs before every durable commit point and at the end of the run. Lockfiles and ledgers stay durable. On Windows, each artifact is still fsynced at write time.
• Per-run group commit with journal replay (V2): lock, manifest, config and ledger edits are committed once per run through .socket/vendor/.commit-journal.json. A crash leaves the pre-run wiring, or the next locked command rolls it forward.
  • A failed commit write now fails the run with the new top-level error vendor_commit_failed (exit 1). It replaces the per-purl vendor_state_write_failed / redirect_ledger_write_failed events.
  • After such a failure, locks and ledgers stay pre-run, and artifacts written in the loop remain until the next run.
  • vendor_stale_artifact_removed events now follow the per-package events.
• Ledger schema version 2 (X11): whole-file wiring snapshots (maven, nuget, pylock/PEP 723/hatch) are stored as {snapshot, ops} edits of the record's own original.
  • Version 1 ledgers are still read and revert byte for byte. Other ledgers stay byte-identical v1.
  • state.json shrinks: maven 29.8 → 15.1 MB, pylock 43.9 → 22.0 MB.
• Maven coordinates from the ~/.m2 path (MVN-1): canonical POMs are not opened once their top-level group directory is confirmed. A hand-placed POM that disagrees with its path reports the path's coordinates.
• -e <eco> crawl scope (MVN-4): without --prune/--sync, only the named ecosystems are crawled. lockfileOnlyPackages counts only them.
• API concurrency 32, adaptive: authenticated patch-API windows are n/4 of a step's requests, clamped to 8..32 (main sends them one at a time). VEX record fetches go up to 10, and the public proxy stays at 4. SOCKET_API_CONCURRENCY (1–32) overrides the cap; 1 is strictly serial, and on the proxy it can only lower the cap. A tight descriptor limit falls back to serial.
• CACHEDIR.TAG pruning: a node_modules inside or below a directory with a signed CACHEDIR.TAG is no longer crawled. The scan root is always crawled.
  • This affects scan, scan --prune / --sync, and discovery in apply, rollback, remove, repair, vendor, get and vex.
  • Under --prune / --sync, such a package counts as not installed and is garbage-collected unless a lockfile resolves it.
• Bounded 429/503 retry, and new --json warnings: every patch-API JSON call retries a 429/503 up to 3 times (SOCKET_API_MAX_RETRIES, 0–10).
  • Backoff honors Retry-After. A value over 30 s gives up at once. Without one, it waits 0.5 / 1 / 2 s with jitter.
  • All retries in a run end within a 60 s wall-clock window.
  • Never retried: 401/403, which still drive the proxy fallback, and the proxy's permanent 503 "Patch API is not configured".
  • New error suffixes say why retrying stopped: (gave up after 3 retries), (Retry-After <n> s exceeds the 30 s retry cap) or (the run's 60 s retry window has closed).
  • New additive run-level --json warnings: api_batch_failed and patch_details_failed.
  • On a token-less legacy proxy without POST /patch/batch, a package still throttled after its retries fails its whole batch query instead of being skipped silently.
• Refuse locally first (narrowed): on scan --mode vendored and get --mode vendored, some lock-text refusals now happen before the package's view and pristine source are fetched. These are the pnpm, yarn classic and yarn berry lock-text gates, and cargo's locked_version_mismatch when it is the first refusal.
  • Scope: only packages the vendor loop would hand to its backend, meaning installed, or resolved by the lock to a verifiable source. A package absent from the lock and not installed keeps its skipped / package_not_installed event, as before.
  • Each such package becomes a download.patches[] record with action: "failed" and the backend's errorCode and error. download.downloaded drops and download.failed rises by the same number.
  • vendor.events loses their failed events (and, for lockfile-only packages, their vendor_fetched_missing events), and vendor.summary.failed drops.
  • The nested vendor.status becomes success when these were the vendor step's only failures. The exit code and the top-level status do not change.
  • Because no view is fetched, the lock refusal now wins over outcomes that come from the view (paid 403, failed view fetch, no-applicable-files).
• Batch size 500: when neither --batch-size nor SOCKET_BATCH_SIZE is set, a batch holds up to 500 purls on the authenticated API. That's the server's own MAX_PURLS_PER_BATCH. The public proxy keeps 100.
  • A chunk whose body would exceed 256 KiB is split deterministically.
  • A mid-run downgrade to the proxy keeps the chunks already built. The test pins authenticated [500] and then proxy [500, 500, 1].
  • Output is unchanged; only the batch request count and the --help default change (depscan 56 → 12 batch POSTs).

Disclosed test edits. No existing test was ignored or deleted. Existing test edits pin an approved change:

• The two ledger-write-failure tests in covgap_commands_vendor.rs now expect the new vendor_commit_failed envelope error instead of a per-purl failed event (V2). The redirect test no longer asserts that the artifact is absent, because it is written before the commit fails. It asserts instead that nothing is committed or rewired and no journal remains.
• The v1 legacy-ledger fixture for cargo was regenerated with the pre-batch binary to match main's Wire vendored cargo patches through Cargo.toml and tag the copy #254 cargo wiring.
• The batch-size change made mechanical edits (batch_size: 100 → Some(100), and the parser default from 100 to None).
• The yarn cacheKey refusal tests again assert that the vendor step emits nothing for the refused package.

Known follow-ups

• The by-package GETs could start while later batches are still in flight. That would save about 0.5 s at 100 ms, but it has trade-offs, so it is not done here.
• Fix Cargo hosted workspace redirects #256 (on main) added about 1.5 G instructions to the cargo zed hosted dry run. That cost is on main too; it is not a regression from this PR.
• Pre-existing silent skips, unchanged here: the baseline view in scan/discovery.rs, and .ok()?? in repair_vendor.rs.
• Nested-project discovery for monorepos is planned as a separate PR.
• The crash/journal and fsync semantics are covered by unit, failpoint and injection tests, not by a live crash fixture.

Test plan

• cargo clippy --workspace --all-targets -- -D warnings is clean.
• cargo test --workspace --no-fail-fast on the head: 9,326 passed, 0 failed, 136 ignored (265 test binaries).
• New tests cover:
  • ordering under reversed latencies, and proxy fallback at any chunk;
  • crawler, rewriter, lazy-source and group equivalence against oracles;
  • plan-vs-loop parity for every npm flavor;
  • failpoint crash tests at the durability barrier and inside the group commit;
  • journal replay;
  • ledger v1 ↔ v2 round trips;
  • retry policy and window;
  • the early-refusal scope;
  • batch sizing, including the mid-run downgrade, verified by mutation.
• External check: depscan's 25-job cross-OS Maven matrix passed against f9fc6765 with real mvn.
• CI: all 417 checks pass on 98d4b1d5. Two Poetry native cells hit a production-API/network error; the same binary had passed them on 7d5d414e, and they passed on re-run.

Review

Each work package went through one to three adversarial reviewers, checking behavior equivalence, concurrency and test integrity. The retry and batch-size changes each had a dedicated reviewer, including mutation tests. Every finding was either fixed in a commit or refuted with evidence; the batch comments list them. The branch merged main (#256) as a merge commit and was never force-pushed.

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

WP-P2 pushed: cfeafb0c..5492794c (19 commits cherry-picked from perf/pp2, no conflicts)

Items

• H1: Aho-Corasick presence probe. Hosted redirect confirmation makes one multi-needle pass per lock text instead of one scan per needle.
• H4: parallel rewriter groups. Independent hosted rewriter groups run concurrently. A group runs inline if the OS refuses its thread. A group oracle pins the cross-rewriter reads.
• H3: embedded --vex reuses the crawl snapshot. Hosted scan --vex answers its copy lookup from scan's npm crawl. It walks node_modules again after a waiting prompt.
• DC-1: walk pool 4 / perf-cores + SOCKET_WALK_THREADS. The node_modules and Maven walks run on 4 threads by default (fewer on machines with fewer performance cores). SOCKET_WALK_THREADS overrides this.
• H2-code: adaptive windows. Each patch-API window is sized from its own request count.

Before → after (PR head cfeafb0c → 5492794c, release builds, replayed stores, single runs)

run	before	after
polyglot hosted dry Dmitri Iouchtchenko (@0) ms	3.61 s / 40.2 G instr	2.03 s / 24.9 G
polyglot hosted wet Dmitri Iouchtchenko (@0) ms	3.45 s / 40.6 G	1.56 s / 25.2 G
polyglot vendored fresh Dmitri Iouchtchenko (@0) ms	8.74 s / 107.1 G	7.87 s / 100.8 G
polyglot vendored rerun Dmitri Iouchtchenko (@0) ms	3.69 s / 40.0 G	2.67 s / 33.9 G
polyglot hosted wet Devin Soni (@100) ms	18.42 s	5.82 s (concurrency 32)
depscan hosted dry ×2 Dmitri Iouchtchenko (@0) ms	1.77 / 1.46 s (11.7 / 10.5 G)	1.49 / 1.46 s (10.0 / 10.0 G)
depscan vendored fresh ×2 Dmitri Iouchtchenko (@0) ms	1.70 / 1.60 s	1.71 / 1.59 s (flat)

Equivalence gate. In every polyglot step at 0 ms (hdry, hwet, vfresh, vrerun), canonical stdout, the full work-tree digest, request count and per-ecosystem purls are identical to cfeafb0c. Depscan is identical too: hosted dry stdout, and vendored fresh stdout, lock, workspace, package.json, state.json, files and markers. The polyglot tree has 5 signed CACHEDIR.TAG dirs (target/, build/, tools/patcher/target/, two .venv/). None contains a node_modules, so the pruning is not expected to change output here, and it does not.

⚠️ Behavior changes (approved)

• API concurrency 8 → 32. The window is adaptive: n/4 of a step's requests, clamped to 8..32. VEX record fetches go up to 10. The public proxy stays at 4. SOCKET_API_CONCURRENCY overrides the cap (1-32; on the proxy it can only lower it).
• CACHEDIR.TAG pruning. A node_modules inside or below a directory with a signed CACHEDIR.TAG is no longer crawled. The scan root is always crawled. This reaches scan, scan --prune / --sync (such a package counts as not installed and is garbage-collected unless a lockfile resolves it), and the discovery of apply, rollback, remove, repair, vendor, get and vex.
• Bounded 429/503 retry.
  • Policy: every patch-API JSON call retries up to 3 times. A Retry-After is honored; one over 30 s gives up at once, and one under the jittered first step (0, a past date) is floored to it. Without one, backoff is 0.5 / 1 / 2 s (steps capped at 8 s) with equal jitter. All retries in a run must end within a 60 s wall-clock window opened by the first retry, so parallel waits overlap instead of adding up.
  • SOCKET_API_MAX_RETRIES (0-10; 0 = old single attempt).
  • Never retried: 401/403, which still drive the proxy fallback, and the proxy's permanent 503 "Patch API is not configured" on any path.
  • New error text names why retrying stopped: (gave up after 3 retries), (Retry-After <n> s exceeds the 30 s retry cap), (the run's 60 s retry window has closed). An over-capacity 503 is the typed ApiError::ServiceUnavailable.
  • New additive --json run-level warnings: api_batch_failed and patch_details_failed.
  • Legacy proxy batch: on a token-less proxy without POST /patch/batch, a package still throttled after its retries fails its whole batch query instead of being skipped silently.
  • Listed under CHANGELOG "Changed (BREAKING)" and the semver note; the --json warnings are under "Added".

Tests

cargo clippy --workspace --all-targets -D warnings is clean. cargo test --workspace --no-fail-fast on the PR head: 9273 passed, 0 failed, 136 ignored.

Review notes

• pp2: 3 adversarial reviewers, then a fixer.
• The retry got a dedicated adversarial reviewer, then a fixer. Its fixes landed in fix(api): bound retries by a wall-clock window; never retry "not configured":
  • The summed 60 s budget starved concurrent requests; it is now a wall-clock window.
  • A Retry-After over the cap now gives up at once, and 0 / past dates are floored.
  • The "not configured" 503 is no longer retried or treated as throttling on per-package and view calls.
  • Throttle classification is typed instead of matched on a message prefix.
  • New test: a persistent 503 / 429 on the proxy batch makes zero per-package GETs.
• Pushed commit subjects keep their original ! markers. fix(scan)! (the --json warnings) is additive and is documented under "Added"; history was not rewritten.
• Known pre-existing follow-up, unchanged here: two paths still drop failures silently, the scan/discovery.rs baseline view and repair_vendor.rs's .ok()??.

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

Merged origin/main (#256) into this branch as 2b380e3. It's a merge commit, not a rebase or force-push.

CI was red because GitHub builds the PR merged with main. #256 removed CARGO_TOML_PACKAGE_RE and dropped the capture group from CARGO_TOML_REGISTRY_VAL_RE, and switched cargo manifest reads to parsing values as TOML (cargo_toml_string / cargo_toml_inline_string). The only code of this branch still on the old regexes was CargoRegistryPins::of, the one-pass-per-manifest pin extractor behind the vendored cargo PIN_MEMO.

Resolution:

• CargoRegistryPins::of now reads values the same way Fix Cargo hosted workspace redirects #256's cargo_socket_registry_pin does. That covers header-table and dotted keys via cargo_toml_string, and inline tables via cargo_toml_inline_string. cargo_socket_registry_pin is still its #[cfg(test)] equivalence oracle, now with Fix Cargo hosted workspace redirects #256's semantics.
• The V-7 pin-equivalence corpus now includes Fix Cargo hosted workspace redirects #256's spellings: literal strings, quoted inline keys, and a trailing comment on a header-table line.
• Everything else from Fix Cargo hosted workspace redirects #256 merged untouched: workspace-inheritance resolution, (name, version) lock owners, validate_cargo_toml_pins, and annotated [registries.*] headers. This branch's lock_block_end bounded search and its oracle, the parse memos, the presence probe and the parallel rewriter groups needed no changes.

Verification:

• clippy --workspace --all-targets -D warnings is clean.
• cargo test --workspace --no-fail-fast: 9286 passed, 0 failed, 136 ignored. That includes Fix Cargo hosted workspace redirects #256's new cargo tests and every cargo redirect and equivalence test.
• Cargo zed hosted dry run at 0 ms: output is identical to origin/main. It retires 9.76G instructions, against 9.82G on main@Fix Cargo hosted workspace redirects #256, 8.0G on this branch before the merge, 8.26G on main@Wire vendored cargo patches through Cargo.toml and tag the copy #254, and 203.5G on the old baseline. The roughly 1.5G Fix Cargo hosted workspace redirects #256 added shows up on main as well.
• Polyglot hosted dry and wet runs at 0 ms: stdout and full work-tree digests are identical to main@Fix Cargo hosted workspace redirects #256 (95dcaa380de0 / aead384e6318), at 27G instructions versus 95G on main.
• CI: all 417 checks pass on 2b380e3.

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

WP-P3 pushed: vendored-loop prefetch across ecosystems, pre-staging, early lock refusals (2b380e3 → 35a42b0, fast-forward)

• Per-backend gate refactor (9d4c024c): each service-backed backend (cargo, composer, gem, golang, maven, nuget, pypi) now has a pre-service prelude plus a service_preflight plan gate. The checks, their order and their outcomes are unchanged.
• V-1: exact cross-ecosystem prefetch with a slow-start window (1074b871, 4738e4c5): the download plan now covers every service-backed record, not only npm, and each backend's own gate keeps out the packages that backend refuses first. That keeps the plan exact: a refused package gets no download grant. The prefetch window starts at 4, grows by one per good answer up to the API's in-flight cap, and drops back to 4 on an availability failure. A 128 MiB budget limits how many fetched archives wait for the loop: it stops new downloads from starting, but downloads already in flight still finish, so held bytes can go over it.
• V-3: pre-staging (976f3e39, 8298526d): verified archives are extracted next to each backend's stage while the loop runs, and the backend claims the result with one rename. Trees left behind by a crashed or interrupted run are removed at the start of the next wet vendor loop. Dry runs delete nothing.
• V-4a: early lock-text refusal (895dd939, narrowed in 0bc98bf8): the behavior change described below.

Before → after (replay harness, polyglot monorepo and depscan)

run	PR head before (2b380e30)	after (35a42b0c)
polyglot scan --mode vendored, fresh, 100 ms RTT	107.2 s	14.8 s
polyglot vendored fresh, 0 ms	8.26 s (2250 req)	7.01 s (2167 req)
polyglot vendored re-run, 0 ms	2.82 s (791 req)	2.89 s (708 req)
depscan vendored fresh, 100 ms (×2)	7.50 / 7.32 s (344 req)	4.24 / 4.20 s (341 req)

Output equality

• V-1 + V-3 change nothing observable. Against this PR's previous head, polyglot fresh and re-run at 0 ms produce byte-identical work trees (digest abfa49ef5bc7 for both binaries, both steps), with 0 replay misses. Stdout is identical after sorting arrays. The only differences are the V-4a records listed below.
• The depscan vendored fresh run (a copy containing only the lockfiles) produces an identical lock, workspace, package.json, ledger, vendored files and markers. Stdout differs only by the V-4a records listed below.

⚠️ Behavior change (approved): lock-text refusals happen before the download

On scan --mode vendored and get --mode vendored (both the search path and the uuid path), some packages are now refused before their patch view is fetched:

• Which refusals: the pnpm, yarn classic and yarn berry lock-text gates, and cargo's locked_version_mismatch when it is the crate's first refusal. These packages are now refused before their patch view, and any pristine source, is fetched.
• Which packages: only packages the vendor loop would actually hand to its backend. That means a package installed on disk, or one the lockfile resolves to a verifiable registry source.
• What moves: each such package becomes a download.patches[] record with action: "failed" and the backend's exact errorCode and error.
• Count changes: download.downloaded goes down and download.failed goes up by the number of these packages. vendor.events loses their failed events, and vendor.summary.failed drops by the same number. For lockfile-only packages, their vendor_fetched_missing events also disappear.
• Nested status: vendor.status becomes success when these refusals were the vendor step's only failures (depscan: partialFailure → success).
• What does not change: the exit code and the top-level status (partial_failure, exit 1).
• What stays as it was: a package that is absent from the lock and not installed never reached a backend. It still downloads and stays skipped / package_not_installed, exactly as before. As a result, cargo refuses early only a crate that is installed at an unlocked version.
• Also unchanged: purls claimed by the hosted redirect ledger, and all purls when that ledger is malformed; other lock flavors; --dry-run.
• Precedence: because the view is never fetched, the lock refusal now wins over outcomes that come from the view (a paid-access 403, a failed view fetch, the no-applicable-files guardrail). The Bun refusal and the ledger's already-vendored skip still come first.
• Manifest-driven vendor: the failed events stay. A refused lockfile-only package that the lock resolves is no longer fetched, so there is no vendor_fetched_missing event and no registry request. With the registry unreachable, the lock-gate code replaces vendor_fetch_failed.
• Measured:
  • Polyglot (fresh and re-run): 80 packages move to the download phase (74 vendor_lock_entry_not_found, 4 vendor_override_conflict, 2 vendor_lock_entry_unsupported). For every one of them, the old vendor-loop refusal had the same code and wording. That saves 80 views and 3 registry tarballs per run.
  • Depscan: 3 packages move (2 vendor_lock_entry_unsupported, 1 vendor_override_conflict).

Tests

cargo test --workspace --no-fail-fast: 9316 passed, 0 failed (136 ignored); cargo clippy --workspace --all-targets -D warnings clean. The PR head is the exact commit that was tested.

New tests (10), each shown to fail when the fix it covers is removed:

• V-4a scope: pnpm and cargo, across scan, exact-purl get and vendor.
• The plan gate: plan_service_downloads directly, plus the composer zero-grant e2e, now with the refused package in the middle of the plan.
• Stale pre-stage cleanup: a core unit test, and an e2e covering --offline and --dry-run.
• Slow-start window: the window arithmetic, opening at 4 inside a 32-wide cap, and growing to exactly the cap.

The yarn cacheKey refusal tests again check that the vendor step emits nothing for the refused package.

Review

One adversarial reviewer raised 8 findings; all are addressed in the commits above:

• the V-4a scope was too wide (narrowed);
• the composer gate test never exercised the gate (fixed and unit-tested);
• stale .socket-prestage trees were never removed (now cleaned up at loop start);
• the prefetch window jumped from 4 to 32 (now slow-starts);
• the byte budget and pre-stage limits were undocumented (documented);
• the new precedence over view outcomes was undocumented (documented);
• the lock-text dry run passed an empty installed path (fixed);
• the yarn tests had lost their vendor-step assertion (restored).

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

Pushed 35a42b0c..7d5d414e: WP-P4. It contains one commit, the approved batch-size change.

⚠️ Behavior change (approved): batch size 500 on the authenticated API

• Default batch size: when neither --batch-size nor SOCKET_BATCH_SIZE is set, a batch holds up to 500 purls on the authenticated API. That's the server's own limit, MAX_PURLS_PER_BATCH in depscan batch.ts. The public proxy keeps 100.
• Explicit values: an explicit value applies on both endpoints, and 0 still means 1.
• Body-size cap: any chunk whose JSON body would exceed 256 KiB is split deterministically into consecutive chunks. 256 KiB is the public proxy's body limit. When no chunk reaches the cap, chunking is exactly the same as plain fixed-size chunking.
• Mid-run downgrade: if a run falls back to the proxy after a 401/403, it keeps the chunks it has already built. The proxy has no purl-count limit of its own and forwards to the same 500-purl upstream, so this is safe. The reviewer confirmed it directly: authenticated [500], then proxy [500, 500, 1].
• What changes for users: only the shape and count of batch requests, plus the --help default text. Output is unchanged. README, CLI_CONTRACT and CHANGELOG are updated.

Run, 0 ms	Batch POSTs	Canonical stdout	Work-tree digest
polyglot hosted dry	147 → 30	identical	identical
polyglot hosted wet	147 → 30	identical	identical
depscan hosted dry	56 → 12	identical	n/a

All non-batch requests are identical. depscan hosted dry at 100 ms: 3.30 s / 3.08 s → 2.68 s / 2.57 s.

Not done: DC-4, overlapping the crawl with the network

The plan assumed the list of batch purls was sorted, so non-npm chunks could go out while npm was still crawling. In fact, all_purls is in crawl order with npm first, and npm is the last crawler to finish on every benchmark. The most that could be overlapped is 18–84 ms, not the 1.5–2 s the plan projected, so this was dropped. The measurements are in the WP report.

A different overlap would be to start the by-package GETs while later batches are still in flight. That might save about 0.5 s at 100 ms, but it has trade-offs, so it is listed as a follow-up.

Tests: 9,325 passed, 0 failed, 136 ignored. Clippy is clean. There are 5 new unit tests and 4 new subprocess tests (scan_batch_sizing_e2e.rs). The only changes to existing tests are mechanical: batch_size: 100 became Some(100), and the parser's default assertion changed from 100 to None. One adversarial reviewer checked this against depscan's source and ran mutation tests. Verdict: push as-is.

🤖 Generated with Claude Code