Skip to content

Make async lock initialization thread-safe - #2073

Merged
martindurant merged 2 commits into
fsspec:masterfrom
Sanjays2402:fix/thread-safe-async-lock
Sep 23, 2026
Merged

martindurant merged 2 commits into
fsspec:masterfrom
Sanjays2402:fix/thread-safe-async-lock

Conversation

@Sanjays2402

Copy link
Copy Markdown
Contributor

get_lock() lazily initialized the module lock without synchronization, so concurrent first callers could receive different locks and initialize duplicate I/O loops. Initialize the process-local lock eagerly, recreate it after fork, and cover the race with a deterministic two-thread test.

Closes #1783

Tests: pytest -q (1493 passed, 215 skipped, 2 xfailed)

Eagerly create the process-local lock so concurrent first callers cannot
receive different lock objects. Recreate it after fork and add a
deterministic concurrent regression test.

Fixes fsspec#1783
@itzzdev09

Copy link
Copy Markdown
Contributor

The fix itself is correct, and actually fixes more than the PR description says: eagerly creating _lock at module import time (relying on CPython's import lock to guarantee it runs exactly once) genuinely eliminates the double-checked-locking race in the old get_lock(). It also fixes a separate, pre-existing bug along the way — reset_after_fork() on main currently does global lock; ...; lock = None, which references a module-level name lock that's never used anywhere else (the real variable is _lock), so the post-fork reset was silently a no-op. Worth calling that out explicitly in the PR description/changelog since it's a real deadlock-risk fix, not just a docs/naming cleanup.

I did find a problem with test_get_lock_is_thread_safe, though — I don't think it can actually fail. I reintroduced the exact pre-fix racy implementation on top of this branch:

def get_lock():
    global _lock
    if not _lock:
        _lock = threading.Lock()
    return _lock

and reran the test: it still passed in ~0.09s. The issue is the test calls fsspec.asyn.reset_lock() before installing the barrier-synchronized mock threading.Lock, and reset_lock() unconditionally assigns _lock = threading.Lock() using the real threading module at that point (mocking happens after). So by the time the two worker threads call get_lock(), _lock is already a truthy Lock object, if not _lock: is False for both, and the mocked/barrier-synced make_lock is never invoked by either the buggy or the fixed implementation. Under the new design there's no path back to a falsy _lock at all (import-time creation, and reset_lock() always assigns a fresh lock immediately), so the test can't currently exercise the race it's named for.

Might be worth dropping the reset_lock() call at the top of the test (or asserting on some other observable to prove the mock path really ran), so the test would actually fail if this regressed.

@martindurant
martindurant merged commit 20bc727 into fsspec:master Sep 23, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Thread safety of get_lock in asyn.py

3 participants