Skip to content

hbbr: relay legs arriving together are never paired (race in make_pair_) #708

Description

@wb113

Describe the bug
make_pair_ in src/relay_server.rs looks up the waiting peer and registers a new one under two separate acquisitions of the PEERS mutex:

let mut peer = PEERS.lock().await.remove(&rf.uuid);   // lock #1
if let Some(peer) = peer.as_mut() { /* pair + relay */ }
else {
    PEERS.lock().await.insert(rf.uuid.clone(), Box::new(stream));   // lock #2
    sleep(30.).await;
    PEERS.lock().await.remove(&rf.uuid);
}

If both legs of a relay (same uuid) arrive between those two acquisitions, both miss and both take the else branch. The second insert replaces (and drops) the first stream, so neither leg is ever paired: one side is closed immediately and the other waits the full 30 s before being dropped. The client sees a failed relay connection.

Describe the environment

  • Install environment: docker (official rustdesk/rustdesk-server image, hbbs + hbbr, -k _, ENCRYPTED_ONLY=1)
  • Version in production: 1.1.14. Reproduced on current master (a7736be) built from source, unmodified.
  • Clients: RustDesk 1.4.7 (Windows).

How to Reproduce the bug

  1. Build hbbr from master and run it on a spare port: hbbr -p 31317 -k "".
  2. From a non-loopback address of that host (loopback connections go to hbbr's admin console), open two TCP connections, send the same RequestRelay frame (same uuid) on both back-to-back, then send a byte on one leg and expect it on the other. Script below.
  3. Some pairs are never relayed. Each lost uuid shows two New relay request <uuid> lines and no got paired.

Results, 200 simultaneous pairs per run (a control run with the two legs 0.2 s apart pairs 20/20):

build run 1 run 2
master a7736be, unmodified 177/200 176/200
master + patch below 200/200 200/200

All 47 unpaired uuids in the unmodified runs show the double New relay request signature. 1.1.14 behaves the same (125/150 unmodified, 150/150 patched).

Expected behavior
Both legs of a relay are always paired, regardless of how close together they arrive.

Additional context
In production (1.1.14) we found 11 such uuids in about four months, mostly between legs with very low latency to the relay (same LAN). On the target side the RustDesk log shows create_relay requested ... uuid: <uuid> with no following Connection opened, i.e. the connection attempt silently failed.

Open PR #642 restructures this block but still does the remove and the insert under separate PEERS acquisitions, so it does not close this race.

Suggested fix: do the check and the insert under one lock.

diff --git a/src/relay_server.rs b/src/relay_server.rs
index de1a7ea..609f0cb 100644
--- a/src/relay_server.rs
+++ b/src/relay_server.rs
@@ -468,7 +468,22 @@ async fn make_pair_(stream: impl StreamTrait, addr: SocketAddr, key: &str, limit
                     return;
                 }
                 if !rf.uuid.is_empty() {
-                    let mut peer = PEERS.lock().await.remove(&rf.uuid);
+                    // Check-and-insert under one lock: with separate lock acquisitions, two
+                    // legs of the same uuid arriving together can both miss and never pair.
+                    let mut peers = PEERS.lock().await;
+                    let mut peer = peers.remove(&rf.uuid);
+                    if peer.is_none() {
+                        log::info!("New relay request {} from {}", rf.uuid, addr);
+                        peers.insert(rf.uuid.clone(), Box::new(stream));
+                        drop(peers);
+                        sleep(30.).await;
+                        PEERS.lock().await.remove(&rf.uuid);
+                        return;
+                    }
+                    drop(peers);
                     if let Some(peer) = peer.as_mut() {
                         log::info!("Relayrequest {} from {} got paired", rf.uuid, addr);
                         let id = format!("{}:{}", addr.ip(), addr.port());
@@ -485,11 +500,6 @@ async fn make_pair_(stream: impl StreamTrait, addr: SocketAddr, key: &str, limit
                             log::info!("Relay of {} closed", addr);
                         }
                         USAGE.write().await.remove(&id);
-                    } else {
-                        log::info!("New relay request {} from {}", rf.uuid, addr);
-                        PEERS.lock().await.insert(rf.uuid.clone(), Box::new(stream));
-                        sleep(30.).await;
-                        PEERS.lock().await.remove(&rf.uuid);
                     }
                 }
             }
Repro script (Python 3, stdlib only)
# A/B harness for the hbbr pair race. Usage: hbbr_pair_race_test.py <port> <pairs> <gap_s> <non-loopback-host-ip>
import socket, sys, uuid, time
PORT, N, GAP, HOST = int(sys.argv[1]), int(sys.argv[2]), float(sys.argv[3]), sys.argv[4]

def frame(u):
    inner = b'\x12' + bytes([len(u)]) + u          # RequestRelay.uuid (field 2)
    msg = b'\x92\x01' + bytes([len(inner)]) + inner  # RendezvousMessage.request_relay (field 18)
    return bytes([len(msg) << 2]) + msg              # hbb_common 1-byte length header

ok = 0
for _ in range(N):
    u = str(uuid.uuid4()).encode()
    a = socket.create_connection((HOST, PORT)); b = socket.create_connection((HOST, PORT))
    for s in (a, b): s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
    time.sleep(0.02); f = frame(u); a.send(f)
    if GAP: time.sleep(GAP)
    b.send(f); time.sleep(0.1)
    try:
        a.send(b'ping-from-a'); b.settimeout(1); ok += (b.recv(64) == b'ping-from-a')
    except Exception:
        pass
    a.close(); b.close(); time.sleep(0.05)
print(f'port {PORT} gap={GAP}: paired+relayed {ok}/{N}', flush=True)

Usage: python3 hbbr_pair_race_test.py <port> <pairs> <gap_seconds> <non-loopback-host-ip>, e.g. python3 hbbr_pair_race_test.py 31317 200 0 10.0.0.5.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions