DTLS 1.3: accept a fragmented first ClientHello when cookies are disabled

[rizlik]

Description

First PR in a series to implement DTLS server that can handle bounded fragmented Client Hello.
This first PR allow to switch return routability check completely by disabling cookies.
A server with cookies disabled is compatible with other libraries that fragments ClientHellos to accomodate big PQC keyshares.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m0plus

• FLASH: .text +748 B (+1.1%, 67,847 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m3

• FLASH: .rodata.SetCipherList_ex.str1.1 +10 B, .text +1,080 B (+0.9%, 127,689 B / 262,144 B, total: 49% used)
• RAM: .bss +4 B (+0.5%, 820 B / 65,536 B, total: 1% used)

gcc-arm-cortex-m4

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.SetCipherList_ex.str1.1 +10 B, .rodata.str1.1 +61 B, .text +1,088 B (+0.6%, 206,444 B / 262,144 B, total: 79% used)
• RAM: .bss +4 B (+0.5%, 840 B / 65,536 B, total: 1% used)

gcc-arm-cortex-m4-baremetal

• FLASH: .text +768 B (+1.1%, 70,307 B / 262,144 B, total: 27% used)

gcc-arm-cortex-m4-crypto-only

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +61 B, .text +640 B (+0.4%, 179,549 B / 262,144 B, total: 68% used)

gcc-arm-cortex-m4-dtls13

• FLASH: .rodata +136 B, .text +1,536 B (+0.9%, 189,636 B / 1,048,576 B, total: 18% used)
• RAM: .bss +4 B (+0.5%, 856 B / 262,144 B, total: 0% used)

gcc-arm-cortex-m4-min-ecc

• FLASH: .text +704 B (+1.1%, 65,093 B / 262,144 B, total: 25% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .rodata +312 B, .text +2,304 B (+0.3%, 787,932 B / 1,048,576 B, total: 75% used)
• RAM: .bss +1,564 B (+1.1%, 138,508 B / 262,144 B, total: 53% used)

gcc-arm-cortex-m4-pkcs7

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +61 B, .text +832 B (+0.4%, 218,590 B / 262,144 B, total: 83% used)

gcc-arm-cortex-m4-pq

• FLASH: .rodata +80 B, .text +1,280 B (+0.4%, 304,152 B / 1,048,576 B, total: 29% used)
• RAM: .bss +8,804 B (+26.3%, 42,228 B / 262,144 B, total: 16% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .rodata +80 B, .text +1,408 B (+0.4%, 335,600 B / 1,048,576 B, total: 32% used)
• RAM: .bss +8,804 B (+26.3%, 42,224 B / 262,144 B, total: 16% used)

gcc-arm-cortex-m4-sp-math

• FLASH: .text +704 B (+1.1%, 65,093 B / 262,144 B, total: 25% used)

gcc-arm-cortex-m4-tls12

• FLASH: .rodata.SetCipherList_ex.str1.1 +10 B, .text +1,088 B (+0.9%, 128,445 B / 262,144 B, total: 49% used)
• RAM: .bss +4 B (+0.5%, 820 B / 65,536 B, total: 1% used)

gcc-arm-cortex-m4-tls13

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.SetCipherList_ex.str1.1 +10 B, .rodata.str1.1 +61 B, .text +1,216 B (+0.5%, 244,510 B / 262,144 B, total: 93% used)
• RAM: .bss +8,804 B (+26.4%, 42,176 B / 65,536 B, total: 64% used)

gcc-arm-cortex-m7

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.SetCipherList_ex.str1.1 +10 B, .rodata.str1.1 +61 B, .text +1,088 B (+0.6%, 206,444 B / 262,144 B, total: 79% used)
• RAM: .bss +4 B (+0.5%, 840 B / 65,536 B, total: 1% used)

gcc-arm-cortex-m7-pq

• FLASH: .rodata +80 B, .text +1,216 B (+0.4%, 305,048 B / 1,048,576 B, total: 29% used)
• RAM: .bss +8,804 B (+26.3%, 42,228 B / 262,144 B, total: 16% used)

gcc-arm-cortex-m7-tls13

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.SetCipherList_ex.str1.1 +10 B, .rodata.str1.1 +61 B, .text +1,216 B (+0.5%, 244,574 B / 262,144 B, total: 93% used)
• RAM: .bss +8,804 B (+26.4%, 42,176 B / 65,536 B, total: 64% used)

linuxkm-pie

• Data: __patchable_function_entries +832 B (+3.1%, 27,960 B)

linuxkm-standard

• Data: __patchable_function_entries +904 B (+1.8%, 50,864 B)

stm32-sim-stm32h753

• FLASH: .text +1,824 B (+1.0%, 191,116 B / 2,097,152 B, total: 9% used)

[Copilot]

🟡 Changes recommended

Unresolved moderate DTLS handling issues and critical test build-guard failures block approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds configurable DTLS cookie handling so servers can process fragmented initial DTLS 1.3 ClientHellos when cookies are disabled.

Changes:

• Adds cookie policy APIs and shared secret lifecycle handling.
• Updates DTLS accept paths and callback behavior.
• Adds DTLS 1.2/1.3 tests and documentation.

File summaries

File	Change	Final findings
wolfssl/ssl.h	Public cookie APIs and callback documentation	None
wolfssl/internal.h	Internal state and helper declarations	None
tests/api/test_tls13.c	TLS 1.3 API coverage	None
tests/api/test_dtls13.h	DTLS 1.3 test registration	None
tests/api/test_dtls13.c	DTLS 1.3 handoff and fragmentation tests	None
tests/api/test_dtls.h	Shared DTLS test declarations	None
tests/api/test_dtls.c	DTLS cookie, fragmentation, and policy tests	Critical (1 vote): Missing server-build guards cause compilation/link failures. Two tests also require both client and server support.
src/tls13.c	TLS 1.3 cookie and accept handling	Moderate (1 vote): Stateful processing must account for disabled HRR-cookie support.
src/ssl.c	Callback state reset	None
src/ssl_api_hs.c	Generic accept state handling	Moderate (1 vote): No-HRR-cookie builds can reject valid no-cookie DTLS 1.3 ClientHellos and must retain DTLS 1.2 HVR downgrade behavior.
src/ssl_api_dtls.c	Cookie policy and callback implementation	None
src/internal.c	DTLS cookie initialization	Moderate (1 vote): Reset chGoodCbDone when initializing a reused server connection.
doc/dox_comments/header_files/ssl.h	Public API documentation	None

Review details

Suppressed comments (3)

src/internal.c:2440

• This initializer is also used by wolfSSL_set_accept_state(), so a reused server object can enter a new handshake with chGoodCbDone still set from its previous no-cookie connection. DtlsNoCookieChGood() will then suppress the callback for the new ClientHello. Reset this per-connection bit alongside the cookie policy when initializing the server side.
  src/ssl_api_hs.c:931
• In a build with WOLFSSL_SEND_HRR_COOKIE disabled, DTLS 1.3 has no HRR-cookie path, but InitSSL_DtlsServer() still sets sendCookie to 1. This leaves the generic accept path stateless before parsing the version, so DoClientHelloStateless() rejects a valid DTLS 1.3 no-cookie ClientHello instead of transitioning to stateful processing; generic methods also need to retain the DTLS 1.2 HVR downgrade behavior.

        if (!ssl->options.sendCookie)
            ssl->options.dtlsStateful = 1;

src/tls13.c:17781

• This test assumes sendCookie means that a DTLS 1.3 cookie mechanism is available. With WOLFSSL_SEND_HRR_COOKIE disabled, InitSSL_DtlsServer() still initializes this bit to 1 for every DTLS server, so this path leaves dtlsStateful false and sends the first no-cookie ClientHello through DoClientHelloStateless(), which rejects it (and cannot reassemble a first fragment). Please make the stateful decision account for the compile-time absence of HRR-cookie support; the generic wolfSSL_accept() path needs the same correction.

        if (!ssl->options.sendCookie)
            ssl->options.dtlsStateful = 1;

• Files reviewed: 13/13 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Frauschi]

Went through the DTLS cookie-mode changes. The feature itself looks right, but making sendCookie the single master switch pulls several existing APIs along with it, and a few of those are silent behaviour changes for current callers.

The ones I would want addressed before merge are the WOLFSSL_NEITHER_END regressions - wolfSSL_send_hrr_cookie(), wolfSSL_disable_hrr_cookie() and wolfDTLS_accept_stateless() all now return SIDE_ERROR on objects that used to work - and wolfSSL_set_accept_state() silently undoing wolfSSL_disable_cookie(). I built the branch and reproduced all four. The rest are smaller: a missing state guard on the new enable/disable pair, the widened scope of wolfSSL_disable_hrr_cookie() which needs a ChangeLog entry, and a non-atomic failure path in wolfSSL_enable_cookie().

[Frauschi]

Rebuilt at d3dbe4c and re-ran the cases from last round; the side, ordering, state-guard, callback-retry and atomicity points are resolved. I withdraw the wolfSSL_disable_hrr_cookie() one (see thread). Left: ChangeLog/doxygen wording and two small cleanups.

[julek-wolfssl]

Fixes worth doing now (all small)

1. P2, diff-caused doc gap — wolfSSL_send_hrr_cookie dox attributes BAD_STATE_E solely to committed stateful processing, but the code also returns it when a secret must be generated and ssl->rng == NULL (proven by test_dtls12_cookie_secret_generate_fail/test_dtls13_hrr_cookie_secret_generate_fail). Add one clause to the BAD_STATE_E bullet (same clause optionally for wolfSSL_enable_cookie).
2. P2, headline-feature inertness undocumented — fragmented first CH only works when WOLFSSL_DTLS_CH_FRAG is compiled; dtls13ChFrag defaults on only with MLKEM, and the new wolfSSL_disable_cookie dox + ChangeLog never state the requirement. Document it (--enable-dtls-frag-ch / wolfSSL_dtls13_allow_ch_frag()), otherwise users in non-MLKEM builds get silent fragment drops. (Alternatively arm dtls13ChFrag whenever WOLFSSL_DTLS_CH_FRAG is compiled — that's a behavior change, so doc-first.)
3. P2, diff-caused, debug-config abort — InstallCookieSecret now calls wc_MemZero_Add on every rotation (pre-diff code only Added on size change). On WOLFSSL_CHECK_MEM_ZERO builds whose free path doesn't remove tracker entries (heap-hint/static-memory variant of wolfSSL_Free), ≥256 same-size secret rotations overflow the 256-entry cache and abort() (memory.c:256-283). I independently verified this mechanism. One-line fix: wc_MemZero_Check(dst->buffer, dst->length) in FreeCookieSecret after ForceZero, before XFREE (removes the entry; no-op when absent).
4. Two pre-existing stale dox lines the diff touched around: wolfSSL_DTLS_SetCookieSecret's "\return COOKIE_SECRET_SZ returned if the secret size is 0" (code returns 0) and the HRR-secret default-size fallback claim (WC_SHA_DIGEST_SIZE; code falls back SHA384→SHA512→SM3).