NIP-44: nip44.vectors.json still lists 65536 as an invalid encrypt length, contradicting the extended-prefix prose

[realmeylisdev]

The vectors file contradicts the prose it is checksummed by

44.md defines the 6-byte extended length prefix as valid, and the vectors file it pins still lists that length as an invalid encrypt length. Both are current, so an implementer following the prose fails their own conformance suite.

The prose

• 44.md:90-91 — "If length is less than 65536: prefix is 2 bytes (u16) […] If length is 65536 or greater: prefix is 6 bytes (2 zero bytes + u32)"
• 44.md:168 — "extended_prefix_threshold is 65536"
• 44.md:206-209 — pad emits concat([0, 0], write_u32_be(unpadded_len)) at or above the threshold
• 44.md:86 — "Minimum is 1 byte, maximum is 4,294,967,295 bytes"

The vectors file

44.md:303 publishes:

269ed0f69e4c192512cc779e78c555090cebc7c785b609e338a62afc3ce25040  nip44.vectors.json

A file matching that checksum contains:

"invalid": { "encrypt_msg_lengths": [0, 65536, 100000, 10000000] }

65536 and 100000 are valid encrypt lengths under the prose above. They sit in the invalid list.

Why this is awkward to work around

The checksum has not moved, so there is no regenerated file to fetch — re-vendoring reproduces the same contradiction. An implementer either edits a pinned artifact, which defeats the pin, or carries a local exception.

The inline table is a usable substitute, and it is correct

44.md:331-352 supplies the extended-prefix cases separately, noting the payloads are too large to inline in JSON. I implemented the decrypt side against that table and reproduced all three published payload_sha256 values byte-for-byte, including the 65535 u16 case as a control:

plaintext_len	prefix	payload_sha256	reproduced
65535	u16	6d8c2810d1e870fbaa1f0a0937126cca837a15f9260e27060c331d70a3c0bc84	yes
65536	extended	b7b4edb36ba92e267d322d56d9aebc22e7fa96ff52e3c12adc07f07a43cbc616	yes
65537	extended	eeb7c7c5373894ea2c1547cfd3ccb15d5a0b2d619da852e5c79df792dcc9e435	yes

So the table is trustworthy. The JSON is the piece left behind.

Suggestion

Regenerate nip44.vectors.json so invalid.encrypt_msg_lengths drops 65536 and 100000, fold the three boundary cases in beside valid.encrypt_decrypt_long_msg, which already uses checksums rather than inline payloads, and update the published sha256. Failing that, a sentence in 44.md saying the vectors file predates the extended prefix would save the next implementer the detour.

Happy to open a PR for the regenerated file if that helps.

[staab]

This does sound helpful, please open a PR

[realmeylisdev]

Correction to my own report: I understated which entries are wrong.

I wrote that invalid.encrypt_msg_lengths should drop 65536 and 100000. It should also drop 10000000 — max_plaintext_size is 4,294,967,295 (44.md:168), so all three are valid encrypt lengths under the current prose. Every entry except 0 is wrong.

The bounds the list can actually assert are 1 and 4,294,967,295, so [0, 4294967296] covers both ends rather than just the lower one.

Working on the PR now. One structural note, since it splits across two repos: nip44.vectors.json is not in this repo — it lives at paulmillr/nip44, and 44.md:303 pins only its sha256. I verified the file there hashes to exactly the published 269ed0f6…5040 and still contains the stale list, so the artifact and the checksum agree with each other and disagree with the prose. That repo was last pushed 2024-12-04.

So the fix needs two changes that cannot ship together:

• a PR here adding the note that the pinned vectors predate the extended prefix, pointing implementers at the inline table — mergeable on its own;
• a PR to paulmillr/nip44 regenerating the JSON, after which 44.md:303 needs the new checksum.

I will open both and link them here.

[realmeylisdev]

Opened two PRs, because the vectors file lives in paulmillr/nip44 and this repo only pins its checksum:

• Extended length prefix: JS implementation and regenerated test vectors (nips#1907) paulmillr/nip44#27 regenerates nip44.vectors.json and brings the JS reference implementation up to the current spec: 65536, 100000 and 10000000 leave invalid.encrypt_msg_lengths; the inline table from 44.md plus three larger cases join valid.encrypt_decrypt_long_msg; valid.calc_padded_len gains entries above the threshold; invalid.decrypt gains a payload that uses the 6-byte prefix for a short message.
• NIP-44: update test vectors checksum for the extended length prefix #2456 updates the checksum here and links the file.

One correction to the issue text: 10000000 had to go as well. It is below the 4,294,967,295 maximum, so a compliant implementation encrypts it without error.

[realmeylisdev]

Both PRs are open, @staab:

• NIP-44: note that the pinned vectors predate the extended length prefix #2457 — the note in 44.md. Mergeable on its own, nothing normative changes.
• Update vectors for the extended length prefix paulmillr/nip44#28 — regenerates nip44.vectors.json: invalid.encrypt_msg_lengths becomes [0, 4294967296], and the three boundary cases join encrypt_decrypt_long_msg. It also fixes testing.md, which listed "65536b plaintext" as an invalid case for the same stale reason.

Verification behind #28: all six encrypt_decrypt_long_msg entries — the three already there and the three added — reproduce both checksums and round-trip through decrypt. The three existing ones passing is the part that matters, since it shows the same pad still produces them unchanged and this only adds coverage.

Sequencing: #28 changes the file checksum from 269ed0f6…5040 to 3e1e8780…4684, so 44.md:303 needs bumping after it merges. I left that out of #2457 deliberately — pinning a checksum for a file that does not exist yet would be worse than the current state. Happy to send the bump as a follow-up, or fold it into #2457 if you would rather land both together once #28 is in.

One thing worth flagging: paulmillr/nip44 was last pushed 2024-12-04. If it stays dormant, #2457 is the only part that reaches implementers.

[staab]

I'm doing some initial validation, but this will need @paulmillr's eyes before merge. Why did you open two PRs to the nip44 repository? Seems to me you should only open the complete one, paulmillr/nip44#28 only has test vector changes.

[staab]

A smoke test from my clanker comes back ok, with one issue:

decodePayload now enforces no upper bound at all. 44.md still says implementations SHOULD reject oversized payloads early in decode_payload, before base64 decoding.

A configurable cap with a default would probably the correct solution.

Other smaller things:

• unpad reads getUint32(2) with no length guard. Safe through decodePayload (min 34-byte ciphertext), but v2.utils.unpad is exported, and called directly on a short array it throws a RangeError instead of "invalid padding".
• valid.encrypt_decrypt_long_msg now obliges every implementation to handle a 1,000,000-byte plaintext, while the spec says pick your own cap. Mild tension, not wrong.

[realmeylisdev]

Sorry for the noise: the fix was run twice in parallel on my side, which produced the second pair. #28 and #2457 are closed; #27 and #2456 are the ones to review.

Your smoke test is addressed in #27:

• decodePayload has a cap again, checked before base64 decoding and configurable per call: decrypt(payload, key, { maxPayloadSize }), default 4 MiB of base64 (about 2.5 MB of plaintext). The number is the part most open to preference.
• unpad rejects inputs shorter than their prefix with invalid padding instead of a RangeError.
• The 1,000,000-byte vector is gone; 100,000 is now the largest encrypt_decrypt_long_msg entry.

On invalid.encrypt_msg_lengths: #27 keeps [0] rather than #28's [0, 4294967296]. Harnesses materialize the plaintext, so a 2^32 entry either allocates 4 GiB or passes for the wrong reason ('a'.repeat(2 ** 32) throws a RangeError in V8).

#2456 carries the updated checksum.