BIP93: Generalize codex32 format for any hrp and fix typos - #2040
BenWestgate wants to merge 13 commits into
Conversation
Clarify codex32 format for different hrp values, specify master seed encoding standard, add new test vectors and enhance readability.
jonatack
added
Proposed BIP modification
| errors. The human-readable part is processed by first | ||
| feeding the higher bits of each character's US-ASCII value into the | ||
| checksum calculation followed by a zero and then the lower bits of each<ref>'''Why are the high bits of the human-readable part processed first?''' | ||
| This results in the actually checksummed data being ''[high hrp] 0 [low hrp] [data]''. This means that under the assumption that errors to the |
There was a problem hiding this comment.
The lengths limitations of the codex32 strings are working under the assumption that the HRP is not subject to error correction. We more or less cannot do that anyways as all sorts of various bech32 formats have appeared all with different checksums and characteristics. In order to run the checksum algorithm you have to know the prefix first in order to know which checksum algorithm to try.
This isn't really a problem in practice since there are only a small finite number of prefixes, and from context only a few are going to be applicable anyways.
There was a problem hiding this comment.
This was copied over from BIP-00173. Delete it?
Bech32 attempts to decode two checksums, a universal bech32 decoder could try decoding the string with the bech32, bech32m and codex32 checksums to discover the format.
Unless covering the HRP exceeds the max length at HD=9, 2 subsitutions in the HRP will always be detected by every format.
If HRP is swapped between formats the chances of false verification is:
-
1 in 2^65 for a "codex32 checksum" validating when the encoding was Bech32/Bech32m
-
~1 in 2^30 for "Bech32 checksum" validating when the encoding was Codex32.
There was a problem hiding this comment.
This text was certainly the design goal of BIP-173, but we are not using their checksum, and we haven't realized this part of their design in codex32 in part because our 13 character checksum unfortunately works only on relatively short strings.
Instead we process this HRP in this way because that is what BIP-173 does, and we still want the HRP to change the residue to catch random errors, so me might as well do it in the standard way.
Unless covering the HRP exceeds the max length at HD=9, 2 subsitutions in the HRP will always be detected by every format.
The problem is that our particular 13 character checksum's max length for its error detection and correction properties is limited to 93 bech32 characters. That's why our payload is limited to 74 characters add in 13 character checksum and 6 characters for the header and we get 93 bech32 characters, with nothing left over to detect or correct errors in the HRP. Yes, in cases where the payload is 72 characters or less, our error correction / detection properties extend to the low 5 bits of the ascii characters of a 2 character prefix, but that doesn't apply to 73 or 74 character payloads.
I don't know if we really want to get into these subtleties. I'm not even sure correcting and detecting errors in the HRP is useful to begin. If you are a hardware wallet expecting a master seed and someone gives you a "cl" codex32 string, you don't need a fancy error correction algorithm to detect the "cl" prefix is wrong; if it is a expecting a master seed then the "cl" prefix must be wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
If we REQUIRE every registered HRP be unique in the lower 5 bits then:
- we don't have to ever distinguish errors in the high bits they're a lookup table, VALID_HRP.
- the expanded data we covered by the checksum will be < 93 single counting hrp characters.
- With the valid HRP table, Correcting errors in the low bits, corrects any errors in the high bits.
- We know which checksum is being used by the length of the string, which is far simpler than a per hrp (impossible) design or one that double weights hrp characters towards max_length.
There was a problem hiding this comment.
With the valid HRP table, Correcting errors in the low bits, corrects any errors in the high bits.
It's not that simple. For maximum length codex32 string, errors in the high bits appear as errors in the checksum at the end of the string because for BCH codes, any polynomial longer than the maximum length (93 in our case) effectively wraps around.
Edit: And errors in the high bits are not going to be uncommon. Let me tell you the number of times I've mistaken a 5 for an S.
There was a problem hiding this comment.
Then let's guarantee to detect 8 errors, 13/15 if contiguous in the low bits, BIP-0173 style but not error correct the HRP. Trying different suspected HRPs will have to be the way to correct a damaged prefix, like our rationale suggests doing.
There was a problem hiding this comment.
mistaking a 5 for an S in the HRP counts as two errors.
There was a problem hiding this comment.
BIP-0173 error detection doesn't tell you where the errors are. In particular it doesn't tell whether the HRP is correct or not. You need to invoke the error correction to find locations.
| As with bech32 strings, a codex32 string MUST be entirely uppercase or entirely lowercase. | ||
| For presentation, lowercase is usually preferable, but uppercase SHOULD be used for handwritten codex32 strings. | ||
| If a codex32 string is encoded in a QR code, it SHOULD use the uppercase form, as this is encoded more compactly. | ||
| The lowercase form is used when determining a character's value for checksum purposes. |
There was a problem hiding this comment.
This doesn't make sense. The lowercase form and uppercase form of Bech32 characters have the same value.
There was a problem hiding this comment.
Not for HRP which needs to be lower cased during decoding or bech32_hrp_expand(hrp) would return a different result.
This line is repeated from the test vectors, why explain the rules about case in the vectors instead of up here?
There was a problem hiding this comment.
I guess we should reword this to make it more clear that the relevance is for the HRP.
There was a problem hiding this comment.
"When constructing or verifying a checksum, the human-readable part MUST be interpreted in lowercase, as specified in BIP-0173."
There was a problem hiding this comment.
I might say "MUST be converted to lowercase" instead.
There was a problem hiding this comment.
That seems to imply mutating the string when verifying a checksum.
Something BIP-93 omitted was that encoders should always emit lower-case strings. Did we relax that requirement?
Currently I have the sentence as:
"Encoders MUST emit lowercase; decoders MUST reject mixed-case and MUST lowercase the human-readable part during checksum verification."
And I am adding a section with a codex32_encode and codex32_decode definitions as I think it's easier to see these rules in code than english.
Uppercase/lowercase
The lowercase form is used when determining a character's value for checksum purposes.
Encoders MUST always output an all lowercase Bech32 string. If an uppercase version of the encoding result is desired, (e.g.- for presentation purposes, or QR code use), then an uppercasing procedure can be performed external to the encoding process.
Decoders MUST NOT accept strings where some characters are uppercase and some are lowercase (such strings are referred to as mixed case strings).
For presentation, lowercase is usually preferable, but inside QR codes uppercase SHOULD be used, as those permit the use of alphanumeric mode, which is 45% more compact than the normal byte mode.
There was a problem hiding this comment.
As far as I'm concerned both all lowercase and all uppercase strings are valid, so encoders can produce either format with lowercase is generally preferred. I'm not really sure what BIP-173 thinks it is achieving by talking about encoders being somewhat different from a post-processing step. Maybe they are just trying to say that when creating a checksum, of course, a lowercase HRP must be used.
That seems to imply mutating the string when verifying a checksum.
This is exactly what the BIP-173 reference python decoder does:
https://github.com/sipa/bech32/blob/master/ref/python/segwit_addr.py#L78
However, "The lowercase form is used …" is also fine wording though.
There was a problem hiding this comment.
"Decoders MUST use the lowercase form of the human-readable part during checksum verification."
… case in checksum
| # Choose a threshold value ''k'' between 2 and 9, inclusive | ||
| # Choose a 4 bech32 character identifier | ||
| #* We do not define how to choose the identifier, beyond noting that it SHOULD be distinct for every master seed the user may need to disambiguate. | ||
| #* We do not define how to choose the identifier, beyond noting that it SHOULD be distinct for every secret the user may need to disambiguate |
There was a problem hiding this comment.
Should we say "secret or set of shares" because this is the reshare case you mentioned that SHOULD have a unique identifier?
Here we make it sound like it's OK to reuse an identifier if the secret is the same which is false.
There was a problem hiding this comment.
I used "set of shares" in for an existing secret and "secret" in for a fresh secret.
This is technically correct, no need to say both "secret and set of shares" in existing secret, if you follow that process you always get a fresh set of shares and that is what needs to be uniquely identified not the secret per se.
Clarify codex32 specification and examples for encoding and decoding processes, including detailed explanations of parameters and checksum handling.
|
Yes let's keep the silly exception for now for the sake of getting a agreeable PR. We should hammer out the master seed bit-size restrictions in a separate PR. If you want to say that length 96 ms seeds are deprecated that's okay too. But I still want to argue for the merits of 160 bit master seeds. |
| def codex32_verify_checksum(hrp, data): | ||
| if len(data) >= 96: # See Long codex32 Strings | ||
| return ms32_verify_long_checksum(data) | ||
| return codex32_verify_long_checksum(bech32_hrp_expand(hrp) + data) | ||
| if len(data) <= 93: | ||
| return ms32_polymod(data) == MS32_CONST | ||
| return codex32_polymod(bech32_hrp_expand(hrp) + data) == CODEX32_CONST | ||
| return False |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
I can probably write this code out if you want, but my thoughts are we should have codex32_decode, an independent long_codex32_decode and an ms_decode that can call both of them.
There was a problem hiding this comment.
That makes sense! Bech32 has an encode/decode for the format and then a separate encode/decode function for segwit addresses.
However they do encode/decode both Bech32/Bech32m checksums at once.
We need codex32_encode and codex32_decode function to handle both checksums. That has to be format level, not application level in order to detect/correct HRP errors.
There was a problem hiding this comment.
Below is untested code that is approximately what I'm thinking
def bech32_hrp_expand(s):
return [ord(x) >> 5 for x in s] + [0] + [ord(x) & 31 for x in s]
CODEX32_CONST = 0x10ce0795c2fd1e62a
def codex32_polymod(residue, values):
if len(values) > 93:
return False
GEN = [
0x19dc500ce73fde210,
0x1bfae00def77fe529,
0x1fbd920fffe7bee52,
0x1739640bdeee3fdad,
0x07729a039cfc75f5a,
]
for v in values:
b = (residue >> 60)
residue = (residue & 0x0fffffffffffffff) << 5 ^ v
for i in range(5):
residue ^= GEN[i] if ((b >> i) & 1) else 0
return residue
CODEX32_LONG_CONST = 0x43381e570bf4798ab26
def codex32_long_polymod(residue, values):
if len(values) > 1023:
return False
GEN = [
0x3d59d273535ea62d897,
0x7a9becb6361c6c51507,
0x543f9b7e6c38d8a2a0e,
0x0c577eaeccf1990d13c,
0x1887f74f8dc71b10651,
]
for v in values:
b = (residue >> 70)
residue = (residue & 0x3fffffffffffffffff) << 5 ^ v
for i in range(5):
residue ^= GEN[i] if ((b >> i) & 1) else 0
return residue
def codex32_verify_checksum(hrp, data):
return codex32_polymod(1, bech32_hrp_expand(hrp) + data) == CODEX32_CONST
def codex32_verify_long_checksum(hrp, data):
return codex32_long_polymod(1, bech32_hrp_expand(hrp) + data) == CODEX32_LONG_CONST
def codex32_create_checksum(hrp, data):
polymod = codex32_polymod(1, bech32_hrp_expand(hrp) + data + [0] * 13)
if polymod:
polymod = polymod ^ MS32_CONST
return [(polymod >> 5 * (12 - i)) & 31 for i in range(13)]
return False
def codex32_create_long_checksum(hrp, data):
polymod = codex32_long_polymod(1, bech32_hrp_expand(hrp) + data + [0] * 15)
if polymod:
polymod = polymod ^ MS32_LONG_CONST
return [(polymod >> 5 * (14 - i)) & 31 for i in range(15)]
return False
def ms32_verify_checksum(data):
if len(data) >= 96:
return codex32_verify_long_checksum("ms", data)
return codex32_polymod(codex32_polymod(1, bech32_hrp_expand("ms")), data) == CODEX32_CONST
def ms32_create_checksum(data):
if len(data) > 80:
return codex32_create_long_checksum("ms", data)
polymod = codex32_polymod(codex32_polymod(1, bech32_hrp_expand("ms")), data + [0] * 13)
polymod = polymod ^ CODEX32_CONST
return [(polymod >> 5 * (12 - i)) & 31 for i in range(13)]
As you can see, I think it is up to the particular application to handle switching between the long codex32 format and the regular codex32 format.
There was a problem hiding this comment.
We can't assume "ms" to know which checksum to verify.
If there's an hrp substitution error, we need to know which checksum is being used to detect/correct it, but if which checksum depends on which hrp application, instead of just codex32 string length, then we're stuck.
That made our length 96 exception a bug, as how can a decoder know this rule applies if it can't detect the integrity of what determines its applicability?
We need a codex32_decode function that if it validates it has the correct HRP or more than 8 errors, so applications can't choose their checksum, the format defines which to use.
Like BIP-0173's HRP detection assumption, our error correction guarantee only applies to lower (or upper) 5 bits of HRP characters. As the swaps that produce an upper bit change are very unlikely. But we can guarantee to correct 2 "double" errors.
There was a problem hiding this comment.
The short/long format must be format level.
def codex32_encode(hrp, data, spec):
"""Compute a codex32 string given HRP and data values."""
combined = data + codex32_create_checksum(hrp, data, spec)
return hrp + "1" + "".join([CHARSET[d] for d in combined])
def codex32_decode(codex=""):
"""Validate a Codex32/Codex32 Long string, and dermine HRP and data."""
if (any(ord(x) < 33 or ord(x) > 126 for x in codex)) or (
codex.lower() != codex and codex.upper() != codex
):
return None, None, None
codex = codex.lower()
pos = codex.rfind("1")
if pos < 1 or pos + 20 > len(codex) or pos + len(codex) > 1023:
return None, None, None
if not (codex[pos + 1].isdigit() and all(x in CHARSET for x in codex[pos + 1 :])):
return None, None, None
hrp = codex[:pos]
data = [CHARSET.index(x) for x in codex[pos + 1 :]]
spec = codex32_verify_checksum(hrp, data)
if spec is None or codex[pos + 1] == "0" and codex[pos + 6] != "s":
return None, None, None
return hrp, data[: -13 if spec is Encoding.CODEX32 else -15], specThere was a problem hiding this comment.
I can probably write this code out if you want, but my thoughts are we should have
codex32_decode, an independentlong_codex32_decodeand anms_decodethat can call both of them.
Yes to ms_decode but that one should go all the way to bytes. Do we add it to this PR in the master seed format section along with ms_encode?
I'd prefer if codex32_decode returned a spec it's a hassle to check the string length everywhere to know which checksum is being used. But the code in my latest commit isn't ugly.
|
No.
…On Thu, Nov 27, 2025, 2:20 AM Ben Westgate ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In bip-0093.mediawiki
<#2040 (comment)>:
> +def codex32_verify_checksum(hrp, data):
if len(data) >= 96: # See Long codex32 Strings
- return ms32_verify_long_checksum(data)
+ return codex32_verify_long_checksum(bech32_hrp_expand(hrp) + data)
if len(data) <= 93:
- return ms32_polymod(data) == MS32_CONST
+ return codex32_polymod(bech32_hrp_expand(hrp) + data) == CODEX32_CONST
return False
Needs to become now to:
def codex32_verify_checksum(hrp, data):
combined = bech32_hrp_expand(hrp) + data
if len(combined) >= 96:
return codex32_verify_long_checksum(combined)
if len(combined) <= 93:
return codex32_polymod(combined) == CODEX32_CONST
return False
Missing:
- the thorny zero length "ms" rule.
- the check in codex32_decode() for the upper long codex32 length
limit.
Because of this new max length rule rule we have the curious situation
where valid "long codex32" strings can actually be shorter overall (and in
data part characters) than regular codex32.
May want to rename that format any thoughts?
Ex:
"long" codex32 format: 10 hrp characters + 1 + 6 header characters + 54
payload characters + 15 checksum characters = 86
codex32 format: "ms" hrp characters + 1 + 6 header characters + 74 payload
characters + 13 checksum characters = 96
—
Reply to this email directly, view it on GitHub
<#2040 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BNFR33IQPDGIR4ICII5KOVT362Q2BAVCNFSM6AAAAACM4BCWD6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTKMJTHAYTQNRYGE>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
There was a problem hiding this comment.
This is what I have:
- compatibility encoding for BIP-39, allowing to Shamir split mnemonic & extended private key wallets
- HRP:
cc - encodes chaincode + private key of BIP-32 master extended key (64 bytes)
cc10zcvjs5klr60nyt8usd553sge7r5glcy2ztwfv2d2smmcs7m3mq6dduwavccnjzjchlkffjfx8p3cjjx64q9vkxdt8q9qzuu3s8jfgjysa5pc5nezf2qkfhqpfwf
- HRP:
I only have one HRP, I do not differentiate between testnet/mainnet, even tho I use extended key data, I'm also using it for menmonics, where I first generate master BIP-32 key and then use those values for the codex32 secret share. DO you consider the lack of testnet/mainnet separation an issue?
Your addition of HRP into checksum definitely broke my tests wrt checksum for cc hrp secrets (not an issue, I haven't released yet - but I'm planning to in few weeks)
My HWW implementation is pretty much in accordance with https://github.com/BlockstreamResearch/codex32/blob/master/docs/wallets.md . My implementation is not ECW. I even provide generate support for secret share S. I only allow to generate 128 & 256 bit MS secrets (but allow to import also 512 bit). In short:
- TRNG 256 entropy bits
- r = sha256(sha265(entropy))
- x = r[:byte_len]
- x is new master secret, and default ID is 20 MSB from master XFP (but user can change if he wishes to)
What are the chances of this patch-set to be accepted? Is this spec stable enough to start releasing it ?
|
How I generate non-secret shares:
|
"bc" and "tb" for Bech32 addresses were an upgrade in human-readable prefix from the base58 encoding. I consider it a regression if you use less characters to encode a human-readable prefix than the base58 extended key format did. "xpriv" is an option here.
Does your format need a 65th byte for the public key that is zero when encoding private keys? There are many advantages to the strings needing disambiguation having the same byte length.
Yes, this is a huge regression from the current bip32 extended key format we want to upgrade. Mostly that I can't tell by looking at the descriptor if it's for real funds or not.
HRP was always in the checksum, it just was pre-computed for "ms" so the checksums for other HRP were wrong. I noticed when I tried to validate the CLN HSM secret examples in my python-codex32 package.
@roconnor has a PR in codex32 that does ECC you could test.
I have a codex32 PR to update wallets.md guidance for generation, you may see something useful, especially in the HWW case. In short:
You can and probably should use the entropy bits directly. If they lack entropy, sha256d is an illusion of security.
It will need wider community review than us. But there's comments by P. Wuille as far back as 2020 stating a 4 error correcting bech32 encoding of extended keys is needed. So high acceptance changes once it's correct and shiney. This spec PR will not change anything that affects your encoding of ~78 bytes or whatever an extended key has. We're mostly debating behavior at the limit between short and long checksums. Yours unambiguously use long codex32. |
It is unsafe to child derive shares from the secret they recover. They should be independently random. When part of the secret is compromised and an attacker tries to brute force the rest: the dependent relation between the secret and share A allows an attacker with k-1 shares or share A to check his guesses against this. This is far faster than checking an address. |
I see now...
I do not want to use randomness here, as I want to split existing secret, and I require the "split" to be deterministic, so that if user is splitting the exact same secret, uses same hrp, same threshold, same id, and same number of shares - application always produces the exact same shares. I could add an option to to choose, if random, or deterministic split, but deterministic is a hard requirement. ...also it is 5 hardened derivation steps plus hmac_sha512
there are plenty other brute-force options if attacker has part of secret, I do not consider this scenario of yours to be something I should optimize for
I do not encode extended key (or full extended key), I only encode chaincode + privkey, without any other data as I just want to be able to restore naked xpriv from it, without any more meta extended keys carry. As I use it for both mnemonics and extended keys. That is why I dismissed the idea of doing testnet/mainnet differentiation as I consider my 64bytes to be the "secret" |
|
@BenWestgate your "application agnostic checksum selector" looks good to me, in that (I think) neither I nor roconnor will be bothered by it, since it continues to work unchanged for codex32 while being easier to read. But I guess you should open a separate (small) PR to change the reference implementation so we can discuss it independent of anything else. |
|
@apoelstra: we can avoid the legacy exception by invalidating these three "ms" byte lengths. That creates an even easier to read checksum-selector: def checksum_for_shape(hrp, data):
expanded = 2 * len(hrp) + 1 + len(data)
if expanded <= 93:
return CODEX32
if 96 <= expanded <= 1023:
return CODEX32_LONG
raise InvalidLength |
|
Yeah, I see what you're getting at. Basically, we unconditionally consider the HRP to be covered by the checksum for purposes of deciding whether the checksum is long enough. In the current BIP text, we exclude the HRP on the basis that if the HRP were corrupt, you don't even know if you're using codex32 (though I forget whether we said that). So we used exactly your function in the current BIP text but used Your proposal is the same except we add in the HRP, effectively shifting the length boundary by 5. (We continue to have a 2-byte gap because the checksums' lengths differ by 2.) So people who currently have data lengths of 89 through 93 will be affected. You are telling them to use codex32_long instead of codex32. Subtracting the checkum, this is 76 through 80; subtracting the 6-character header, this is 70 through 74; converting to bytes this is 43 through 47. (Without dropping the 6-character header, it's byte lengths 48 through 50...but if you don't have the header, you aren't doing codex32 and shouldn't be using the So this is technically a breaking change for codex32 users but IMO it seems fine to me. It's hard for me to imagine anybody using byte lengths greater than 32 (maaybe 33 for a BIP39 seed or for a pubkey or something) at all, let alone the weird lengths 43 through 47. So concept ACK. But please open a new PR instead of making proposals in the comments of an unrelated closed PR :P. |
|
Done see you at #2258. |
Include the five values from the expanded "ms" HRP when selecting the checksum. Use the regular checksum through expanded length 93, reject lengths 94 and 95, and use the long checksum through 1023. This intentionally invalidates old regular-checksum encodings of 44-, 45-, and 46-byte seeds. Those seed lengths remain supported using the long checksum. Enforce each checksum's maximum period in its variant verifier, while keeping codex32 format selection in ms32_verify_checksum and ms32_create_checksum. Add reproducible vectors for 43- through 47-byte seeds and direct checks for the 93/94/95/96 and 1023/1024 boundaries. refs bitcoin#2040
Specify the human-readable part as in BIP-0173 instead of requiring "ms", so other applications, such as the registered "cl", can use the codex32 format. Master seeds and their shares keep "ms". Pass the human-readable part to the checksum functions and cover its BIP-0173 expansion, which already counts toward the checksum length limits. Rename the ms32 functions and constants that now serve every human-readable part to codex32. Results for "ms" are unchanged. Require every share in a set to have the same human-readable part, state that the interpolation helpers do not check the set conditions, and limit fresh-secret generation to applications that accept every payload. Implementations should not correct the human-readable part unless its application specifies how. Checked that the new functions match the previous ms32 functions for "ms" on random data of every length from 0 to 1029, that interpolation is unchanged, and that all 36 valid and 55 invalid vector occurrences decode as before. Link-format, README table, and whitespace checks pass. Refs: bitcoin#2040, bitcoin#2258, bitcoin#2285 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add a Core Lightning "cl" secret from that project's test suite, whose checksum differs from the same data with "ms"; a share set whose 11-character human-readable part requires the long checksum for a 73-character data part; and a secret with an 83-character human-readable part containing "1". Add invalid strings for the 94 and 95 expanded-length gap, a checksum computed over an uppercase human-readable part, and an 84-character human-readable part. Split the existing prefix examples by reason: two of them are valid codex32 strings with the human-readable parts "m" and "s", so they are invalid only as master seeds. Checked every new vector with the specification's Python code and a separate implementation, python-codex32. Both agree except that python-codex32 does not yet limit human-readable parts to 83 characters. Checked that each invalid group fails for its stated reason and that the checks detect altered vectors. Link-format, README table, and whitespace checks pass. Refs: bitcoin#2040 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Record the human-readable part generalization as version 0.3.0. It is a backward-compatible extension, so BIP 3 calls for a minor version bump: strings with the human-readable part "ms" are unchanged. Preamble, README table, link-format, and whitespace checks pass. Refs: bitcoin#2040 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Specify the human-readable part as in BIP-0173 instead of requiring "ms", so other applications, such as the registered "cl", can use the codex32 format. Master seeds and their shares keep "ms". Pass the human-readable part to the checksum functions and cover its BIP-0173 expansion, which already counts toward the checksum length limits. Rename the ms32 functions and constants that now serve every human-readable part to codex32. Results for "ms" are unchanged. Require every share in a set to have the same human-readable part, state that the interpolation helpers do not check the set conditions, and limit fresh-secret generation to applications that accept every payload. Implementations should not correct the human-readable part unless its application specifies how. Checked that the new functions match the previous ms32 functions for "ms" on random data parts of every length from 0 to 1029 symbols (expanded lengths 5 to 1034), covering both checksums, the 94-95 gap, and lengths past 1023. Also checked that interpolation is unchanged and that all 36 valid and 55 invalid vector occurrences decode as before. Link-format, README table, and whitespace checks pass. Refs: bitcoin#2040, bitcoin#2258, bitcoin#2285 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add a Core Lightning "cl" secret from that project's test suite, whose checksum differs from the same data with "ms"; a share set whose 11-character human-readable part requires the long checksum for a 73-character data part; and a secret with an 83-character human-readable part containing "1". Add invalid strings for the 94 and 95 expanded-length gap, a checksum computed over an uppercase human-readable part, and an 84-character human-readable part. Split the existing prefix examples by reason: two of them are valid codex32 strings with the human-readable parts "m" and "s", so they are invalid only as master seeds. Checked every new vector with the specification's Python code and a separate implementation, python-codex32. Both agree except that python-codex32 does not yet limit human-readable parts to 83 characters. Checked that each invalid group fails for its stated reason and that the checks detect altered vectors. Link-format, README table, and whitespace checks pass. Refs: bitcoin#2040 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Record the human-readable part generalization as version 0.3.0. It is a backward-compatible extension, so BIP 3 calls for a minor version bump: strings with the human-readable part "ms" are unchanged. Preamble, README table, link-format, and whitespace checks pass. Refs: bitcoin#2040 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@apoelstra Before I ask for this to be reopened: the version rebased on #2285 (branch, HRP change in 2dcf913) limits the HRP to BIP-0173's 1 thru 83 characters, the loosest limit you said you'd accept (comment). BIP-0173 gets 83 from its 90-character limit, which keeps [low hrp][data] within its error detection guarantees. codex32 now counts the full [high hrp] 0 [low hrp][data] expansion toward the 1023 limit, so the equivalent bound here is 500 (2 * 500 + 1 + 21 = 1022 with an empty payload). The catch is that rust-bech32's HRP is capped at 83. Keep 83, or allow HRP length at most 500? |
|
I'd prefer to keep 83. For many applications "if you don't know the HRP you don't even know what checksum to use" so it would complicate API design if some checksums had different HRP limits than others. Furthermore, with a limit of 83 rust-bech32 is able to somewhat-reasonably represent HRPs without any heap allocations. If we increased that to 500 that would start to become impractical for some systems. |
|
83 is more than enough for all of my conceived applications. I like it has a terse HRP definition option: "as per BIP-0173". |
murchandamus
removed
Pending acceptance
|
The reopen button is greyed out with the explanation that the “branch was recreated or force-pushed”. It looks like I can’t reopen this PR either way, so perhaps you could create another PR for what was left here as well. |
Specify the human-readable part as in BIP-0173 instead of requiring "ms", so other applications, such as the registered "cl", can use the codex32 format. Master seeds and their shares keep "ms". Pass the human-readable part to the checksum functions and cover its BIP-0173 expansion, which already counts toward the checksum length limits. Rename the ms32 functions and constants that now serve every human-readable part to codex32. Results for "ms" are unchanged. Require every share in a set to have the same human-readable part, state that the interpolation helpers do not check the set conditions, and limit fresh-secret generation to applications that accept every payload. Implementations should not correct the human-readable part unless its application specifies how. Checked that the new functions match the previous ms32 functions for "ms" on random data parts of every length from 0 to 1029 symbols (expanded lengths 5 to 1034), covering both checksums, the 94-95 gap, and lengths past 1023. Also checked that interpolation is unchanged and that all 36 valid and 55 invalid vector occurrences decode as before. Link-format, README table, and whitespace checks pass. Refs: bitcoin#2040, bitcoin#2258, bitcoin#2285 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add a Core Lightning "cl" secret from that project's test suite, whose checksum differs from the same data with "ms"; a share set whose 11-character human-readable part requires the long checksum for a 73-character data part; and a secret with an 83-character human-readable part containing "1". Add invalid strings for the 94 and 95 expanded-length gap, a checksum computed over an uppercase human-readable part, and an 84-character human-readable part. Split the existing prefix examples by reason: two of them are valid codex32 strings with the human-readable parts "m" and "s", so they are invalid only as master seeds. Checked every new vector with the specification's Python code and a separate implementation, python-codex32. Both agree except that python-codex32 does not yet limit human-readable parts to 83 characters. Checked that each invalid group fails for its stated reason and that the checks detect altered vectors. Link-format, README table, and whitespace checks pass. Refs: bitcoin#2040 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Record the human-readable part generalization as version 0.3.0. It is a backward-compatible extension, so BIP 3 calls for a minor version bump: strings with the human-readable part "ms" are unchanged. Preamble, README table, link-format, and whitespace checks pass. Refs: bitcoin#2040 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add a Core Lightning "cl" secret from that project's test suite, whose checksum differs from the same data with "ms"; a share set whose 11-character human-readable part requires the long checksum for a 73-character data part; and a secret with an 83-character human-readable part containing "1". Add invalid strings for the 94 and 95 expanded-length gap, a checksum computed over an uppercase human-readable part, and an 84-character human-readable part. Split the existing prefix examples by reason: two of them are valid codex32 strings with the human-readable parts "m" and "s", so they are invalid only as master seeds. Checked every new vector with the specification's Python code and a separate implementation, python-codex32. Both agree except that python-codex32 does not yet limit human-readable parts to 83 characters. Checked that each invalid group fails for its stated reason and that the checks detect altered vectors. Link-format, README table, and whitespace checks pass. Refs: bitcoin#2040
Record the human-readable part generalization as version 0.3.0. It is a backward-compatible extension, so BIP 3 calls for a minor version bump: strings with the human-readable part "ms" are unchanged. Preamble, README table, link-format, and whitespace checks pass. Refs: bitcoin#2040
Specify the human-readable part as in BIP-0173 instead of requiring "ms", so other applications, such as the registered "cl", can use the codex32 format. Master seeds and their shares keep "ms". Pass the human-readable part to the checksum functions and cover its BIP-0173 expansion, which already counts toward the checksum length limits. Rename the ms32 functions and constants that now serve every human-readable part to codex32. Results for "ms" are unchanged. Require every share in a set to have the same human-readable part, state that the interpolation helpers do not check the set conditions, and limit fresh-secret generation to applications that accept every payload, with rejection sampling allowed for those that do not. Implementations should not correct the human-readable part unless its application specifies how. Checked that the new functions match the previous ms32 functions for "ms" on random data parts of every length from 0 to 1029 symbols (expanded lengths 5 to 1034), covering both checksums, the 94-95 gap, and lengths past 1023. Also checked that interpolation is unchanged and that all 36 valid and 55 invalid vector occurrences decode as before. Link-format, README table, and whitespace checks pass. Refs: bitcoin#2040, bitcoin#2258, bitcoin#2285
Add a Core Lightning "cl" secret from that project's test suite, whose checksum differs from the same data with "ms"; a share set whose 11-character human-readable part requires the long checksum for a 73-character data part; and a secret with an 83-character human-readable part containing "1". Add invalid strings for the 94 and 95 expanded-length gap, a checksum computed over an uppercase human-readable part, and an 84-character human-readable part. Split the existing prefix examples by reason: two of them are valid codex32 strings with the human-readable parts "m" and "s", so they are invalid only as master seeds. Checked every new vector with the specification's Python code and a separate implementation, python-codex32. Both agree except that python-codex32 does not yet limit human-readable parts to 83 characters. Checked that each invalid group fails for its stated reason and that the checks detect altered vectors. Link-format, README table, and whitespace checks pass. Refs: bitcoin#2040
Record the human-readable part generalization as version 0.3.0. It is a backward-compatible extension, so BIP 3 calls for a minor version bump: strings with the human-readable part "ms" are unchanged. Preamble, README table, link-format, and whitespace checks pass. Refs: bitcoin#2040
7 participants
Summary of Changes:
Describe codex32 format for arbitrary human-readable parts not just "ms", specify master seed encoding standard, add new test vectors and enhance readability. This makes the document more like BIP-0173: proposing an encoding "codex32", then defining a standard for something using it.
See discussion on #2023 (comment).
Spec:
hrpTest Vectors:
ms32_verify_checksumfunction