in icu_utf8_char(): codepoint value 0x110000 is accepted, even if it generates invalid UTF-8

[verdy-p]

Inside: static int icu_utf8_char(lua_State *L)
at:

icu4lua/src/icu.utf8.c

Line 368 in 7b37d34

else if (codePoint > 0x110000) {

the test is incorrect:

       else if (codePoint > 0x110000) {
            return luaL_argerror(L,i+1,"invalid codepoint");
        }
        else {
            // 00000000 000zzzzz yyyyyyyy xxxxxxxx
            // 11110zzz 10zzyyyy 10yyyyxx 10xxxxxx
            ...

It seems you have reversed the original test that was

       else if (codePoint < 0x110000) {
            // 00000000 000zzzzz yyyyyyyy xxxxxxxx
            // 11110zzz 10zzyyyy 10yyyyxx 10xxxxxx
            ...
        }
        else {
            return luaL_argerror(L,i+1,"invalid codepoint");
        }

But you forgot the equal sign! So the code point 0x110000 is accepted as if it was valid and will generate a non-standard UTF-8 sequence (0xF8,0x80,0x80,0x80): the bytes 0xC0, 0xC1, and 0xF8 to 0xFF are NEVER part of any valid UTF-8 text.

[mogando668]

@ verdy-p : "0xF8 to 0xFF"

I don't know whether you've lost internet access for 10+ years or what, but it's been quite some time since Unicode Consortium has also invalidated 0x F5 F6 F7 from UTF-8. Ancient documents that describe UTF-8 up to 6-bytes are extremely outdated.

[mogando668]

And you're also waaaaay off on what U+ 110 000 hypothetically would look like coming out of a non-conformant encoder function -

It should be \364 \220 \200 \200 not \370 \200 \200 \200, since the current permissible code point max of U+10FFFF is nothing close to saturating the 4-byte space. And it would make absolutely no sense that U+110000 would be \370\200\200\200 when U+10FFFF was only \364\217\277\277.

[verdy-p]

@mogando668
I have absolutely not lost any Internet access.

Please reread carefully what I signaled precisely and accurately: U+110000 is NOT a valid codepoint, but your code validates it because of a forgotten equal sign in the condition.

This report is a true bug in the C code of Lua which generates incorrect UTF-8.

The alternate sequence your propose instead is also completely incorrect. instead it should raise the error return luaL_argerror(L,i+1,"invalid codepoint");

This means that this lua_c function is really a non-conformant encoder function (and has always been non-conformant : the UTF-8 conformance was defined in the standard long before this C function was written here; and it's the developer that wrote it that forgot to test it).

[mogando668]

it's not "my code". This isn't my project. Which part of the spec makes you think the next character after \364\217\277\277 should be \370\200\200\200 ???????

[mogando668]

even going by this ancient document

https://datatracker.ietf.org/doc/html/rfc2279

it says 4-byte range goes up to U+1F FFFF. Today it's only at U+10 FFFF. So even the 4-byte range is nothing close to saturated.

[mogando668]

0 -> 1,114,111 (U+ 0000 to U+ 10 FFFF),

compared to hypothetical max of U+1F FFFF for sequences up to 4 bytes,

is only 53.125 % of the space. But somehow you think the correct way is to waste that nearly 47% and go straight to F8 \370. And your proposed sequence for U+11 0000 at \370 is still wrong, since that would be a 5-byte sequence already, not 4-bytes like what you said.

[verdy-p]

it's not "my code". This isn't my project. Which part of the spec makes you think the next character after \364\217\277\277 should be \370\200\200\200 ???????

I say exactly the opposite. Once gain you don't read carefully. NO sequence at all should be generated for code point value 0x110000 (only you are affirming that in your statement: "It should be \364 \220 \200 \200 not \370 \200 \200 \200", YOU are wrong!), but an error must be returned like with any value larger than it (so yes, there's no valid UTF-8 sequences with more than 4 bytes, but this is not what I reported in this bug, I have never claimed that!)

@mogando668, don't assume things with superficial reading if you don't understand it. I am very clear, but you are confused. All your comments above are just wrong, you are adding more confusion by adding false statements about what I wrote or about how the code should behave.

Only one solution is possible for the invalid code point value 0x110000, it must raise an error and no byte sequence at all (not the 4-byte sequence that the existing function incorrectly returns, which cannot even be decoded with standard UTF-8 decoder or even with any decoder for the older RFC/ISO10646 version that has been obsoleted but that encoded it differently with 5 bytes, and none of the two different 4-byte sequences that you've incorrectly included in your confusing comments).