[Lexer] Add Unicode identifier and whitespace recognition - #23

Open

kuilpd wants to merge 9 commits into

DIL-add-utf8-identifiers

Commits on Feb 24, 2025

Add basic support for UTF-8 identifiers
kuilpd
committed
56aa0de
Replace IsWord with identifier UTF recognition and add ASCII characters to it
kuilpd
committed
97fa509

Commits on Feb 25, 2025

Refactor code & add unicode whitespace skipping
kuilpd
committed
526df72

Commits on Feb 27, 2025

Add test for an identifier with a right-to-left script
kuilpd
committed
ccab3d1

Commits on Mar 3, 2025

Count UTF-8 width of skipped whitespaces
kuilpd
committed
4e79115
Allow identifier to be contain anything except operators and whitespaces
kuilpd
committed
6520218

Commits on Mar 7, 2025

Manually count identifier and whitespace width using charWidth
kuilpd
committed
20ee555

Commits on Mar 10, 2025

Remove Unicode whitespaces recognition
kuilpd
committed
ce05c75

Commits on Mar 11, 2025

Replace unicode codepoint conversion with UTF8 validity check
kuilpd
committed
daf1735