Grammar does not scope function names, user-defined types, or stdlib calls

[magnus-madsen]

The grammar scopes keywords, comments, strings, numbers and the built-in primitive types, but leaves identifiers unscoped — so in a typical Flix snippet most of the names carry no colour.

This shows up on both sites that consume this grammar: flix.dev via Shiki, and blog.flix.dev, which just moved to it in flix/blog.flix.dev#18.

What renders today

Given

def main(): Int32 \ IO =
    println("Hello World!");
    sum(123, 456)

def sum(x: Int32, y: Int32): Int32 =
    let result = x + y;
    result

the tokens come out as:

token	scope
def, let, ;	keyword.*
Int32	entity.name.type
"Hello World!"	string.quoted.double
123, 456	constant.numeric.decimal
main, sum, println, result, x, y	(none — falls through to source.flix)

Three gaps

1. Function names are never scoped. repository.keywords.patterns[21] matches def as keyword.declaration.flix, but nothing scopes the name after it. The same applies to the name after enum, trait, struct and eff. entity.name.function is not used anywhere in the grammar.

2. entity.name.type is a fixed list of primitives (repository.types.patterns[0]: Unit|Bool|Char|Float32|Float64|Int8|Int16|Int32|Int64|BigInt|String), so user-defined types are plain text. In def foo(e: Exp): Exp = match e { ... }, both occurrences of Exp are unscoped while Int32 in the neighbouring snippet is coloured — the inconsistency is visible in a single post. A rule for capitalised identifiers in type position would cover this the way most ML-family grammars do.

3. No support.function for the standard library. println, List.map, Map.insert and friends read as ordinary identifiers. This one is a judgement call — hardcoding stdlib names is a maintenance burden, and it may be fair to leave it out deliberately — but noting it since it is the most visible difference from the highlight.js grammar blog.flix.dev used previously, which classified them as built_in.

For scale: measured over the 8197 characters of Flix code on blog.flix.dev, the old highlight.js grammar classified 464 characters as function/type names and 335 as stdlib built-ins. All of that is now uncoloured.

Unrelated, noticed while reading

Inside string interpolation the interpolated expression keeps string.quoted.double. punctuation.definition.template-expression.begin/end mark the ${ and }, but the body is not switched back to code, so ${x} renders as string text rather than as an expression.

Happy to open a PR for 1 and 2 if that direction seems right.