Peek(1) affects next token.

[tiagolr]

I'm using different rulesets in a custom parser for erazor lib.

Frequently i use peek(1) on a ruleset to verify if it should process what comes next or leave it to another ruleset.

The problem is that if a function using ruleset(A) decides not to process what comes next after peek(1), and another function with ruleset(B) processes what comes next, the first token will be the one defined by function (A) peek(1)

example:

function A() {
  stream.ruleset = MyLexer.ASomeRules;

  var tok = peek(1);
  switch(tok) {
      case SOME_RULE: B(); // let B process what comes next.
  }
}

function B() {
    stream.ruleset = MyLexer.BOtherRules;  

    // will throw SOME_RULE has no match in this switch.
    switch stream { 
          case OTHER_RULE: null;
    } 
}

[tiagolr]

It would be great if peek did not change state at all, other times i also have problems with using consequent peek(1), using peek(1), peek(1) is the same as using peek(2).

Maybe a reject() call to revert the effects from peek() or if peek did not affect the state would probably be perfect, just return the token at a certain position, think its possible?

[Simn]

Yes, this has been a problem ever since rulesets were added. Originally the problem was that the lexer was based on a data representation which did not support rewinding. I think we can fix this now by resetting the position upon changing rulesets, as well as clearing the token cache in the parser.

This might require some architectural thought though. We have to be careful not to make a mess of the Lexer/Parser/LexerTokenSource inter-dependencies.

[tiagolr]

Would be great, thanks, meanwhile ill see if i can workaround it for this parser.

Btw if only peek would not change the state and fetch the token, things would work great imo (even if at some performance cost), I don't know if thats feasible or what is involved compared to reseting the position when changing rulesets, sounds more robust, peek(1) peek(1) would even return the same token. Just a suggestion.

[Simn]

This would mean disabling the token cache completely, which I don't think is viable performance-wise. The lexer shouldn't have to tokenize the same thing twice if not necessary.

[tiagolr]

Well peek(0) is used internally to retrieve the current token sometimes, but if peek() was only called by the user and would use a different method other than Lexer.token() to retrieve the token, it might be possible to skip the cache and let the user be responsible for performance lost if using peek() too much.

I'm not sure how the state transitions are generated or how the cache works, if its state = state.trans.get(i) in Lexer.token() that alters the state permanently, it could work to skip that for peek(). But I'm just guessing.

[tiagolr]

Ok, what i did to avoid unwanted tokens parsed by peek(1) was to involve the case in a try catchand have no rule/match for the specific case i want other function/rule to process.

For now it works, guess i was lucky i did not rely on the peek(1) too much and in most cases the resulting token could be junked (junk()) after reading with peek(1), anyway it would be nice to have rewind or reverting cache after ruleset changes.

Thanks for this lib, did some benchmarks and the performance is about 2x faster than the previous parser using string comparison, not to mention its much easier to modify and read, with some luck will be ready in a couple of days.

[Simn]

I'll see what I can do about the caching issue soon. Out of curiosity, what are you trying to parse?

[tiagolr]

https://github.com/ciscoheat/erazor/blob/master/src/demo/Main.hx

You can see the templating language there, it uses string comparison to process the templates, trying to extend the templating language i ended rewriting the parser to make it easier to extend and add some nice features, now tweaking it to pass all the tests.

The tests are probably more readable:
https://github.com/ciscoheat/erazor/blob/master/test/erazor/TestParser.hx

[fussybeaver]

I don't know if it's relevant to the discussion, but for stipple I'm using a forward matching regular expression, then rewind the lexer position.

[Simn]

The problem is this:

• You fetch a token with ruleset A.
• The parser caches it.
• You don't consume it.
• Instead you try to fetch a token with ruleset B.
• The parser still has the previous token cached.

This shouldn't be hard to solve because it's just a matter of resetting the state upon ruleset changes. The problem is finding a good architecture between lexer, parser and token source.

[tiagolr]

This stipple looks awesome, I'll have to spend some time looking at it, funny i was inspired by handlebars and i was trying to implement some of its features into erazor (thats how i got here).

With some luck it may fit perfectly what i was trying to do, which is to use a good templating engine with ufront. Thanks a lot!

[tiagolr]

Hey @fussybeaver , stipple looks really great, when will it be released as an haxelib? Guess it will have to wait until a new hxparse and haxe versions are released?

Anyway this makes my work unnecessary, thanks a lot, looking at the tests it seems to perfect, i can go on making my site, and perhaps using this beta version of stipple.

I'll let you know manage to add it to ufront.

[fussybeaver]

As you say, I am waiting for the new haxe version to be released, and I hope @Simn will release a haxelib version of hxparse too.

Regarding hxparse I think I would like to see improvements in a non-macro based API, as I am using macros to improve runtime performance, and without build macros this doesn't seem to be quite as idiomatic for using hxparse.

[Simn]

I'm not sure what that would look like unless you want to call peek and junk manually all the time, which I think you already can anyway. One thing we could do is add a flag which writes the parser output to a .hx file so it can be used as a preprocessor.

[fussybeaver]

You can call peek and junk manually, it's works well though it feels like the API could be made nicer. Would a preprocessor require compiling twice ? Maybe simply having non-build macros would also work.