Custom location tracking in jison-gho

Question

I need to parse a "token-level" language, i.e. the input is already tokenized with a semicolon as a delimiter. Sample input: A;B;A;D0;ASSIGN;X;. Here's also my grammar file.

I'd like to track location columns per-token. For the previous example, here's how I'd like to have columns defined:

Input: A;B;A;D0;ASSIGN;X;\n
Col:   1122334445555555666

So basically I'd like to increment column every time a semicolon is hit. I made a function that increments column count when semicolon is hit and for all actions I just set column in yylloc to my custom column count. However, with this approach I have to copypaste a function call to every action. Do you please know if there's any other cleaner way? Also there'll be no lexical errors in the input since it's autogenerated.

Edit: Nevermind, my solution actually doesn't work. So I'll be happy for any suggestions :)

%lex

%{

var delimit = (terminal) => { this.begin('delimit'); return terminal }

var columnInc = () => {
  if (yy.lastLine === undefined) yy.lastLine = -1
    if (yylloc.first_line !== yy.lastLine) {
      yy.lastLine = yylloc.first_line
      yy.columnCount = 0
  }
  yy.columnCount++
}

var setColumn = () => {
  yylloc.first_column = yylloc.last_column = yy.columnCount
}

%}

%x delimit

%%

"ASSIGN"                   { return delimit('ASSIGN'); setColumn() }
"A"                        { return delimit('A'); setColumn() }
<delimit>{DELIMITER}       { columnInc(); this.popState(); setColumn() }
\n                         { setColumn() }
...

Answer 1

There are a few ways to accomplish this in jison-gho. As you're looking to implement a token counter which is tracked by the parser, this invariably means we need to find a way to 'hook' into the code path where the lexer passes tokens to the parser.

Before we go look at a few implementations, a few thoughts that may help others who are facing similar, yet slightly different problems:

1. completely custom lexer for prepared token streams: as your input is a set of tokens already, one might consider using a custom lexer which would then just take the input stream as-is and do as little as possible while passing the tokens to the parser. This is doable in jison-gho and a fairly minimal example of such is demonstrated here:

   https://github.com/GerHobbelt/jison/blob/0.6.1-215/examples/documentation--custom-lexer-ULcase.jison

   while another way to integrate that same custom lexer is demonstrated here:

   https://github.com/GerHobbelt/jison/blob/0.6.1-215/examples/documentation--custom-lexer-ULcase-alt.jison

   or you might want to include the code from an external file via a %include "documentation--custom-lexer-ULcase.js" statement. Anyway, I digress.

   Given your problem, depending on where that token stream comes from (who turns that into text? Is that outside your control as there's a huge overhead cost there as you're generating, then parsing a very long stream of text, while a custom lexer and some direct binary communications might reduce network or other costs there.

   The bottom line is: if the token generator and everything up to this parse point is inside your control, I personally would favor a custom lexer and no text conversion what-so-ever for the intermediary channel. But in the end, that depends largely on your system requirements, context, etc. and is way outside the realm of this SO coding question.

2. augmenting the jison lexer: of course another approach could be to augment all (or a limited set of) lexer rules' action code, where you modify yytext to pass this data to the parser. This is the classic approach from the days of yacc/bison. Indeed, yytext doesn't have to be a string, but can be anything, e.g.

[a-z]   %{
    yytext = new DataInstance(
      yytext, // the token string 
      yylloc, // the token location info
      ...     // whatever you want/need...
   );
   return 'ID';  // the lexer token ID for this token
%}

For this problem, this is a lot of code duplication and thus a maintenance horror.

3. hooking into the flow between parser and lexer: this is new and facilitated by the jison-gho tool by pre_lex and post_lex callbacks. (The same mechanism is available around the parse() call so that you can initialize and postprocess a parser run in any way you want: pre_parse and post_parse are for that.

   Here, since we want to count tokens, the simplest approach would be using the post_lex hook, which is only invoked when the lexer has completely parsed yet another token and passes it to the parser. In other words: post_lex is executed at the very end of the lex() call in the parser.

   The documentation for these is included at the top of every generated parser/lexer JS source file, but then, of course, you need to know about that little nugget! ;-)

   Here it is:

   parser.lexer.options:

   • pre_lex: function()

     optional: is invoked before the lexer is invoked to produce another token.

     this refers to the Lexer object.

   • post_lex: function(token) { return token; }

     optional: is invoked when the lexer has produced a token token; this function can override the returned token value by returning another. When it does not return any (truthy) value, the lexer will return the original token.

     this refers to the Lexer object.

Do note that options 1 and 3 are not available in vanilla jison, with one remark about option 1: jison does not accept a custom lexer as part of the jison parser/lexer spec file as demonstrated in the example links above. Of course, you can always go around and wrap the generated parser and thus inject a custom lexer and do other things.

Implementing the token counter using post_lex

Now how does it look in actual practice?

Solution 1: Let's do it nicely

We are going to 'abuse'/use (depending on your POV about riding on undocumented features) the yylloc info object and augment it with a counter member. We choose to do this so that we never risk interfering (or getting interference from) the default text/line-oriented yylloc position tracking system in the lexer and parser.

The undocumented bit here is the knowledge that all data members of any yylloc instance will be propagated by the default jison-gho location tracking&merging logic in the parser, hence when you tweak an yylloc instance in the lexer or parser action code, and if that yylloc instance is propagated to the output via merge or copy as the parser walks up the grammar tree, then your tweaks will be visible in the output.

Hooking into the lexer token output means we'll have to augment the lexer first, which we can easily do in the %% section before the /lex end-of-lexer-spec-marker:

// lexer extra code

var token_counter = 0;

lexer.post_lex = function (token) {
    // hello world
    ++token_counter;
    this.yylloc.counter = token_counter;
    return token;
};

// extra helper so multiple parse() calls will restart counting tokens:
lexer.reset_token_counter = function () {
    token_counter = 0;
};

where the magic bit is this statement: this.yylloc.counter = token_counter.

We hook a pre_lex callback into the flow by directly injecting it into the lexer definition via lexer.post_lex = function (){...}.

We could also have done this via the lexer options: lexer.options.post_lex = function ... or via the parser-global yy instance: parser.yy.post_lex = function ... though those approaches would have meant we'ld be doing this in the parser definition code chunk or from the runtime which invokes the parser. These two slightly different approaches will not be demonstrated here.

Now all we have to do is complete this with a tiny bit of pre_parse code to ensure multiple parser.parse(input) invocations each will restart with the token counter reset to zero:

// extra helper: reset the token counter at the start of every parse() call:
parser.pre_parse = function (yy) {
    yy.lexer.reset_token_counter();
};

Of course, that bit has to be added to the parser's final code block, after the %% in the grammar spec part of the jison file.

Full jison source file is available as a gist here.

How to compile and test:

# compile
jison --main so-q-58891186-2.jison
# run test code in main()
node so-q-58891186-2.js

Notes: I have 'faked' the AST construction code in your original source file so that one can easily diff the initial file with the one provided here. All that hack-it-to-make-it-work stuff is at the bottom part of the file.

Solution 2: Be a little nasty and re-use the yylloc.column location info and tracking

Instead of using the line info part of yylloc, I chose to use the column part instead, as to me that's about the same granularity level as a token sequence index. Doesn't matter which one you use, line or column, as long as you follow the same pattern.

When we do this right, we get the location tracking features of jison-gho added in for free, which is: column and line ranges for a grammar rule are automatically calculated from the individual token yylloc info in such a way that the first/last members of yylloc will show the first and last column, pardon, token index of the token sequence which is matched by the given grammar rule. This is the classic,merge jison-gho behaviour as mentioned in the --default-action CLI option:

--default-action

Specify the kind of default action that jison should include for every parser rule.

You can specify a mode for value handling ($$) and one for location tracking (@$), separated by a comma, e.g.:

       --default-action=ast,none

Supported value modes:

• classic : generate a parser which includes the default

                 $$ = $1;
  

  action for every rule.

• ast : generate a parser which produces a simple AST-like tree-of-arrays structure: every rule produces an array of its production terms' values. Otherwise it is identical to classic mode.

• none : JISON will produce a slightly faster parser but then you are solely responsible for propagating rule action $$ results.

  The default rule value is still deterministic though as it is set to undefined: $$ = undefined;

• skip : same as none mode, except JISON does NOT INJECT a default value action ANYWHERE, hence rule results are not deterministic when you do not properly manage the $$ value yourself!

Supported location modes:

• merge : generate a parser which includes the default @$ = merged(@1..@n); location tracking action for every rule, i.e. the rule's production 'location' is the range spanning its terms.
• classic : same as merge mode.
• ast : ditto.
• none : JISON will produce a slightly faster parser but then you are solely responsible for propagating rule action @$ location results. The default rule location is still deterministic though, as it is set to undefined: @$ = undefined;
• skip : same as "none" mode, except JISON does NOT INJECT a default location action ANYWHERE, hence rule location results are not deterministic when you do not properly manage the @$ value yourself!

Notes:

• when you do specify a value default mode, but DO NOT specify a location value mode, the latter is assumed to be the same as the former.

  Hence:

       --default-action=ast
  

  equals:

       --default-action=ast,ast
  

• when you do not specify an explicit default mode or only a "true"/"1" value, the default is assumed: classic,merge.

• when you specify "false"/"0" as an explicit default mode, none,none is assumed. This produces the fastest deterministic parser.

Default setting: [classic,merge]

Now that we are going to 're-use' the fist_column and last_column members of yylloc instead of adding a new counter member, the magic bits that do the work remain nearly the same as in Solution 1:

augmenting the lexer in its %% section:

// lexer extra code

var token_counter = 0;

lexer.post_lex = function (token) {
    ++token_counter;

    this.yylloc.first_column = token_counter;
    this.yylloc.last_column = token_counter;

    return token;
};

// extra helper so multiple parse() calls will restart counting tokens:
lexer.reset_token_counter = function () {
    token_counter = 0;
};

Side Note: we 'abuse' the column part for tracking the token number; meanwhile the range member will still be usable to debug the raw text input as that one will track the positions within the raw input string.

Make sure to tweak both first_column and last_column so that the default location tracking 'merging' code in the generated parser can still do its job: that way we'll get to see which range of tokens constitute a particular grammar rule/element, just like it were text columns.

Could've done the same with first_line/last_line, but I felt it more suitable to use the column part for this as it's at the same very low granularity level as 'token index'...

We hook a pre_lex callback into the flow by directly injecting it into the lexer definition via lexer.post_lex = function (){...}.

Same as Solution 1, now all we have to do is complete this with a tiny bit of pre_parse code to ensure multiple parser.parse(input) invocations each will restart with the token counter reset to zero:

// extra helper: reset the token counter at the start of every parse() call:
parser.pre_parse = function (yy) {
    yy.lexer.reset_token_counter();
};

Of course, that bit has to be added to the parser's final code block, after the %% in the grammar spec part of the jison file.

Full jison source file is available as a gist here.

How to compile and test:

# compile
jison --main so-q-58891186-3.jison
# run test code in main()
node so-q-58891186-3.js

Aftermath / Observations about the solutions provided

Observe the test verification data at the end of both those jison files provided for how the token index shows up in the parser output:

Solution 1 (stripped, partial) output:

  "type": "ProgramStmt",
  "a1": [
    {
      "type": "ExprStmt",
      "a1": {
        "type": "AssignmentValueExpr",
        "target": {
          "type": "VariableRefExpr",
          "a1": "ABA0",
          "loc": {
            "range": [
              0,
              8
            ],
            "counter": 1
          }
        },
        "source": {
          "type": "VariableRefExpr",
          "a1": "X",
          "loc": {
            "counter": 6
          }
        },
        "loc": {
          "counter": 1
        }
      },
      "loc": {
        "counter": 1
      }
    }
  ],
  "loc": {
    "counter": 1
  }

Note here that the counter index is not really accurate for compound elements, i.e. elements which were constructed from multiple tokens matching one or more grammar rules: only the first token index is kept.

Solution 2 fares much better in that regard:

Solution 2 (stripped, partial) output:

      "type": "ExprStmt",
      "a1": {
        "type": "AssignmentValueExpr",
        "target": {
          "type": "VariableRefExpr",
          "a1": "ABA0",
          "loc": {
            "first_column": 1,
            "last_column": 4,
          }
        },
        "source": {
          "type": "VariableRefExpr",
          "a1": "X",
          "loc": {
            "first_column": 6,
            "last_column": 6,
          }
        },
        "loc": {
          "first_column": 1,
          "last_column": 6,
        }
      },
      "loc": {
        "first_column": 1,
        "last_column": 7,
      }
    }

As you can see the first_column plus last_column members nicely track the set of tokens which constitute each part. (Note that the counter increment code implied we start counting with ONE(1), not ZERO(0)!)

Parting thought

Given the input A;B;A;D0;ASSIGN;X;SEMICOLON; the current grammar parses this like ABA0 = X; and I wonder if this is what you really intend to get: constructing the identifier ABA0 like that seems a little odd to me.

Alas, that's not relevant to your question. It's just me encountering something quite out of the ordinary here, that's all. No matter.

Cheers and hope this long blurb is helpful to more of us. :-)

Source files:

• original OP file as gist
• solution 1 JISON file
• solution 2 JISON file
• current jison-gho release example grammars, including several which demo advanced features