I am use the transform function of the Python Lark parser and am trying to recover line and column numbers. The native parse tree has Tree objects, which have the line and column numbers. But, when I go to do a transform from the initial parse tree to my AST, all matched symbols lose that information. For instance, here is a snippet to parse simple integer expressions:
binary_int_expr: _int_expr binary_op _int_expr
binary_op: "+" -> binary_add
| "-" -> binary_sub
| "*" -> binary_mult
| "%" -> binary_mod
when I process this with:
def binary_add(self, value): ....
value is just []. I could use !binary_add to get [+],but the Tree object is gone. Instead I do this:
binary_int_expr: _int_expr binary_op _int_expr
binary_op: binary_add_loc -> binary_add
| binary_sub_loc -> binary_sub
| binary_mult_loc -> binary_mult
| binary_mod_loc -> binary_mod
binary_add_loc: "+"
binary_sub_loc: "-"
binary_mult_loc: "*"
binary_mod_loc: "%"
Here are the transform rules:
def Loc..... # pull out the line and column info
def binary_int_expr(self, value):
(loc,op) = value[1]
return BinaryOp(loc, op, value[0], value[2])
def binary_add(self, value): return (Loc(value[0]), BinOps.Add)
def binary_sub(self, value): return (Loc(value[0]), BinOps.Sub)
def binary_mult(self, value): return (Loc(value[0]), BinOps.Mult)
def binary_mod(self, value): return (Loc(value[0]), BinOps.Mod)
Note there is no hook/rule for binary_add_loc. Now, when binary_add is called, I see value=[Tree...] as returned from binary_add_loc without any hook. I can extract line and column from the Tree node. But, this means a good bit of my grammar is these pointless rules just to capture the line and column number. Is there a saner way?
I tried various grammar rule versions:
_binary_add_loc (inlining)
binary_op: "+" -> binary_add
!binary_op: "+" -> binary_add
binary_op: !"+" -> binary_add
All of these either fail outright of elide the Tree node.
Ideal would be some way to return literals as Tree nodes, e.g.
binary_op: "+" -> binary_add
would call binary_add with Tree.... Maybe it's there and I just cannot find it.