ANTLR4 - VisitChildren returns null, even when child returns some object

Question

I've been trying to implement the Visitor Pattern to parse some specific SQL Statements into an internal object structure consisting of TableDefinition and ColumnDefinition objects.

This is a small (stripped down) portion from the grammar:

column_definition
 : column_name datatype? column_constraint*
 ;
 
column_constraint
 : ( K_CONSTRAINT name )?
   ( K_PRIMARY K_KEY ( K_CLUSTERED | K_NONCLUSTERED )? ( K_ASC | K_DESC )? K_AUTOINCREMENT?
   | K_AUTOINCREMENT
   | K_NOT? K_NULL
   )
 ;

datatype
 : K_CHAR ( '(' unsigned_integer ')' )?                                     #char
 | K_DATE                                                                   #date
 ;

And here is one of the derived BaseVisitors which is meant to return ColumnDefinitions:

namespace SqlParser.Visitor
{
    public class DataTypeVisitor: SqlAnywhereParserBaseVisitor<ColumnDefinition>
    {
        public override ColumnDefinition VisitColumn_definition([NotNull] SqlAnywhereParser.Column_definitionContext context)
        {
            var res = VisitChildren(context);
            var constraint = (SqlAnywhereParser.Column_constraintContext[])context.column_constraint();

            if (res != null) // Add NULL attributes
            {
                if (constraint.Any(c => c.K_NULL() != null && c.K_NOT() == null))
                    res.IsNullable = true;

                if (constraint.Any(c => c.K_NULL() != null && c.K_NOT() != null))
                    res.IsNullable = false;
            }

            return res;
        }

        public override ColumnDefinition VisitChar([NotNull] SqlAnywhereParser.CharContext context)
        {
            return new ColumnDefinition()
            {
                DataType = DbType.StringFixedLength,
                Length = int.Parse(context.unsigned_integer()?.GetText() ?? "1") 
            };
        }
   }
}

When I debug the process, I can observe how the call to VisitChildren goes into VisitChar which returns a ColumnDefinition object. When VisitChar completes and the cursor jumps back to continue in VisitColumn_definition the variable res is null.

Did I miss something crucial or did I misunderstand the visitor pattern? Before I tried VisitChildren I used to use the base.VisitColumn_definition(context) call, which basically only calls VisitChildren.

Does anyone have a hint, which mistakes I made? Why doesn't my ColumnDefinition result created at the VisitChar leaf bubble up?

Below is my testinput:

CREATE TABLE "DBA"."pbcattbl" (
    "pbt_tnam"                       char(129) NOT NULL
   ,"pbt_tid"                        char(5) NULL
);

Answer 1

I found a solution:

protected override List<ColumnDefinition> AggregateResult(List<ColumnDefinition> aggregate, List<ColumnDefinition> nextResult)
{
    if (aggregate != null && nextResult != null) aggregate.AddRange(nextResult);
    return aggregate ?? nextResult;
}

I converted the result to List<ColumnDefinition> and added an appropriate override to AggregateResult.

Thank you @kaby76 for pointing me into the right direction with your comment. Also thanks to all others for the feedback and quick responses!

Answer 2

You'll have to override all Visit...(... context) calls (at least all the ones that your parse tree has in it). Let's say you have this grammar:

grammar T;

parse
 : expr EOF
 ;

expr
 : expr ( '*' | '/' ) expr #multExpr
 | expr ( '+' | '-' ) expr #addExpr
 | NUMBER                  #numberExpr
 ;
 
NUMBER
 : [0-9]+ ( '.' [0-9]+ )?
 ;

SPACES
 : [ \t\r\n]+ -> skip
 ;

And you're parsing the expression "42". Then it's not sufficient to override just the method VisitNumberExpr(TParser.NumberExprContext context):

using System;
using Antlr4.Runtime;

namespace AntlrTest
{
    class Program
    {
        static void Main(string[] args)
        {
            var lexer = new TLexer(CharStreams.fromstring("42"));
            var parser = new TParser(new CommonTokenStream(lexer));
            var root = parser.parse();
            var evaluated = new CustomVisitor().Visit(root);
            
            Console.WriteLine($"evaluated: {evaluated}");
        }
    }

    class CustomVisitor : TBaseVisitor<decimal>
    {
        public override decimal VisitNumberExpr(TParser.NumberExprContext context)
        {
            return decimal.Parse(context.GetText());
        }
    }
}

It will return the default 0. In this case, you should also override VisitParse(TParser.ParseContext context):

class CustomVisitor : TBaseVisitor<decimal>
{
    public override decimal VisitParse(TParser.ParseContext context)
    {
        return Visit(context.expr());
    }
    
    public override decimal VisitNumberExpr(TParser.NumberExprContext context)
    {
        return decimal.Parse(context.GetText());
    }
}

which now returns 42.

If you don't want to override/implement too many rules, you could use a listener instead:

using System;
using Antlr4.Runtime;
using Antlr4.Runtime.Tree;

namespace AntlrTest
{
    class Program
    {
        static void Main(string[] args)
        {
            var lexer = new TLexer(CharStreams.fromstring("42"));
            var parser = new TParser(new CommonTokenStream(lexer));
            var root = parser.parse();
            var listener = new CustomListener();
            
            ParseTreeWalker.Default.Walk(listener, root);
            
            Console.WriteLine($"Result: {listener.Result}");
        }
    }

    class CustomListener : TBaseListener
    {
        public decimal Result { get; private set; }

        public override void EnterNumberExpr(TParser.NumberExprContext context)
        {
            Result = decimal.Parse(context.GetText());
        }
    }
}