How to use a generated Antlr4 lexer

[Korporal]

I'm wondering how one leverages a generated lexer. I might want to write my own recursive descent parser and create my AST as I do so. I could then leverage the Atlr4 Lexer as my token source and still get a huge benefit.

Do people "do this"? has anyone taken this approach?

Also, kind of related, is there any way to have the generated parser "call back" a handler that I supply, but as it parses the input?

This would be an alternative strategy for creating an AST in addition to the visitor and listener options.

[kaby76]

Yes, you can call only the lexer if you want. I do this when printing out tokens in a driver written as a template. https://github.com/kaby76/Domemtech.Trash/blob/main/src/trgen/templates/CSharp/Test.cs#L231-L239

But, if this is to provide a parser-sensitive lexer, it doesn't make sense to use the generated lexer. That's because the API does not pass the state of the parser into LT(). That, in fact, is the "call back" from the parser to the lexer. To change this behavior if you were to still use an Antlr parser, you would need to add code to provide state to the lexer in at least two codes: the generated parser code, and the parser ATN engine--basically anywhere LT() is called. This would not be easy depending on how much "state" of the call stack you need to provide to the lexer.

[Korporal]

Yes, you can call only the lexer if you want. I do this when printing out tokens in a driver written as a template. https://github.com/kaby76/Domemtech.Trash/blob/main/src/trgen/templates/CSharp/Test.cs#L231-L239

But, if this is to provide a parser-sensitive lexer, it doesn't make sense to use the generated lexer. That's because the API does not pass the state of the parser into LT(). That, in fact, is the "call back" from the parser to the lexer. To change this behavior if you were to still use an Antlr parser, you would need to add code to provide state to the lexer in at least two codes: the generated parser code, and the parser ATN engine--basically anywhere LT() is called. This would not be easy depending on how much "state" of the call stack you need to provide to the lexer.

These are interesting points. Is there a quick way to look at the generated code to see if the parser ever actually does "call" the lexer?

I'm considering writing my own parser that leverages the lexer, but this is just an idea at this stage.

[kaby76]

Sure. Just do a grep -R -e for "LT(" in the source.

$ grep . -R -e 'LT[(]'  | grep -v [.]xml
Binary file ./.vs/Antlr4/FileContentIndex/95059e4b-6cfb-4552-8a2c-cf448b6419de.vsidx matches
./Atn/ParserATNSimulator.cs:                                                              " line " + input.LT(1).Line + ":" + input.LT(1).Column);
./Atn/ParserATNSimulator.cs:                                                               " line " + input.LT(1).Line + ":" + input.LT(1).Column);
./Atn/ParserATNSimulator.cs:                      //                                                       " line "+input.LT(1)Line+":"+input.LT(1).Column);
./Atn/ParserATNSimulator.cs:                                                                                            input.LT(1),
./BufferedTokenStream.cs:        /// <see cref="LT(int)">LT(1)</see>
./BufferedTokenStream.cs:            return LT(i).Type;
./BufferedTokenStream.cs:        public virtual IToken LT(int k)
./CommonTokenStream.cs:    /// <see cref="LT(int)"/>
./CommonTokenStream.cs:        public override IToken LT(int k)
./CommonTokenStream.cs:            //System.out.println("enter LT("+k+")");
./DefaultErrorStrategy.cs:                // state in ATN; must be a case where LT(1) is in the recovery
./DefaultErrorStrategy.cs:        /// <c>LT(1)</c>
./DefaultErrorStrategy.cs:            IToken lookback = ((ITokenStream)recognizer.InputStream).LT(-1);
./ITokenStream.cs:        /// <c>LT(k).getType()==LA(k)</c>
./ITokenStream.cs:        IToken LT(int k);
./NoViableAltException.cs:        /// <summary>Which configurations did we try at input.index() that couldn't match input.LT(1)?</summary>
./Parser.cs:                _output.WriteLine("enter   " + this._enclosing.RuleNames[ctx.RuleIndex] + ", LT(1)=" + this._enclosing._input.LT(1).Text);
./Parser.cs:                _output.WriteLine("exit    " + this._enclosing.RuleNames[ctx.RuleIndex] + ", LT(1)=" + this._enclosing._input.LT(1).Text);
./Parser.cs:                return _input.LT(1);
./Parser.cs:            _ctx.Start = _input.LT(1);
./Parser.cs:            _ctx.Start = _input.LT(1);
./Parser.cs:            _ctx.Stop = _input.LT(-1);
./Parser.cs:            _ctx.Start = _input.LT(1);
./Parser.cs:            previous.Stop = _input.LT(-1);
./Parser.cs:            _ctx.Stop = _input.LT(-1);
./UnbufferedTokenStream.cs:        /// <c>LT(1)</c>
./UnbufferedTokenStream.cs:        /// <c>LT(-1)</c>
./UnbufferedTokenStream.cs:        /// <c>LT(-1)</c>
./UnbufferedTokenStream.cs:        /// <c>LT(1)</c>
./UnbufferedTokenStream.cs:        public virtual IToken LT(int i)
./UnbufferedTokenStream.cs:                throw new ArgumentOutOfRangeException("LT(" + i + ") gives negative index");
./UnbufferedTokenStream.cs:            return LT(i).Type;
./UnbufferedTokenStream.cs:            // track last token for LT(-1)

LT() is actually for the token stream, but that calls Lexer.NextToken().