Adding a literal token to a grammar file when it had already been defined in the lexer file - is this legal?

[The-Futurist]

If I have a grammar g4 file and a lexer g4 file.

Then inadvertently insert a literal token like { into the grammar (when that same token has actually been defined in the lexer already) then is that legal or not?

I did this by mistake and it causes all sorts of weiredness in the antlr VS Code extension (which I just reported) but antlr itself doesn't complain, it process these files with that kind of change and generates .tokens files and does not report any error.

So in my lexer I had this (which I'd forgotten about):

Which I'd totally forgotten about. But in my grammar g4 I simply changed this:

to this

then saved the change and then reran antlr and it ran fine but the antlr GUI tool does not, it gets very upset!

[kaby76]

You can use either '{' or ASM_LBRACE in a parser rule, because Antlr looks up the string '{' and finds that there's the lexer rule ASM_LBRACE : '{';. They resolve to the same token type. If you use a string literal in a parser rule but you didn't define a lexer rule for it, then Antlr defines a new lexer rule, and inserts it before all other lexer rules you wrote. Note, the lexer rule you define has to have the string literal on the RHS and only that string literal for Antlr to match '{' and 'ASM_LBRACE. So, if you defined it as ASM_LBRACE : '{' | '((' ;`, then the aliasing won't happen. If the grammar is split, then the parser rule cannot define new string literals--you have to have a lexer rule for it.

[The-Futurist]

You can use either '{' or ASM_LBRACE in a parser rule, because Antlr looks up the string '{' and finds that there's the lexer rule ASM_LBRACE : '{';. They resolve to the same token type. If you use a string literal in a parser rule but you didn't define a lexer rule for it, then Antlr defines a new lexer rule, and inserts it before all other lexer rules you wrote. Note, the lexer rule you define has to have the string literal on the RHS and only that string literal for Antlr to match '{' and 'ASM_LBRACE. So, if you defined it as ASM_LBRACE : '{' | '((' ;`, then the aliasing won't happen. If the grammar is split, then the parser rule cannot define new string literals--you have to have a lexer rule for it.

Thanks, that's helpful to know, it seems then there's a bug because I do have a split grammar, I did add a literal lexer token to the parser and antlr consumed that and built the parser code (but as I mentioned, the VS Code extension went crazy).

[kaby76]

For a "split grammar", you should have "parser grammar ImperiumParser; options { tokenVocab=ImperiumLexer; }" at the top of the parser grammar ImperiumParser.g4, and "lexer grammar ImperiumLexer;" at the top of the lexer grammar ImperiumLexer.g4. Otherwise, you won't have identical .tokens files, and it's not really a proper "split grammar". When you see "Implicit definition of token IDENTIFIER in parser", it means that you're parser grammar is not working with your lexer grammar. You either don't have it declared as a parser grammar, or you ran "antlr4 ImperiumParser.g4" before running "antlr4 ImperiumLexer.g4". Do not use "import" in the parser grammar file; Make sure to use the tokenVocab option in the parser grammar; Make sure to generate the lexer code first, then the parser code.