ANTLR Lexer rules in v4 Priority rules


Example

Several lexer rules can match the same input text. In that case, the token type will be chosen as follows:

  • First, select the lexer rule which matches the longest input
  • If the text matches an implicitly defined token (like '{'), use the implicit rule
  • If several lexer rules match the same input length, choose the first one, based on definition order

The following combined grammar:

grammar LexerPriorityRulesExample;

// Parser rules

randomParserRule: 'foo'; // Implicitly declared token type
    
// Lexer rules
    
BAR: 'bar';
IDENTIFIER: [A-Za-z]+;
BAZ: 'baz';

WS: [ \t\r\n]+ -> skip;

Given the following input:

aaa foo bar baz barz

Will produce the following token sequence from the lexer:

IDENTIFIER 'foo' BAR IDENTIFIER IDENTIFIER
  • aaa is of type IDENTIFIER

    Only the IDENTIFIER rule can match this token, there is no ambiguity.

  • foo is of type 'foo'

    The parser rule randomParserRule introduces the implicit 'foo' token type, which is prioritary over the IDENTIFIER rule.

  • bar is of type BAR

    This text matches the BAR rule, which is defined before the IDENTIFIER rule, and therefore has precedence.

  • baz is of type IDENTIFIER

    This text matches the BAZ rule, but it also matches the IDENTIFIER rule. The latter is chosen as it is defined before BAR.

    Given the grammar, BAZ will never be able to match, as the IDENTIFIER rule already covers everything BAZ can match.

  • barz is of type IDENTIFIER

    The BAR rule can match the first 3 characters of this string (bar), but the IDENTIFIER rule will match 4 characters. As IDENTIFIER matches a longer substring, it is chosen over BAR.

As a rule of thumb, specific rules should de defined before more generic rules. If a rule can only match an input which is already covered by a previously defined rule, it will never be used.

Implicitly defined rules such as 'foo' act as if they were defined before all other lexer rules.