Precedence When Matching/Lexing

[cgbsu]

Possible to use precedence in pattern matching step?

I have 3 terms on for digits ("[0-9]+"), "u" and one for identifiers "[a-zA-Z_][a-zA-Z_0-9]+"

I "u" to identify an unsigned integer and I can specify its bit arity with a number after it

e.g "100u8" is an an unsigned 8-bit integer with value 100.

100u works fine (unsigned integer with value 100 and unspecified bit-arity)

However 100 is pared as "Digits" <- "Base10Digits", (I do this because I can parse digits in different bases) then "u8" is parsed as an identifier, and there is no rule to match this.

Right now "u" has a higher precedence than "Identifier" but identifier gets matched first.

Would it be possible to:
A ) Try matching patterns with higher precedence first
B ) If there is no suitable rule for the matched term, fallback and see if there is a suitable term (case: Unexpected Identifier)?

[peter-winter]

Right now there is no mechanism for terms precedence.
The simplest one which is just whichever is first passed in terms(...) call in parser definition should have the biggest precedence shouldn't be hard to implement.
I will consider adding the feature in near future.

[peter-winter]

After a brief analyze of the code I can tell that the term that is first on the terms(...) call in parser definition is actually matched first.
So I can't see the problem here, can you please provide some code?
Is "u" actually separate term? Why? It seems it should be part of the number term.

[evan-goode]

I'm trying to write a parser for Java's MANIFEST.MF format (spec) and this behavior would be useful for me:

B ) If there is no suitable rule for the matched term, fallback and see if there is a suitable term (case: Unexpected Identifier)?

A MANIFEST.MF file looks like:

Manifest-Version: 1.0
Implementation-Title: foobar
Implementation-Version: 1.2.3

The language specification requires the value for Manifest-Version to be in a specific format:

version-info:                 Manifest-Version : version-number
version-number:               digit+{.digit+}*

So I use the following regex_term:

// version-number: digit+{.digit+}*
// digit:          {0-9}
constexpr char version_number_pattern[] = "[0-9](\\.[0-9])*";
constexpr regex_term<version_number_pattern> version_number("version-number");

And the rule:

version_info("Manifest-Version", ':', ' ', version_number)

For other key-value pairs, I use a more general regex_term:

// otherchars:      any UTF-8 character except NUL, CR and LF
// I match these one at a time, unfortunately adding a * to make [^\r\n]* segfaults gcc. But that's another issue.
constexpr char otherchar_pattern[] = "[^\\r\\n]";
constexpr regex_term<otherchar_pattern> otherchar("otherchar");

My problem is that the version-number term is a special case of the otherchar term. If I list version-number first in the terms(...) call, then MyKey: 1 would not parse correctly since the parser parses the 1 as a version-number, not as otherchar, and errors Syntax error: Unexpected 'version-number'.

If I instead list otherchar first in the terms list, then I get the opposite error whenever my grammar expects a version-number: Syntax error: Unexpected 'otherchar'.

I don't have much experience with parsing, so I may be simply misunderstanding something.

[peter-winter]

First of all, if your language doesn't do any specific whitespace processing (like indentation sensitive languages) you should not include them in the grammar at all, ctpg generated parsers skip whitespace by default.

I would do it like you do, with version-number term defined first so it has precedence, and simply include the version-number term in a syntax rule for 'other key value pairs'. Can you show me the grammar as well?

[evan-goode]

First of all, if your language doesn't do any specific whitespace processing (like indentation sensitive languages) you should not include them in the grammar at all, ctpg generated parsers skip whitespace by default.

Thanks for your reply. Yes, I might be able to get by with only parsing newlines, but the grammar does not have an equivalent of semicolons, lines are terminated only with newlines.

I would do it like you do, with version-number term defined first so it has precedence, and simply include the version-number term in a syntax rule for 'other key value pairs'.

I think I see what you're saying. In general though, this could be a lot of manual work. Correct me if I'm wrong, but you'd need to:

• For each terminal a in the terms list:
  • Create a nonterminal A with type string and the rule A(a) >= create<string>()
  • In the existing rules, replace occurrences of a with A
  • For each terminal b that precedes a:
    • If there exists an instance of a that could be recognized as b (e.g. in my case, a = otherchar, b = version-string):
    • Add the rule A(b) >= create<string>()

Can you show me the grammar as well?

Sure, here is the parser I've got so far: https://gist.github.com/evan-goode/a5d97348f96328c956792c0660e414b0. And the grammar specification is here: https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Manifest_Specification.