Accessing skipped whitespace

[pdubroy]

In earlier versions of Ohm there was a way to access the implicitly-skipped whitespace. This got removed at some point and wasn't added back.

We should add this back, but it's not (yet) obvious what the right way to do it is.

[pdubroy]

Randomly ran across this on the Roslyn wiki:

How are comments stored in the syntax tree (and how to use the Syntax Visualizer)?

Roslyn trees store comments as trivia on tokens. The heuristic used is to attach trivia to the token following it (that is, as
leading trivia on the node containing the token). However, if the trivia is after the last token on a line, it attaches to the preceding token (that is, as trailing trivia).

In the following example, the first comment attaches as LeadingTrivia to the ‘int’ token in the declaration following it. You can also see the leading trivia on the LocalDeclarationStatementSyntax node containing the ‘int’ token. The first comment trivia attaches to the first token following it even though ‘int’ is on a different line because there are no tokens on the line with the comment.

 int method1 () {
  int a = 1;
  // comment1
  int b = 1; // comment 2
}

The second comment attaches as TrailingTrivia to the semi-colon, and you can also see it on the LocalDeclarationStatementSyntax node.

[pdubroy]

Current plan that @alexwarth and I discussed is to making leading whitespace available via a property/method on the node wrappers. Something a little like this:

semantics.addOperation('interpret', {
  // …
  AddExp_plus(x, _, y) {
    if (x.leadingSpaces.sourceString.length > 0) { // 👈
      console.log('x has leading spaces!');
    }
    return x.interpret() + y.interpret();
  },
});

One tricky thing is that when you call match('MyRule') in Ohm, it implicitly means "match MyRule followed by end", and if MyRule is syntactic, we skip spaces before the end. But there's currently no way to write a semantic action for that implicit top-level expression, and no way to get access to the node corresponding to end.

A few options:

1. We discussed adding a special _root action, that would allow you to write a semantic action for the top-level expression:

semantics.addOperation('interpret', {
  _root(myRule, end) {
    if (end.leadingSpaces.sourceString.length > 0) { // 👈
      console.log('founding trailing spaces!');
    }
  },
  // …
});

• This keep uniformity; in this view, all skipped spaces are leading spaces, and are accessed the same way.
• We have to introduce a new, specially-named _root action, and understanding it requires understand the implicit top-level expression. Currently, our documentation says "If the start rule is a syntactic rule, both leading and trailing spaces are skipped around the top-level application", which is (imo) a simpler way to describe it.
• I'm having trouble coming up with another thing you'd use the _root action for…which makes me think there may be a simpler solution.

2. We could also have a way of getting trailing spaces from the MatchResult:

const matchResult = g.match('MyRule');
if (matchResult.trailingSpaces.sourceString.length > 0) {
  console.log('founding trailing spaces!');
}

• We lose uniformity — everything else is leading space, this is trailing space.
• It seems to preclude using any kind of operation on the node, because we're not in the context of a semantics. It would work if we just want to expose spaces as raw characters though.

3. We could expose a trailingSpaces property on all nodes, and it would only be non-null on the root node. So you could write an action like this:

  semantics.addOperation('interpret', {
    MyRule(a, b) {
      if (this.trailingSpaces.sourceString.length > 0) { // 👈
        console.log('founding trailing spaces!');
      }
    },
    // …
  });