Discrepancies Between [code Node Behavior and Its Documentation

[tajmone]

@pml-lang, I'm a bit confused by the [code node behaviour in PML 2.3.0...

The following example compiles without error:

[doc [title Bug Test]

    [code print("Hello World!") code]
    [code print("Hello World!") ]

]

Except that the results are not as expected, i.e. it seems that the closing tag is no longer node] but just ], since I get the following output:

print("Hello World!") code

print("Hello World!")

Both the current documentation and PML JSON Tags still mention the closing tag to be code], but from the above examples is clear that only the bracket is taken into account.

From the Sublime PML point of view, what should I do? Should I amend the syntax to match the real behaviour of PML? or is this a bug that will be fixed in the next update (PML 3)?

[pml-lang]

the results are not as expected

The reason is that node code (as well as nodes input, output, and html) are configured to be of PDML type raw_text. As explained in chapter raw_text, there are three syntax variations supported for these nodes. However, only the primary syntax (called Delimited Text Syntax) is currently mentioned in the PML docs, and thus officially supported. I've now gotten aware that the three syntax variations need to be mentioned in the PML docs. There should at least be a link to PDML's raw_text type.

Note: In older PML versions, only the Text Block Syntax syntax variation was supported.

Sorry for the confusion.

the current documentation and PML JSON Tags still mention the closing tag to be code]

This has been changed already.

In the reference manual, only the node name is mentioned, the closing tag has been removed.

And in the JSON file, field closing_tag has been removed (as mentioned in the new Changelog), and field is_raw_text_block is set to true for all nodes of type raw_text.

[tajmone]

Do these variations also apply to [table_data nodes? Since this node is also indicated as is_raw_text_block in the generated JSON tags?

[pml-lang]

Do these variations also apply to [table_data nodes?

Yes, they apply to all nodes with is_raw_text_block=true

[tajmone]

As explained in chapter raw_text, there are three syntax variations supported for these nodes. However, only the primary syntax (called Delimited Text Syntax) is currently mentioned in the PML docs, and thus officially supported.

I didn't know that. So, since in my example the contents are on the same line as the tags I was unknowingly using the Standard Text Syntax, hence the code in code] was being treated as part of the node contents.

I've now gotten aware that the three syntax variations need to be mentioned in the PML docs. There should at least be a link to PDML's raw_text type.

Indeed you should. It seems like I'll have to rework the entire Sublime PML syntax in order to support these variations (hoping that it's even possible).

The Variations Might Hinder Editor Syntaxes

Honestly, the existence of these three different variations is going to make writing editor syntaxes much harder (possibly even more that lenient parsing does for attributes).

This is a case that needs to be handled via backtracking, since you don't really know which of the three syntaxes is being used until you've parsed multiple lines. As far as I know, among modern editors with TextMate like syntax definitions, Sublime Text remains the only one that offers syntax backtracking and branching for handling ambiguous grammars — so I'm glad that ultimately I stuck to working on the ST syntax, instead of going for VSCode, since I wouldn't have liked to work on package that was destined to poorly cover the syntax.

One might argue that most syntaxes could just go for the "official variation" mentioned in the docs, but this won't really work since users can still encounter the other variations in third party documents — and the Standard Text Syntax needs to apply escaping rules, and for the others it would be nice to have the block delimiters properly scoped and highlighted.

Anyhow, I'll try to implement the other variations in Sublime PML after migrating to PML 3.0.

[pml-lang]

So, since in my example the contents are on the same line as the tags I was unknowingly using the Standard Text Syntax, hence the code in code] was being treated as part of the node contents.

Yes, exactly.

Honestly, the existence of these three different variations is going to make writing editor syntaxes much harder

True. They are convenient for end-users, but hard to implement in plugins with limited programming capabilities.

This is a case that needs to be handled via backtracking ...

Yes. Without backtracking, it's probably not possible to correctly support the three syntax variations.

One might argue that most syntaxes could just go for the "official variation" mentioned in the docs

That might indeed be a very reasonable approach, especially in an early version of the plugin.