Pipe (bar) character (“|”) in MISC values

[msklvsk]

The guidelines say MISC field

has to be formatted as a list that can be split on the bar character (|) without special escaping.

What if a value in MISC contains the bar character? Like LTranslit=||Translit=|.

[foxik]

I believe the guidelines are unambiguous -- all bar characters are split points.

In UDPipe, we perform an internal escape mechanism to allow bars as values when needed (if you want details, we use c-string-like formatting, where \p is a bar character [pipe], \n newline, \s space, etc, \\ is a backslash).

[dan-zeman]

Then I'm afraid some kind of escaping is necessary. In case of LTranslit and Translit, it might be sensible to simply omit the features (provided they're just copies of LEMMA and FORM, not really transliterations of some non-Latin character). But I have to admit that my own code does not handle such cases correctly.

[msklvsk]

I guess I’ll omit the translit for “|” for time being, leaving it as a surprise for downstream applications.
No real solution without an escaping standart which is inevitable for any such format.

[amir-zeldes]

Another option is to allow XML-style Unicode escapes, i.e. | == |

[msklvsk]

The problem isn’t solved. No intention to introduce escaping?

[dan-zeman]

@msklvsk : Do you mean to standardize escaping in the specification of the CoNLL-U format?

[msklvsk]

Yes, I mean to change this part:

it has to be formatted as a list that can be split on the bar character (|) without special escaping

[dan-zeman]

I think that the intended meaning of "without special escaping" was not to ban escaping of the vertical bar inside individual items of the list. It was meant to say that the escaping must not involve the "|" character itself (e.g. with backslash, "\|") because then a simple split("|") would not work.

[msklvsk]

Understood. Then we should agree on an escaping convention that doesn’t involve the pipe, like what foxik or amir-zeldes suggested. I like \p for pipe and \\ for backslash. XML-style would mean lots of & and may be confusing because actual XML special chars are not escaped.