Python: escape doc strings or make them raw strings to avoid "SyntaxWarning: invalid escape sequence" on Python 3.12

[armijnhemel]

Since Python 3.12 invalid escape sequences give a SyntaxWarning instead of a DeprecationWarning (see python/cpython#98401 ). In the future these will likely be treated as errors.

Kaitai Struct does not properly escape doc strings, which gives a syntax warning:

$ pylint android_bootldr_qcom.py
<unknown>:12: SyntaxWarning: invalid escape sequence '\['
<unknown>:12: SyntaxWarning: invalid escape sequence '\['
...

The easy fix is to make doc strings raw strings.

[generalmimon]

This reminds me of #170 (comment) 🙂

The point is that we've never thought about the possibility that docstrings may contain any special character sequences. Most of the time they don't, but of course they can.

The easy fix is to make doc strings raw strings.

I like this suggestion. I would've probably come up with escaping backslashes and other characters, but this sounds better.

[generalmimon]

I would've probably come up with escaping backslashes and other characters, but this sounds better.

I suppose that the only character sequence that you have to care about in a raw triple-quoted string (r""") is three quotes in a row ("""), so that's nice.

[generalmimon]

@generalmimon - #1226 (comment):

I suppose that the only character sequence that you have to care about in a raw triple-quoted string (r""") is three quotes in a row ("""), so that's nice.

Technically, we would still have to care about backslash \ as well if it's at the end (see https://docs.python.org/3/reference/lexical_analysis.html#escape-sequences):

Even in a raw literal, quotes can be escaped with a backslash, but the backslash remains in the result; for example, r"\"" is a valid string literal consisting of two characters: a backslash and a double quote; r"\" is not a valid string literal (even a raw string cannot end in an odd number of backslashes). Specifically, a raw literal cannot end in a single backslash (since the backslash would escape the following quote character).

Also Why can't raw strings (r-strings) end with a backslash?

If you're trying to build a pathname for a DOS command, try e.g. one of

dir = r"\this\is\my\dos\dir" "\\"
dir = r"\this\is\my\dos\dir\ "[:-1]
dir = "\\this\\is\\my\\dos\\dir\\"

But for our specific use case in docstrings, I think we're actually saved from this trouble by this code in PythonCompiler.scala:150-161:

  override def universalDoc(doc: DocSpec): Unit = {
    val docStr = doc.summary match {
      case Some(summary) =>
        val lastChar = summary.last
        if (lastChar == '.' || lastChar == '\n') {
          summary
        } else {
          summary + "."
        }
      case None =>
        ""
    }

This ensures that the docstring ends with either a dot (.) or a newline (\n), so it can never end with a backslash.

[armijnhemel]

I would've probably come up with escaping backslashes and other characters, but this sounds better.

I suppose that the only character sequence that you have to care about in a raw triple-quoted string (r""") is three quotes in a row ("""), so that's nice.

This would already be the case right now.

[armijnhemel]

I would've probably come up with escaping backslashes and other characters, but this sounds better.

I suppose that the only character sequence that you have to care about in a raw triple-quoted string (r""") is three quotes in a row ("""), so that's nice.

This would already be the case right now.

Quick test, where I added this to a version of png.ksy:

doc: |
  Test files for APNG can be found at the following locations:

    * <https://philip.html5.org/tests/apng/tests.html>
    * <http://littlesvr.ca/apng/>
  """ yomama

and then compiled it:

class Png(KaitaiStruct):
    """Test files for APNG can be found at the following locations:
    
      * <https://philip.html5.org/tests/apng/tests.html>
      * <http://littlesvr.ca/apng/>
    """ yomama
    """

and pylint doesn't like it:

$ pylint png.py 
************* Module png
png.py:687:9: E0001: Parsing failed: 'unterminated triple-quoted string literal (detected at line 704) (<unknown>, line 687)' (syntax-error)

So this is currently already broken. The correct fix would indeed to escape it.

[generalmimon]

@armijnhemel:

The correct fix would indeed to escape it.

Yeah. About the way to escape it, I'm thinking of ending the triple-quoted string literal, inserting '"""' and resuming the triple-quoted string literal again, like this:

meta:
  id: docstring_triple_quote
doc: |
  Hello """ world!

class DocstringTripleQuote(KaitaiStruct):
    r"""Hello """ '"""' r""" world!
    """

Testing in Python REPL:

>>> class DocstringTripleQuote:
...     r"""Hello """ '"""' r""" world!
...     """
...
>>> DocstringTripleQuote.__doc__
'Hello """ world!\n    '

[armijnhemel]

@armijnhemel:

The correct fix would indeed to escape it.

Yeah. About the way to escape it, I'm thinking of ending the triple-quoted string literal, inserting '"""' and resuming the triple-quoted string literal again, like this:

meta:
id: docstring_triple_quote
doc: |
Hello """ world!

class DocstringTripleQuote(KaitaiStruct):
r"""Hello """ '"""' r""" world!
"""

Testing in Python REPL:

class DocstringTripleQuote:
... r"""Hello """ '"""' r""" world!
... """
...
DocstringTripleQuote.doc
'Hello """ world!\n '

Why not replace any occurence of """ with \"\"\" ?

[generalmimon]

@armijnhemel:

Why not replace any occurence of """ with \"\"\" ?

Well, that won't work in a raw string literal r""" (I'm already considering that), because backslashes would be treated literally.

You can have \"\"\" in a raw string literal, but it will represent these 6 characters literally, which is not what we want in this case:

>>> class DocstringTripleQuote:
...     r"""Hello \"\"\" world!
...     """
...
>>> DocstringTripleQuote.__doc__
'Hello \\"\\"\\" world!\n    '
>>> print(DocstringTripleQuote.__doc__)
Hello \"\"\" world!
    
>>>

[armijnhemel]

@armijnhemel:

Why not replace any occurence of """ with \"\"\" ?

Well, that won't work in a raw string literal r""" (I'm already considering that), because backslash would be treated literally.

You can have \"\"\" in a raw string literal, but it will represent these 6 characters literally, which is not what we want in this case:

class DocstringTripleQuote:
... r"""Hello """ world!
... """
...
DocstringTripleQuote.doc
'Hello \"\"\" world!\n '
print(DocstringTripleQuote.doc)
Hello """ world!

Ah yes, you are right. Splitting the string and then concatenating is the right solution. It feels a bit like a hack though :)

[generalmimon]

@armijnhemel:

It feels a bit like a hack though :)

Yes :) But this is nothing unusual in computer languages. As I see it, you basically have to choose whether:

1. you want a whole bunch of characters with special meaning that you have to escape anywhere in the string, but on the flip side you have access to the universal escaping character \ with which you can escape everything,
2. every character is treated literally except for the single terminator character sequence which cannot be escaped and you need to hack around it if you want it literally.

There are many examples of either category: in Bash / POSIX shell the double quote (") is 1., the single quote (') is 2. You also cannot actually escape a single quote in a single-quoted string, you have to resort to 'here '\'' is a single quote', which is a concatenation of 'here ', \' and ' is a single quote'.

In XML/HTML <![CDATA[...]]> is 2. and if you need ]]> literally, you need to do crazy stuff like this: https://en.wikipedia.org/wiki/CDATA#Nesting