Sporadic SyntaxException w/ Python3.11

[ccwienk]

We use mako for rendering CICD Pipeline Definitions (YAML files). It is a rather complex template w/ a lot of Mako-Function-Definitions and nesting + multithreading. The resulting documents vary in size (depending on input-parameters), and are typically between ~300 kiB and ~1.2 MiB.

The template was used w/ Python-versions 3.6 .. 3.10. When upgrading to 3.11, we saw (and still see) sporadic SyntaxExceptions, which roughly occur in about 5% of the time (w/ unchanged template-parameters, of course!). I started working on a minimalised reproducer. If instantiating the same template w/ same parameters 64 times using 2 threads, I almost always see at least one exception stacktrace. The incriminated lines vary, whereas the Mako-part of the stacktrace seems to be always the same.

The error does not seem to occur when limiting concurrency to just one thread. Thus, I suspect a race-condition, probably within Mako's codebase.

The error occurs for latest versions of python3 alpine (3.11.3-r11) when running inside a virtualisation container, and archlinux (3.11.3-1) when running natively.

Example Stacktrace

Traceback (most recent call last):
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/pyparser.py", line 36, in parse
    return _ast_util.parse(code, "<unknown>", mode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/_ast_util.py", line 91, in parse
    return compile(expr, filename, mode, PyCF_ONLY_AST)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SystemError: AST constructor recursion depth mismatch (before=63, after=65)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/shared_profile/src/sap/makobug-reproducer/concourse/replicator.py", line 140, in render
    definition_descriptor = self._render(definition_descriptor)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/shared_profile/src/sap/makobug-reproducer/concourse/replicator.py", line 211, in _render
    t = mako.template.Template(template_contents, lookup=self.lookup)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/template.py", line 300, in __init__
    (code, module) = _compile_text(self, text, filename)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/template.py", line 677, in _compile_text
    source, lexer = _compile(
                    ^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/template.py", line 657, in _compile
    node = lexer.parse()
           ^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/lexer.py", line 248, in parse
    if self.match_python_block():
       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/lexer.py", line 392, in match_python_block
    self.append_node(
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/lexer.py", line 129, in append_node
    node = nodecls(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/parsetree.py", line 158, in __init__
    self.code = ast.PythonCode(text, **self.exception_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/ast.py", line 42, in __init__
    expr = pyparser.parse(code.lstrip(), "exec", **exception_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/pyparser.py", line 38, in parse
    raise exceptions.SyntaxException(
mako.exceptions.SyntaxException: (SystemError) AST constructor recursion depth mismatch (before=63, after=65) ('import os\n\nimport oci.auth as oa\nimport model.cont') at line: 2 char: 1

It is probably worth mentioning that by decreasing the template's output size, the likelihood of this error seems to become smaller.

I could share a copy of my somewhat slimmed-down reproducer; it still contains most of the code from the repository I referenced above, if this is considered helpful.

[zzzeek]

it could be a bug in py3.11 itself. I haven't dealt with that code in probably more than 10 years so, sure, a real reproducer, as small as possible (it really should just be a single template) is an absolute minimum to do anything here.

[ccwienk]

I created a reproducer (makobug-reproducer.tar.gz).

To use, you need (obviously) python3.11 + install packages from requirements.txt.
To run, simply execute run-me.py

For your convenience, I also built an OCI Container Image (aka Docker Image); Dockerfile is included in uploaded tar-file. So as an alternative to installing and running locally, you might run:

• docker pull eu.gcr.io/gardener-project/cc/makobug-reproducer:1
• docker run eu.gcr.io/gardener-project/cc/makobug-reproducer:1 cc-utils/run-me.py

I hard-coded parameters such as amount of worker-threads (line 106, max_workers) and the amount of renderings to do (line 124, range).

To my experience, this script will yield exceptions as the one I pasted above almost always; I did see occasional runs were no stacktraces were printed. Re-running at most twice always gave stacktraces again.

[zzzeek]

hi that .tar.gz is 283 source files. it's an entire application.

unfortunately I can't run an unknown application of enormous complexity and unknown origin within my own environment; instead, please supply a single, self contained mako template that reproduces the problem when compiled.

if this is not feasible, I'm sure if this issue is widespread it will soon enough be reported in other forms as python 3.11 gains adoption.

[ccwienk]

@zzzeek : would the option to run the mentioned container image be acceptable (this will prevent access to your local filesystem, unless you explicitly mount a local directory tree)?

that being said, I agree that a more stripped-down example would be preferable; since I have thus far no clue as to what actually triggers this bug, this might prove to be rather laborious.

I could imagine the bug might be possible to reproduce by crafting a template that references at least one additional template-module in which a couple of (mako) functions are defined, which in turn accept some formal parameters that are used by some <%if> etc. + that have some nesting + that create some kiBs of output.

So I suppose instead of trying to strip-down from the template I sent, I could give it a shot and craft an "artificial" template from scratch. Will require some more time, though.

[zzzeek]

the error was added as part of python/cpython#95185 and is in python/cpython@86eb500

[zzzeek]

can you run your test with pdb breakpoints and just get the actual contents of the "template_contents" variable here? that's really all we need

Traceback (most recent call last):
  File "/mnt/shared_profile/src/sap/makobug-reproducer/concourse/replicator.py", line 140, in render
    definition_descriptor = self._render(definition_descriptor)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/shared_profile/src/sap/makobug-reproducer/concourse/replicator.py", line 211, in _render
    t = mako.template.Template(template_contents, lookup=self.lookup)

[zzzeek]

or print statements, or dump to a file, etc...

[zzzeek]

the error also indicates invalid Python in the first place...

[CaselIT]

The best option is probably to try a custom version of mako that has

try:
  t = mako.template.Template(template_contents, lookup=self.lookup)
except SyntaxException:
  print(template_contents)
  raise

@zzzeek maybe doing something like this, probably logging to a logger, may make sense regardless of this issue though

[zzzeek]

Mako already does this, poster could also turn this on, see https://docs.makotemplates.org/en/latest/usage.html#handling-exceptions

[ccwienk]

@zzzeek : sorry for replying thus late. I dumped the contents of template_contents into a (gzipped) file

I created this using (as suggested) the following code-block:

 try:
     t = mako.template.Template(template_contents, lookup=self.lookup)
 except:
     with open('/src/sap/makobug-reproducer/template_contents.dump', 'w') as f:
         f.write(template_contents)

it is probably worth mentioning that using a bare except was the only way I could actually handle this exception. Neither of SystemError, mako.exceptions.SyntaxException, SyntaxError worked.

[zzzeek]

hi and thanks for this.

Unfortunately no error is reproduced, I can load this template under Python 3.11.3 and compile it without issue.

what happens if you use the given template in a program like this (assume you put the content into foo.mako)?

from mako.template import Template

t = Template(filename="foo.mako")
print(t.code)

what's the exact Python 3.11 version you are using?

[ccwienk]

Unfortunately no error is reproduced, I can load this template under Python 3.11.3 and compile it without issue.

did you try this w/ multithreading (at least two threads) and multiple executions? As I explained initially, this error occurs sporadically, and only (according to my observations) when concurrency is involved.

If using e.g. four threads, the issue occurs almost always at least once if doing the template-instantiation ~256 times

As stated above, I can reproduce this error in the following environments:

• python3 alpine (3.11.3-r11) - running within a virtualisation container (aka docker container)
• python (3.11.3-2 and 3.11.3-2) from arch linux (no virtualisation involved)

If running this just a couple of times, or running it hundreds / thousands of times, but single-threaded, this error never occurs. However, it occurs quite frequently if doing multithreading (but only as of python3.11).

I will try whether I can reproduce the error using the approach you shared and will write another update to this issue.

[zzzeek]

you can try adding threads to the POC but I can't see any way that threads would have an effect here, the Mako compilation process works on a local datastructure that is not accessible to other functions. I had assumed the error was sporadic only because this particular template was not getting compiled every time.