Possible error in LOAD_FAST_LOAD_FAST opcode: test_statistics: test_kde_random spurious failure

[colesbury]

Seen in https://buildbot.python.org/all/#/builders/310/builds/1665/steps/6/logs/stdio

======================================================================
ERROR: test_kde_random (test.test_statistics.TestKDE.test_kde_random) (kernel='gauss')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/test/test_statistics.py", line 2518, in test_kde_random
    big_sample = sorted([rand() for i in range(n)])
                         ~~~~^^
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/statistics.py", line 1122, in rand
    return choice(data) + h * invcdf(random())
           ~~~~~~^^^^^^
  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/random.py", line 352, in choice
    return seq[self._randbelow(len(seq))]
           ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'Random' object is not subscriptable

----------------------------------------------------------------------

cpython/Lib/test/test_statistics.py

Line 2518 in c989e74

big_sample = sorted([rand() for i in range(n)])

cpython/Lib/statistics.py

Lines 1117 to 1122 in c989e74

	prng = _random.Random(seed)
	random = prng.random
	choice = prng.choice
	def rand():
	return choice(data) + h * invcdf(random())

cpython/Lib/random.py

Lines 345 to 352 in c989e74

	def choice(self, seq):
	"""Choose a random element from a non-empty sequence."""
	# As an accommodation for NumPy, we don't use "if not seq"
	# because bool(numpy.array()) raises a ValueError.
	if not len(seq):
	raise IndexError('Cannot choose from an empty sequence')
	return seq[self._randbelow(len(seq))]

cpython/Lib/random.py

Lines 225 to 243 in c989e74

	def __init_subclass__(cls, /, **kwargs):
	"""Control how subclasses generate random integers.
	The algorithm a subclass can use depends on the random() and/or
	getrandbits() implementation available to it and determines
	whether it can generate random integers from arbitrarily large
	ranges.
	"""
	for c in cls.__mro__:
	if '_randbelow' in c.__dict__:
	# just inherit it
	break
	if 'getrandbits' in c.__dict__:
	cls._randbelow = cls._randbelow_with_getrandbits
	break
	if 'random' in c.__dict__:
	cls._randbelow = cls._randbelow_without_getrandbits
	break

[picnixz]

cc @rhettinger

[vstinner]

Seen in https://buildbot.python.org/all/#/builders/310/builds/1665/steps/6/logs/stdio

That's a build on the pull request #130849

Did you see this error on a regular buildbot build? I don't see test_statistics failures on recent "AMD64 Debian root 3.x" builds.

[colesbury]

It seems clear to me that the failure is unrelated to the PR. I don't generally actively look at regular buildbots unless there's a specific failure, so it's easy to miss things that fail rarely, especially since we retry failures.

In order of decreasing confidence I think:

• This is not due to a bug in the PR
• This is not due to a bug in the test, statistics module, or random module
• This is possibly a bug in specialization, MRO cache, or some other related part of the interpreter core

[rhettinger]

  File "/root/buildarea/pull_request.angelico-debian-amd64/build/Lib/random.py", line 352, in choice
    return seq[self._randbelow(len(seq))]
           ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'Random' object is not subscriptable

The failure of line 352 in random.choice() is perplexing. The only way to get this error message is for seq to have been an instance of random.Random; however, if that were true, the prior call to len(seq) inside the brackets would have failed beforehand:

>>> import random
>>> seq = random.Random()
>>> seq[0]
Traceback (most recent call last):
  File "<pyshell#24>", line 1, in <module>
    seq[0]
TypeError: 'Random' object is not subscriptable

>>> len(seq)
Traceback (most recent call last):
  File "<pyshell#26>", line 1, in <module>
    len(seq)
TypeError: object of type 'Random' has no len()

Looking at the disassembly of random.Random.choice, the most likely culprit is LOAD_FAST_LOAD_FAST (seq, self) executing as if it has been written LOAD_FAST_LOAD_FAST (self, self):

  8   L1:     LOAD_FAST_LOAD_FAST     16 (seq, self)
              LOAD_ATTR                5 (_randbelow + NULL|self)
              LOAD_GLOBAL              1 (len + NULL)
              LOAD_FAST                1 (seq)
              CALL                     1
              CALL                     1
              BINARY_OP               26 ([])
              RETURN_VALUE

[colesbury]

I think this is likely a hardware failure. There are other similar unexplainable failures on that worker recently, but not on the other buildbots, and that machine was undergoing thermal stress testing around the same time. See:

• test_math.test_sumprod_stress: "_PyEval_EvalFrameDefault: Assertion `oparg & 1' failed" #130896 (comment).