clu test suite failures in v0.0.6 release

[samuela]

I'm trying to run the clu test suite for the v0.0.6 release (latest release at the time of writing). However, it seems to have a number of failures. They all have the form TypeError: '>' not supported between instances of 'int' and 'SplitInfo'. Here's an example:

__________ DeterministicDataTest.test_same_cardinality_on_all_hosts29 __________

self = <clu.deterministic_data_test.DeterministicDataTest testMethod=test_same_cardinality_on_all_hosts29>
num_examples = 9, host_count = 3

    @parameterized.parameters(*itertools.product(range(20), range(1, 4)))
    def test_same_cardinality_on_all_hosts(self, num_examples: int,
                                           host_count: int):
      builder = MyDatasetBuilder({"train": num_examples})
      cardinalities = []
      for host_id in range(host_count):
>       split = deterministic_data.get_read_instruction_for_host(
            split="train",
            num_examples=num_examples,
            host_id=host_id,
            host_count=host_count,
            drop_remainder=True)

clu/deterministic_data_test.py:314:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
clu/deterministic_data.py:237: in get_read_instruction_for_host
    for ri in read_instruction.to_absolute(split_infos):
/nix/store/qlfmbr7b24d0q2nilw9mzi03hhvm65s5-python3.8-tensorflow-datasets-4.4.0/lib/python3.8/site-packages/tensorflow_datasets/core/tfrecords_reader.py:727: in to_absolute
    return [
/nix/store/qlfmbr7b24d0q2nilw9mzi03hhvm65s5-python3.8-tensorflow-datasets-4.4.0/lib/python3.8/site-packages/tensorflow_datasets/core/tfrecords_reader.py:728: in <listcomp>
    _rel_to_abs_instr(rel_instr, name2len)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

rel_instr = _RelativeInstruction(splitname='train', from_=None, to=None, unit='abs', rounding='closest')
name2len = {'train': <SplitInfo num_examples=9, num_shards=1>}

    def _rel_to_abs_instr(rel_instr, name2len):
      """Returns _AbsoluteInstruction instance for given RelativeInstruction.

      Args:
        rel_instr: RelativeInstruction instance.
        name2len: dict {split_name: num_examples}.
      """
      pct_to_abs = (
          _pct_to_abs_closest
          if rel_instr.rounding == 'closest' else _pct_to_abs_pct1)
      split = rel_instr.splitname
      if split not in name2len:
        raise ValueError('Unknown split "{}". Should be one of {}.'.format(
            split, list(name2len)))
      num_examples = name2len[split]
      from_ = rel_instr.from_
      to = rel_instr.to
      if rel_instr.unit == '%':
        from_ = 0 if from_ is None else pct_to_abs(from_, num_examples)
        to = num_examples if to is None else pct_to_abs(to, num_examples)
      else:
        from_ = 0 if from_ is None else from_
        to = num_examples if to is None else to
>     if abs(from_) > num_examples or abs(to) > num_examples:
E     TypeError: '>' not supported between instances of 'int' and 'SplitInfo'

/nix/store/qlfmbr7b24d0q2nilw9mzi03hhvm65s5-python3.8-tensorflow-datasets-4.4.0/lib/python3.8/site-packages/tensorflow_datasets/core/tfrecords_reader.py:566: TypeError
------------------------------ Captured log call -------------------------------
WARNING  absl:deterministic_data.py:196 `num_examples` is deprecated. Please pass `dataset_info` instead.
WARNING  absl:deterministic_data.py:202 `drop_remainder` is deprecated. Please pass `remainder_options` instead. `remainder_options` is reset with RemainderOptions.DROP.

Is there a fix for these failures?

[andsteing]

I saw this error in clu==0.0.5 but it should have been fixed in clu==0.0.6 (in commit google/CommonLoopUtils@f237d24 )

... Can you make sure you're using the latest version clu==0.0.6? And maybe re-install or test in a new virtual env to double check?

[samuela]

Yes I'm seeing this with the latest version (at the time of writing v0.0.6, on this commit google/CommonLoopUtils@f237d24). Nearly every test in clu/deterministic_data_test.py fails.

[samuela]

You can reproduce this by building this derivation and playing around with commenting out this line.