None values not supported error when using seqio mixture

[StephennFernandes]

i get the error None values not supported when make a seqio_mixtures on multiple span_curruption tasks and using that mixture as a task to pretrain mt5.

However, the same task.py works fine on each individual task, while pretraining T5 or pretraining mt5 only on single language.
the error arises only when using seqio mixture.

I use the default mt5 tokenizer gs://t5-data/vocabs/mc4.250000.100extra/sentencepiece.model witrh extra_ids=0 and i stil get the same error.

The following is the stack track of the same:

    raise e.with_traceback(filtered_tb) from None
  File "/home/stephen/anaconda3/lib/python3.9/site-packages/tensorflow/python/framework/tensor_util.py", line 441, in make_tensor_proto
    raise ValueError("None values not supported.")
ValueError: in user code:

    File "/home/stephen/anaconda3/lib/python3.9/site-packages/seqio/utils.py", line 779, in None  *
        lambda arg: fn(arg, *args, **kargs)
    File "/home/stephen/anaconda3/lib/python3.9/site-packages/seqio/preprocessors.py", line 116, in tokenize_impl  *
        v = vocab.encode_tf(v)
    File "/home/stephen/anaconda3/lib/python3.9/site-packages/seqio/vocabularies.py", line 114, in encode_tf  *
        return self._encode_tf(s)
    File "/home/stephen/anaconda3/lib/python3.9/site-packages/seqio/vocabularies.py", line 413, in _encode_tf  *
        return self.tf_tokenizer.tokenize(s)
    File "/home/stephen/anaconda3/lib/python3.9/site-packages/tensorflow_text/python/ops/sentencepiece_tokenizer.py", line 133, in tokenize  *
        input_tensor = ragged_tensor.convert_to_tensor_or_ragged_tensor(input)

    ValueError: None values not supported.

  In call to configurable 'train' (<function train at 0x7f79d8db2280>)

This is how my task.py file looks

import functools
import seqio
import tensorflow as tf
import t5.data
from datasets import load_from_disk, load_dataset
from t5.data import postprocessors
from t5.data import preprocessors
from t5.evaluation import metrics
from seqio import FunctionDataSource, utils

TaskRegistry = seqio.TaskRegistry
vocabulary = seqio.SentencePieceVocabulary('gs://t5-data/vocabs/mc4.250000.100extra/sentencepiece.model', extra_ids=0)


DEFAULT_OUTPUT_FEATURES = {
    "inputs": seqio.Feature(
        vocabulary=vocabulary, add_eos=True,
        required=False),
    "targets": seqio.Feature(
        vocabulary=vocabulary, add_eos=True)
}



def gen_dataset(split, shuffle=False, seed=None, column="text", dataset_path=None):
    dataset = load_dataset(dataset_path, streaming=True, use_auth_token=True)
    if shuffle:
        if seed:
            dataset = dataset.shuffle(seed=seed)
        else:
            dataset = dataset.shuffle()
    while True:
        for item in dataset[str(split)]:
            yield item[column]


def dataset_fn(split, shuffle_files, seed=None, dataset_path=None):
    return tf.data.Dataset.from_generator(
        functools.partial(gen_dataset, split, shuffle_files, seed, dataset_path=dataset_path),
        output_signature=tf.TensorSpec(shape=(), dtype=tf.string, name=dataset_path)
    )


@utils.map_over_dataset
def target_to_key(x, key_map, target_key):
    """Assign the value from the dataset to target_key in key_map"""
    return {**key_map, target_key: x}

# link to the mt5 sentencepiece tokenizer vocabulary
vocabulary = seqio.SentencePieceVocabulary('gs://t5-data/vocabs/mc4.250000.100extra/sentencepiece.model', extra_ids=0)

TaskRegistry.add(
    "hindi_span_curruption",
    source=seqio.FunctionDataSource(
        dataset_fn=functools.partial(dataset_fn, dataset_path='StephennFernandes/ciil_mega_corpus_hindi'),
        splits=("train", "validation"),
        caching_permitted=False),
    preprocessors=[
        functools.partial(
            target_to_key, key_map={
                "inputs": None,
                "targets": None,
            }, target_key="targets"),
        seqio.preprocessors.tokenize,
        # seqio.CacheDatasetPlaceholder(),
        preprocessors.span_corruption, 
        seqio.preprocessors.append_eos_after_trim,
    ],
    output_features={"targets": DEFAULT_OUTPUT_FEATURES["targets"],"inputs": seqio.Feature(vocabulary=vocabulary,add_eos=True)},
    metric_fns=[]
)
### similar multiple tasks exist for multiple languages. ### 

seqio.MixtureRegistry.add(
  "ciil_mix_3",
  ["assamese_span_curruption", "bengali_span_curruption", 
  "bhisnupuriya_span_curruption", "bodo_span_curruption", 
  "divehi_span_curruption", "dogri_span_curruption", 
  "english_span_curruption", "gujarati_span_curruption",
  "hindi_span_curruption", "kannada_span_curruption", 
  "kashmiri_span_curruption", "konkani_span_curruption", 
  "maithili_span_curruption", "malayalam_span_curruption",
  "manipuri_span_curruption", "marathi_span_curruption",
  "nepali_span_curruption", "odia_span_curruption",
  "panjabi_span_curruption", "sanskrit_span_curruption",
  "tamil_span_curruption", "telugu_span_curruption",
   "urdu_span_curruption" ],
  default_rate=3
)

i further use the ciil_mix_3 mixture in my .gin file
this is how my .gin file looks

from __gin__ import dynamic_registration
import t5.data.mixtures
import __main__ as train_script


include 't5x/examples/t5/mt5/base.gin'
include 't5x/configs/runs/pretrain.gin'

import task 

MIXTURE_OR_TASK_NAME = "ciil_mix_3"
TASK_FEATURE_LENGTHS = {"inputs": 512, "targets": 114}
TRAIN_STEPS = 100000
DROPOUT_RATE = 0.0
BATCH_SIZE = 32


train_script.train:
  eval_period = 2000

I even further tried to work the same with byT5 and the same error occurs:
thr following is the error for byT5

ValueError: in user code:

    File "/home/stephen/anaconda3/lib/python3.9/site-packages/seqio/utils.py", line 779, in None  *
        lambda arg: fn(arg, *args, **kargs)
    File "/home/stephen/anaconda3/lib/python3.9/site-packages/seqio/preprocessors.py", line 116, in tokenize_impl  *
        v = vocab.encode_tf(v)
    File "/home/stephen/anaconda3/lib/python3.9/site-packages/seqio/vocabularies.py", line 114, in encode_tf  *
        return self._encode_tf(s)
    File "/home/stephen/anaconda3/lib/python3.9/site-packages/seqio/vocabularies.py", line 555, in _encode_tf  *
        tf_ids = tf.io.decode_raw(s, tf.uint8) + self._num_special_tokens

    ValueError: Tried to convert 'bytes' to a tensor and failed. Error: None values not supported.

[StephennFernandes]

The following is an update on the same:

when i change my preprocessors target_to_key, key_map input and output from None to text everything works fine and i am able to pretrain the mt5 model.

from this:

preprocessors=[
        functools.partial(
            target_to_key, key_map={
                "inputs": None,
                "targets": None,
            }, target_key="targets"),

to this:

preprocessors=[
        functools.partial(
            target_to_key, key_map={
                "inputs": "text",
                "targets": "text",
            }, target_key="targets"),

i had the following doubts on this:

• what actually happens on changing inputs and targets to "text" from None , is setting it to "text" acceptable in the pretraining pipeline ?

• whats the difference in using output features

  from this:
  output_features={"targets": DEFAULT_OUTPUT_FEATURES["targets"]},

  to this:
  output_features={"targets": DEFAULT_OUTPUT_FEATURES["targets"],"inputs": seqio.Feature(vocabulary=vocabulary,add_eos=True)},

  is this acceptable in mt5 model pretraining ?

(asking because i havent seen any t5 or mt5 using input and target as "text" and the output_features are really confusing on which one to use)

[adarob]

We only use "text" in the context of renaming key "text" to "inputs" or "targets". We are not assigning the value "text" to any feature, as you're doing here. I think you are misunderstanding what preprocessors.rekey does.

[StephennFernandes]

@adarob , thanks for helping me out.

the thing is everything works fine on a single task. (single language)

the following is my script for a single language:

TaskRegistry.add(
    "english_span_curruption",
    source=seqio.FunctionDataSource(
        dataset_fn=functools.partial(dataset_fn, dataset_path='StephennFernandes/ciil_mega_corpus_english'),
        splits=("train", "validation"),
        caching_permitted=False),
    preprocessors=[
        functools.partial(
              target_to_key,
              key_map={
                "inputs": None,
                "targets": None,
            }, target_key="targets"),
          seqio.preprocessors.tokenize,
          #seqio.CacheDatasetPlaceholder(),
          t5.data.preprocessors.span_corruption,
          seqio.preprocessors.append_eos_after_trim,
      ],
      output_features={"targets": DEFAULT_OUTPUT_FEATURES["targets"]},
      metric_fns=[])

the issue is when i create multiple tasks for pretraining with the same format, and use it with the mixture and provide the mixture name in the pretrain.gin file i get the following error.

eg:

eqio.MixtureRegistry.add(
  "ciil_mix_3",
  ["assamese_span_curruption", "bengali_span_curruption", 
  "bhisnupuriya_span_curruption", "bodo_span_curruption", 
  "divehi_span_curruption", "dogri_span_curruption", 
  "english_span_curruption", "urdu_span_curruption" ],
  default_rate=3
)

ValueError: None values not supported.

  In call to configurable 'train' (<function train at 0x7fa1d0f76ca0>)

Its only when i use a mixture i get the following error, rest assured everything works fine on single language task.

ps: i feel like this could be a bug maybe.

please let me know what you think.

[adarob]

Are you sure all of those tasks work individually?

[StephennFernandes]

yes iam pretty sure. ive even trained then for a couple of checkpoints. there are some folks that have even used it in downstream tasks.

the only issue is when used with a mixture to pretrain mt5 the error occurs.

[StephennFernandes]

As I mentioned earlier, Renaming the inputs and outputs from None to "text" i was able to pre-train mt5 and byt5 Base models.

the following are their wandb logs creating from ingesting the tensorboard logs.
mt5 & byt5 i got a reasonable amount of accuracies in them too.

[StephennFernandes]

@adarob hey, by any chance we you able to find any solution to this ?

does my wandb logs on pretraining mt5 & byt5 look good ?
i havent yet tried their performance from downstream task.

but please let me know what you think on their pre-training metrics.

btw just looking at the wandb pretrain metrics do you think the implementation i did is okay and acceptable ?