Setting hugging face datasets in SpeechLM2

[mhenrichsen]

Hi,

I looked through your Canary-qwen example config and was wondering if it's possible to use HF datasets. If not, what format do the trainer expect the datasets to be in?

Column names etc.

[pzelasko]

I'm in progress of writing a detailed tutorial about setting all this up, so in the meantime:

HF datasets are not directly supported yet but I hope enabling this should be straightforward.

Our dataloaders are based on Lhotse, which already supports wrapping HF dataset:
https://github.com/lhotse-speech/lhotse/blob/fda1a986e5e1e72a14c82049b4ee709fc09a81e6/lhotse/cut/set.py#L2643-L2688
We'll only need to make sure there's a way to shuffle the underlying HF dataset with different seeds for different gpu+dataloader workers. If anybody is interested in picking this up before I can, the proper way would be to add a new @data_type_parser here:

NeMo/nemo/collections/common/data/lhotse/cutset.py

Lines 100 to 126 in c17a0bf

	def data_type_parser(name: Union[str, list[str]]):
	"""
	Decorator used to register data type parser functions.
	Parsing function reads a dataloading config and returns a tuple
	of lhotse ``CutSet`` and boolean indicating whether we should use
	iterable dataset (True) or map dataset (False) mechanism ("is tarred").
	Example:
	>>> @data_type_parser("my_new_format")
	... def my_new_format(config):
	... return CutSet(read_my_format(**config)), True
	...
	... fn = get_parser_fn("my_new_format")
	... cuts, is_tarred = fn({"my_arg_0": ..., "my_arg_1": ..., ...})
	"""
	def _decorator(fn):
	global KNOWN_DATA_CONFIG_TYPES
	if isinstance(name, str):
	KNOWN_DATA_CONFIG_TYPES[name] = fn
	else:
	for n in name:
	KNOWN_DATA_CONFIG_TYPES[n] = fn
	return fn
	return _decorator

What data formats are supported and how to specify them

95% of dataloader code in NeMo is currently shared between all speech collections (asr, tts, enhancement, speechlm2, etc), and our approach has been to try and support as many formats as possible for everything, because people tend to have existing data in some specific format and requiring a copy is often prohibitive/throws them off.
For speechlm2 specifically, every format is being mapped into an iterable of NeMoMultimodalConversation.

Generally supported formats:

• NeMo ASR manifest (tarred and non-tarred): https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/datasets.html#preparing-custom-asr-data
• Lhotse CutSet manifests (normal and Lhotse Shar, ie tarred): https://lhotse.readthedocs.io/en/latest/cuts.html#cuts
• JSONL manifests with NeMoMultimodalConversation objects: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/datasets.html#configuring-multimodal-dataloading

This unit test setup is a great example how to format data directly as conversations:

NeMo/tests/collections/common/test_lhotse_multimodal_dataloading.py

Lines 46 to 92 in c17a0bf

	@pytest.fixture(scope="session")
	def multimodal_conversations_path(tmp_path_factory):
	tmp_path = tmp_path_factory.mktemp("text_data")
	en_path = tmp_path / "manifest.json"
	data = [
	{
	"id": "convo_1",
	"conversations": [
	{
	"value": "Can you help summarize the following?",
	"from": "User",
	"type": "text",
	},
	{
	"value": "123.wav",
	"from": "User",
	"type": "audio",
	"duration": 5.73,
	},
	{
	"value": "I'm glad to assist you with your request. Here's a summary:",
	"from": "Assistant",
	"type": "text",
	},
	{
	"value": "123_answer.wav",
	"from": "Assistant",
	"type": "audio",
	"duration": 7.11,
	},
	{
	"value": "Can you further shorten it?",
	"from": "User",
	"type": "text",
	},
	{
	"value": "Of course!",
	"from": "Assistant",
	"type": "text",
	},
	],
	}
	]
	lhotse.serialization.save_to_jsonl(data, en_path)
	dummy_recording(0, 5.73, with_data=True).to_cut().save_audio(tmp_path / "123.wav")
	dummy_recording(0, 7.11, with_data=True).to_cut().save_audio(tmp_path / "123_answer.wav")
	return en_path

Setting up the dataloader via config:

• Docs https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/datasets.html#extended-multi-dataset-configuration-format
• Tutorial (look for NeMoMultimodalConversation specifically) https://github.com/NVIDIA/NeMo/blob/main/tutorials/multimodal/Multimodal%20Lhotse%20Dataloading.ipynb

The tutorial I mentioned will show end-to-end example of how to put together all this stuff, but for now just a bunch of examples to make this more digestible, using the SALM config you pointed to:

Example config for dataloading from NeMo ASR manifests

data:
  train_ds:
    ...
    input_cfg:
      - type: lhotse_as_conversation  # converts nemo and lhotse data to multimodal conversations 
        # non-tarred
        manifest_filepath: /path/to/manifest.json
        # tarred
        # manifest_filepath: /path/to/tarred/sharded_manifests/manifest__OP_0..127_CL_.json
        # tarred_audio_filepaths: /path/to/tarred/audio__OP_0..127_CL_.tar
        audio_locator_tag: ${model.audio_locator_tag}
        tags:
          # Uncomment below line to include a system prompt, if supported by the model's prompt formatter.
          # system_prompt: "some system prompt"
          context: "Transcribe the following:"

Example config for dataloading from Lhotse manifests

data:
  train_ds:
    ...
    input_cfg:
      - type: lhotse_as_conversation  # converts nemo and lhotse data to multimodal conversations 
        # non-tarred
        cuts_path: /path/to/cuts.jsonl.gz
        # tarred
        # shar_path: /path/to/lhotse_shar_data/
        audio_locator_tag: ${model.audio_locator_tag}
        tags:
          # Uncomment below line to include a system prompt, if supported by the model's prompt formatter.
          # system_prompt: "some system prompt"
          context: "Transcribe the following:"

Example config for reading NeMoMultimodalConversation directly

data:
  train_ds:
    ...
    input_cfg:
      - type: multimodal_conversation
        manifest_filepath: /path/to/conversation.jsonl
        # tarred
        # manifest_filepath: /path/to/tarred_conversations/sharded_manifests/manifest__OP_0..127_CL_.json
        # tarred_audio_filepaths: /path/to/tarred_conversations/audio__OP_0..127_CL_.tar
        audio_locator_tag: ${model.audio_locator_tag}
        # tags shouldn't be needed, your data will likely already include all prompts

[JRD971000]

Thanks @pzelasko; hi @mhenrichsen , is your issue resolved?