[feature] Improve general support for local datasets and documents

[supreme-gg-gg]

User Story

Sometimes suitable datasets cannot be found on huggingface and it's better to use a smaller local dataset + synthesis / augmentation using some local files such as PDF, word, slides, etc. Handling these media would be helpful as well to turn documents into trainable datasets!

This issue can be tackled in the following steps:

1. First validate and enhance general support for local datasets uploading + preprocessing
2. Implement support for vision local datasets
3. Implement various synthesis and augmentation strategies powered by https://github.com/meta-llama/synthetic-data-kit

The tool is designed to follow a simple CLI structure with 4 commands:

ingest various file formats
create your fine-tuning format: QA pairs, QA pairs with CoT, summary format
curate: Using Llama as a judge to curate high quality examples.
save-as: After that you can simply save these to a format that your fine-tuning workflow requires.

Acceptance criteria

Upgrades to preprocessing service

This is an end-to-end example with Llama, which we should replicate with Gemma: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Meta_Synthetic_Data_Llama3_2_(3B).ipynb

[haroon0x]

i would like to work on this

[supreme-gg-gg]

@haroon0x feel free to get started, we don't have a design for this yet so if you can briefly describe here what features you would like to add and how to implement them before you start would be great.

In terms of testing, we're still in development and trying to figure out how to best let contributors test their code without accessing our cloud project. Perhaps just test it locally for now / use colab notebooks etc, feel free to pin us if you have questions!

[haroon0x]

Thanks!
I will draft a brief proposal and then based on your feedback i will start working on it.

[haroon0x]

This design outlines a plan to enhance the preprocessing service to support ingesting various document formats and to implement data synthesis strategies inspired by Meta's Synthetic Data Kit, adapted for Gemma.

Phased Implementation Plan

I propose a three-phased approach and a pr for each phase.

Phase 1: Foundational Document Support

Goal: Enhance the existing preprocessing service to extract text from common document formats using the parsers implemented in
synthetic data kit
Currently the preprocessing module supports only - "csv" , "json" , "jsonl" , "parquet" ,"xlsx" ,"xls" ,"txt"

• Scope:

  • PDF - pdfminer.six: For efficient and robust PDF text extraction.
  • Microsoft Word - python-docx: For extracting text from .docx files.
  • Microsoft PowerPoint - python-pptx: For extracting text from .pptx files.
  • html - beautifulsoup4 for HTML parsing

• Implementation: This will be integrated directly into the dataset_loader.py service. The extracted text will be made compatible with the existing pipeline

Phase 2: Vision Dataset Support

Goal: Enable the upload and preprocessing of local image datasets.

• Scope:
  • Common image formats (e.g., .png, .jpg).
  • Handle individual images or archives (.zip) of images.
• Implementation: This will involve extending the dataset_loader.py to recognize image formats and load them, likely in a format compatible with vision-language model training (e.g., mapping image paths to a specific feature column).

Phase 3: Data Synthesis using the Synthetic Data Kit with Gemma

Goal: Integrate the meta-llama/synthetic-data-kit directly into our workflow and leverage it to generate synthetic data. The key modification will be to configure the kit to use gemma-3 models for the data generation and curation steps, instead of the default Llama models.

• Scope: We will expose the functionality of the synthetic data kit's commands (create, curate, save-as) through our preprocessing service.
  1. Integration: The synthetic-data-kit will be added as a dependency or submodule to the project.
  2. Execution: The preprocessing service will execute the kit's command-line scripts to perform the synthesis workflow.
  3. Model Configuration: The kit will be configured to use a local inference endpoint serving a unsloth/gemma-3 model for all generation and curation tasks.
• Implementation: A new module will be created in the preprocessing service to manage the interaction with the synthetic-data-kit. This module will be responsible for:
  • Calling the kit's CLI commands in the correct sequence using shell commands.
  • Configuring the kit (likely via environment variables or config files) to point to our internal Gemma inference endpoint.
  • Retrieving the final, curated dataset from the storage.

---

Modify dataset_loader.py:
- The core logic will be updated in the _load_uploaded_dataset function.
- I will expand the file type detection to include .pdf, .docx, and .pptx.
- For each new file type, I will use the corresponding library to extract the raw text content.
- The extracted text will be passed to the datasets.load_dataset function by treating it as a plain text source. This ensures a seamless integration with the rest of the preprocessing pipeline without requiring major architectural changes.

Testing Strategy

As per the contribution guidelines, all new functionality will be tested locally.

• End-to-End Testing: I will run the preprocessing service locally to test the full flow: uploading a document, processing it, and verifying that the text is correctly extracted and converted into a dataset.

I am ready to begin work on Phase 1 as outlined above and will provide updates as I make progress. I welcome any feedback on this proposal.

[supreme-gg-gg]

@haroon0x this looks good to me, perhaps let's postpone stage 2 (vision dataset) support for now, since vision augmentation / synthesis is not supported by the synthetic dataset kit. plus it will cost us more cloud storage with large image datasets lol

For testing it requires writing to a database on our cloud, I think you can try to comment out relevant parts. You can either just test endpoints directly or run the frontend locally (https://github.com/gemma-facet/frontend-ui). You might also update the frontend for the new config or I can add it later.

[haroon0x]

okay , I'll skip vision dataset support.
For testing i will run the endpoints locally. Regarding the frontend changes i mainly work on backend but i can give it a try. Otherwise i can leave that to you.

[haroon0x]

@supreme-gg-gg

How should I integrate synthetic-data-kit as a package or use the source code?
If we were to use it as a package, then we need to use subprocess to run the CLI commands, or we can vendor the source code.
I am suggesting to use the source code since using it as a package will slow down the pipeline and adds other issues.

Is this the workflow you are thinking?

Currently

• /datasets/upload
  → File gets uploaded
  → Parsed content is stored in storage
• /datasets/process
  → Stored data gets processed
  • Formatting
  • Augmentation
    → Then gets stored

When synthetic data kit is integrated

• /datasets/upload
  → File gets uploaded
  → Parsed content is stored in storage
• /datasets/process
  → Stored data gets passed through synthetic kit
  • Create fine-tuning QA pairs
  • Curate it with judge
    → Then formatting and augmentation
    → Then gets stored

Is this how you want it to be? I want to make sure that we are on the same page.

[supreme-gg-gg]

Thanks for checking, I think it's better if we keep the current formatting only option for HF datasets. Synthetic kit should really used for user uploaded, mostly unstructured data only. The rational here I believe is that synthetic data by ingesting and curating is only valuable when your data is NOT in the standard "user" "assistant" format, and that often happens with custom data uploaded locally. HF datasets can just be formatted and perhaps augmented with existing solutions.