Add dataset cache / mixing support

[vwxyzjn]

This file contains the utility to transform and cache datasets with different configurations.
The main things we are looking for are:

• handle dataset mixing
• handle different tokenization functions
• cache the tokenized dataset so we don't have to re-tokenize every time
  • This is especially important when we have 405B SFT models: 32 nodes are just spending like
    5 minutes to tokenize the dataset. This translates to 32 * 5 * 8 = 1280 minutes = 21 hours of
    wasted H100 time.
  • Sometimes we also launch on places that don't have a shared cache (e.g., GCP), so we would
    download individual datasets 32 times, and wait for concatenation and tokenization (actually
    twice because the with accelerator.main_process_first() function assumes a shared cache)

I did an end-to-end test and found the fine-tuned performance to be the same.

[vwxyzjn]

Confirmed that the tokenized datasets are exactly the same for SFT / DPO

[natolambert]

LGTM some minor nits and double check maybe redundant files

[vwxyzjn]

I removed the duplicate files.