ValueError: Columns defined in both source dataset and generation config

[sn0rkmaiden]

I am trying to create a large dataset with the script provided in the README.md. Here is my script:

hf jobs uv run --namespace $NAMESPACE \ 
-s HF_TOKEN=$HF_TOKEN \
 https://github.com/huggingface/aisheets/raw/refs/heads/main/scripts/extend_dataset/with_inference_client.py \
 snork-maiden/ToySnacks snork-maiden/snacksbench \
--config https://huggingface.co/datasets/snork-maiden/ToySnacks/raw/main/config.yml \
--num-rows 100

I get this error about overlapping columns and I don't really understand what's the problem. I just want to create a larger dataset.
Could someone please explain how to solve this? Thanks in advance.

[frascuchon]

Hi @sn0rkmaiden,

As far as I understand, what you want to do is to augment the original size by adding new rows, right? The provided script does not support this kind of data augmentation. Only adding new columns is supported now. So I think this is the reason for your column overlapping error.

Sorry for the misunderstanding.

[sn0rkmaiden]

Hi, @frascuchon! Thank you for your time and your answer!

So I would like to make sure: it is currently not possible to generate large datasets? Or is there a different way to do it? Thanks!

[frascuchon]

It depends on your use case. This is similar to the topic targeted in this discussion: If you have a small dataset, you can set up a column and define your prompt using the whole data collection as part of the prompt. This could get new cells similar to the provided ones.

cc @Amelie-V

[sn0rkmaiden]

@frascuchon Sorry that I'm asking again: I tried approach that you've suggested but keep getting this error:

ValueError: You must provide an api_key to work with nebius API or log in with `hf auth login`.

I'm pretty sure that I've logged in and I've also run export HF_TOKEN=_____. How can I specify the nebius api key? Thanks again!

[frascuchon]

Where do you receive this response? From the script or when defining configuration in the UI?

[sn0rkmaiden]

Where do you receive this response? From the script or when defining configuration in the UI?

I'm getting it in the "jobs" section logs on Huggingface website

[frascuchon]

Hmmm....Not sure what's happening. Since you're using a namespace to run the job, the org running the job does not have enough privileges to execute inference client calls?

Did you try without a namespace? Or another inference provider?