Stop trying to guess the length of webdataset datasets

[gabrielilharco]

No description provided.

[gabrielilharco]

@rom1504 here's the PR we were discussing, in case we want to stop trying to guess the length of wds datasets

[usuyama]

I like this change! (in some experiments, I wanted to override train_num_samples and shorten the epoch - this change will solve the issue, too)

With webdataset, when setting train_num_samples as smaller than whole dataset, do we get different set of samples per epoch?

[gabrielilharco]

@usuyama with the current code, you can set --train-num-samples to a smaller number, and if you use --dataset-resampled, you'll always get random samples from the entire pool on each "epoch".