Trouble with acquiring the dataset (DMC)

[luccachiang]

Hello Ale, thanks for your awosome work and I am really into it. However, when I follow your README I encounter some problems in the section 'Downloading Data'. I tried to run your python command on my Linux server but found the following exceptions:

Processing dataset download for dmc
Saving dataset to:
/root/misc/viper_rl/viper_rl_data/datasets
dmc_dataset_aa: failed to retrieve item metadata - errors
mv: cannot stat 'dmc_dataset_aa/dmc.tar.partaa': No such file or directory
rmdir: failed to remove 'dmc_dataset_aa': No such file or directory
dmc_dataset_ab: failed to retrieve item metadata - errors
mv: cannot stat 'dmc_dataset_ab/dmc.tar.partab': No such file or directory
rmdir: failed to remove 'dmc_dataset_ab': No such file or directory
dmc_dataset_ac: failed to retrieve item metadata - errors
mv: cannot stat 'dmc_dataset_ac/dmc.tar.partac': No such file or directory
rmdir: failed to remove 'dmc_dataset_ac': No such file or directory
dmc_dataset_ad: failed to retrieve item metadata - errors
mv: cannot stat 'dmc_dataset_ad/dmc.tar.partad': No such file or directory
rmdir: failed to remove 'dmc_dataset_ad': No such file or directory
dmc_dataset_ae: failed to retrieve item metadata - errors
mv: cannot stat 'dmc_dataset_ae/dmc.tar.partae': No such file or directory
rmdir: failed to remove 'dmc_dataset_ae': No such file or directory
cat: 'dmc.tar.part*': No such file or directory
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
rm: cannot remove 'dmc.tar.part*': No such file or directory

I went through the code and the problem seems to lie in this line:
ia download dmc_dataset_$i dmc.tar.part$i
I guess the data is not correctly downloaded. Since I have no idea about what the dataset looks like or how the project is structured, I cannot figure out a way to download the dataset and upload to my server. I guess similar problems will show up if I continue to download Atari dataset as well as both the checkpoints.

Would you plz help me solve the problem or provide an alternative to get the data?

[escontra]

Interesting, what OS are you running this command on? it still work for me on linux. I recommend properly downloading internetarchive as outline here:

https://archive.org/developers/internetarchive/installation.html

[luccachiang]

Yes, I am running on Ununtu 20.04 on a remote server. The ia has been installed but somehow I cannot download the dataset. Maybe it is because that the server has blocked some websites, which I can do nothing about. Sorry if this question is a little bit stupid, are there any alternative ways to acquire the dataset?

[Bailey-24]

how to acquire the RLbench datasets and checkpoint? and how to use train videogpt and policy in RLbench?

[NonsansWD]

@luccachiang I dont know if this problem is still present or relevant for you but I only ran into this when i didnt execute one of the steps before. As long as you follow all the steps including the conda environment stuff you should be good so the question is: are you sure you applied all the steps before in the same terminal?

[JJYSTEM]

how to acquire the RLbench datasets and checkpoint? and how to use train videogpt and policy in RLbench?

Same here. Anyone has solved this issue? Thank you!