How to restore wandb local server data after laptop/computer/machine that hosts the server accidentally died?

[liswp]

Hi Wandb Server Team,

Great work on the local server!

A quick question - how to restore wandb local server data after laptop/computer/machine that hosts the server accidentally died?

If it helps, we did mount /vol to a local persistent directory.

Please help. Many thanks.

[nate-wandb]

Hi @liswp, glad you are enjoying W&B server! If you just copy the volume to whichever machine you would now like to run the server on, you should be able to simply attach it to the new container and data will be restored.

For example: docker run --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local will work if the volume is named wandb on the host machine.

Let me know if that helps!

[liswp]

Hi Nate,

Thanks for the message. When I do this, I got:

*** Running /etc/my_init.d/01_enable-services.sh...
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
chgrp: changing group of '/vol/env': Operation not permitted
chgrp: changing group of '/vol/env/kid.txt': Operation not permitted
chgrp: changing group of '/vol/env/overrides.env': Operation not permitted
chgrp: changing group of '/vol/env/key.pem': Operation not permitted
chgrp: changing group of '/vol/env/minio-config.json': Operation not permitted
chgrp: changing group of '/vol/env/settings.json': Operation not permitted
chgrp: changing group of '/vol/env/jwks.json': Operation not permitted
chgrp: changing group of '/vol/env/minio_creds.env': Operation not permitted
chgrp: changing group of '/vol/env/session.txt': Operation not permitted
chgrp: changing group of '/vol/env/users.htpasswd': Operation not permitted
*** Loading settings...
*** Booting runit daemon...
*** Runit started as PID 45
*** Setting up mysql database...
*** Starting wandb servers...
*** Migrating database...

What do you think? Many thanks.

[liswp]

Hi Nate,

Just one more piece of observation - even if I don't copy the volume and use the existing volume, I get automatically redirected to local:8080/signup page, and I cannot login to my old account.

This issue has been consistent for us throughout the past few months such that we have to create a brand new local server, manually sync every run and format our panels from scratch everytime the linux ubuntu machine we have died or ran out of battery.

Please help. Many thanks.

[nate-wandb]

HI @liswp, sorry for the delay on this. This: chgrp: changing group of '/vol/env/minio_creds.env': Operation not permitted looks like a permissions issue. Could you try using chmod on the volume to make sure you have the ability to make changes on the directory?

Also, here are the official Docker docs on backing up the volume and migrating it to a new container.

Let me know if this helps

[liswp]

Hi Nate,

Yes, I tried chmod -R 777 the entire /vol, didn't work. Please help. Thanks.

[nate-wandb]

@liswp it looks like you will need to create a backup of the volume first docker run --rm --volumes-from wandb -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /wandb

Is there any way to access the original machine and run this?

Also, are you able to run a wandb server on the new machine with a new clean volume? I just wanted to make sure that Docker has the right permissions on the new machine.

[nate-wandb]

Hi @liswp, I just wanted to follow up on this and see if this was still an issue or if you were able to get your server migrated?

[Jaswers]

sudo chown -R 999:root wandb

[andrenatal]

Thanks @Jaswers ! 2 hours of chatgpt back and forth changing permissions everywhere, and your one liner solved it.

My situation was moving from a local disk to a folder mounted over nfs on a synology nas.