all labels replaced with "name"

[daskandalis]

After training is completed, all the labels have been replaced with 'name'. The model seems to do a validation, though I'm not sure what it means.

[eberrigan]

#1700 #1898

[daskandalis]

I guess label merging arises when the task is divided among CPUs and then recombined?

In any case, as far as I can tell, all the .slp files have the correct number of body parts. Where should I be looking for this error?

[daskandalis]

This seems like a bug. @roomrys @talmo?

[eberrigan]

Hi @daskandalis,

I apologize for my late reply.

Can you upload your files, preferably before and after merging versions of your slp files, and your training config, along with the exact commands you used here. I will replicate the issue and get back to you. Please also let us know your version and method of sleap installation.

Thank you!

Elizabeth

[daskandalis]

Just to clarify again, there is no merging on my part. This is simply sleap training on a remote cluster. sleap 1.3.3.

Install script:

module reset

module load anaconda

module load gcc
module load cuda/11.8.0
module load cudnn

conda create -n sleap-1.3.3 python=3.10 -y 
conda activate sleap-1.3.3

wget https://github.com/talmolab/sleap/archive/refs/tags/v1.3.3.tar.gz

tar -zxvf v1.3.3.tar.gz
cd sleap-1.3.3
pip install -r pypi_requirements.txt 
pip install sleap
pip install imgstore==0.2.9 python-dateutil==2.8.2 ruamel.yaml==0.16.13  
pip install pynwb==2.3.3
conda install -n sleap-1.3.3 pyqt -y
pip install segmentation_models

python -c "import sleap; sleap.system_summary()"

[eberrigan]

Hi @daskandalis,

Are you able to use the conda or PyPI package installation instructions here instead? This could be from an incorrect version of tensorflow. That is my hunch.

Also, Are you able to train the model locally? I can try training the model locally for you if you upload the .slp file and training.config here.

Best,

Elizabeth

[gitttt-1234]

Following up from @eberrigan's suggestion, you could modify your script like this:

module reset

module load anaconda

module load gcc
module load cuda/11.8.0
module load cudnn

conda create -n sleap-1.3.3 python=3.7 -y 
conda activate sleap-1.3.3

wget https://github.com/talmolab/sleap/archive/refs/tags/v1.3.3.tar.gz

tar -zxvf v1.3.3.tar.gz
cd sleap-1.3.3
pip install -e ".[pypi]"

python -c "import sleap; sleap.system_summary()"

You could also try directly installing the compiled packages instead of doing it from source:

module reset

module load anaconda

module load gcc
module load cuda/11.8.0
module load cudnn

conda create -n sleap-1.3.3 python=3.7 -y 
conda activate sleap-1.3.3
pip install "sleap[pypi]"

python -c "import sleap; sleap.system_summary()"

Note that we've changed it to use Python 3.7 which is the currently most compatible version for SLEAP v1.3.3 (the next release will explicitly support 3.10).

Let us know if that works for you!

[eberrigan]

To clarify, did you use

conda create --name sleap pip python=3.7.12 cudatoolkit=11.3 cudnn=8.2
conda activate sleap
pip install sleap[pypi]==1.3.3

Why is mamba not available on the remote? This is by far the best way to install all of the dependencies.

Please also try this docker image I have made. Let me know if you have any issues: https://gitlab.com/salk-tm/sleap-train.

[daskandalis]

Thanks for this, I've managed to get the container running (finally). However, would it be possible to build it with 1.3.3 or 1.3.4? 1.4 seems not to be ready yet, and is consistently having a wide range of issues, e.g.,

[eberrigan]

Hi @daskandalis,

Yes I will do that and get back to you shortly.

Was the container difficult to use?

Were you trying to use the brightness augmentation? Would you mind letting me know what your min and max values were as well? I will test that out.

Best,

Elizabeth

[daskandalis]

Thanks, the container was fine, there was an issue with the singularity install. Gamma (0.5-2) and brightness (0-15) both had issues.

[eberrigan]

@daskandalis I have updated the SLEAP container to use SLEAP 1.3.4.
https://gitlab.com/salk-tm/sleap-train/-/merge_requests/3

Best,

Elizabeth

[roomrys]

This looks like it could be related to our attrs >=23.1.0 issue:

• Error during Inference. ValueError: Found Multiple nodes named (name). #1918

where the Skeleton is read into an unsupported format. Then, when using the new slp project with SLEAP installed with an older attrs <23.1.0, the Skeleton is read in with all nodes named "name" (which is actually the attribute name not the attribute value).

[talmo]

Just confirming: this is an issue with newer versions of our dependencies saving skeletons in the training_job_config.json of your models that are incompatible when reading back in.

#1918 has a few solutions, including a script to fix badly formatted data, and the suggestion to upgrade to SLEAP v1.3.4 which constrains the dependencies to use the old format while we work on a more robust fix.