Cannot get predictions when running inference on NVIDIA GPU

[abailey20]

Hi all,

I am trying to run inference after training a top-down model on videos of a single animal. Training seems to go smoothly and validation predictions look good but if I try to get it to predict on a random number of frames I get "Inference ran on number of frames. Instances were predicted on 0 frames (number frames with no instances found).

This is what's in the terminal:

sleap-nn-track --data_path E:/P02/SLEAP_PoseEstimation/Z_arboreus/Daylight_Conditions/labels.v001.slp --video_index 5 --frames 11604 --model_paths E:\P02\SLEAP_PoseEstimation\Z_arboreus\Daylight_Conditions\models\251014_092855.centroid.n=51 --model_paths E:\P02\SLEAP_PoseEstimation\Z_arboreus\Daylight_Conditions\models\251014_113154.centered_instance.n=51 -o E:/P02/SLEAP_PoseEstimation/Z_arboreus/Daylight_Conditions\predictions\labels.v001.slp.251014_125336.predictions.slp --batch_size 4

2025-10-14 12:53:42 | INFO | sleap_nn.predict:run_inference:319 | Started inference at: 2025-10-14 12:53:42.762975
2025-10-14 12:53:42 | INFO | sleap_nn.predict:run_inference:335 | Using device: cuda
Predicting... ---------------------------------------- 100% 1/1 ETA: 0:00:00 Elapsed: 0:00:00 ?
2025-10-14 12:53:44 | INFO | sleap_nn.predict:run_inference:453 | Finished inference at: 2025-10-14 12:53:44.296171
2025-10-14 12:53:44 | INFO | sleap_nn.predict:run_inference:454 | Total runtime: 1.533196210861206 secs
2025-10-14 12:53:45 | INFO | sleap_nn.predict:run_inference:465 | Predictions output path: E:/P02/SLEAP_PoseEstimation/Z_arboreus/Daylight_Conditions\predictions\labels.v001.slp.251014_125336.predictions.slp
2025-10-14 12:53:45 | INFO | sleap_nn.predict:run_inference:466 | Saved file at: 2025-10-14 12:53:45.303320


Process return code: 0

I've checked that the gpu is visible and run diagnostics. I can't see anything wrong:

(sleap) C:\Users\Andrea>python -c "import torch; print(torch.cuda.is_available())"
True

(sleap) C:\Users\Andrea>uv run sleap-diagnostic

==========SYSTEM==========

utc:                    2025-10-14 11:47:05.202705
python:                 3.12.11
system:                 Windows, AMD64, 11, 10.0.26100
path:                   C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\cv2\../../x64/vc17/bin;C:\Users\Andrea\Anaconda\envs\sleap\Scripts;C:\Users\Andrea\Anaconda\envs\sleap;C:\Users\Andrea\Anaconda\envs\sleap;C:\Users\Andrea\Anaconda\envs\sleap\Library\mingw-w64\bin;C:\Users\Andrea\Anaconda\envs\sleap\Library\usr\bin;C:\Users\Andrea\Anaconda\envs\sleap\Library\bin;C:\Users\Andrea\Anaconda\envs\sleap\Scripts;C:\Users\Andrea\Anaconda\envs\sleap\bin;C:\Users\Andrea\Anaconda\condabin;C:\Users\Andrea\Anaconda;C:\Users\Andrea\Anaconda\Library\mingw-w64\bin;C:\Users\Andrea\Anaconda\Library\usr\bin;C:\Users\Andrea\Anaconda\Library\bin;C:\Users\Andrea\Anaconda\Scripts;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\libnvvp;C:\Program Files\Basler\pylon 7\Runtime\x64;C:\Program Files\Basler\pylon 7\Runtime\Win32;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\WINDOWS\System32\OpenSSH;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\MATLAB\R2023a\bin;C:\Program Files\NVIDIA Corporation\Nsight Compute 2024.3.0;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\lib\x64;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\include;C:\Users\Andrea\.local\bin;C:\Users\Andrea\AppData\Local\Microsoft\WindowsApps;C:\Users\Andrea\AppData\Local\Progr象㋐΃ﱀ֑ﰴ֑;C:\Program Files\PsychoPy;C:\Program Files\PsychoPy\DLLs;C:\ffmpeg\ffmpeg-master-latest-win64-gpl\bin

==========IMPORTS==========

sleap import:                   True
sleap path:                     C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\sleap\__init__.py
sleap version:                  1.5.1
pyside2 import:                 False
pyside6 import:                 True
pyside path:                    C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\PySide6\__init__.py
call to C:\Users\Andrea\Anaconda\envs\sleap\python.exe failed
C:\Users\Andrea\Anaconda\envs\sleap\python.exe: can't open file 'C:\\Users\\Andrea\\Anaconda\\envs\\sleap\\Scripts\\sleap-diagnostic': [Errno 2] No such file or directory

cv2 import:                     True

==========GIT==========

call to git failed
fatal: not a git repository (or any of the parent directories): .git

call to git failed
fatal: not a git repository (or any of the parent directories): .git


==========PYTORCH==========

pytorch import:                 True
pytorch version:                        2.8.0+cu128
pytorch path:                   C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\torch\__init__.py
cuda available:                 True
cuda device count:                      1
cuda device 0 name:                     NVIDIA GeForce RTX 3070 Ti

==========CONDA==========

# packages in environment at C:\Users\Andrea\Anaconda\envs\sleap:
#
# Name                    Version                   Build  Channel
aiohappyeyeballs          2.6.1                    pypi_0    pypi
aiohttp                   3.12.15                  pypi_0    pypi
aiosignal                 1.4.0                    pypi_0    pypi
albucore                  0.0.24                   pypi_0    pypi
albumentations            2.0.8                    pypi_0    pypi
annotated-types           0.7.0                    pypi_0    pypi
antlr4-python3-runtime    4.9.3                    pypi_0    pypi
anyio                     4.11.0                   pypi_0    pypi
argon2-cffi               25.1.0                   pypi_0    pypi
argon2-cffi-bindings      25.1.0                   pypi_0    pypi
arrow                     1.3.0                    pypi_0    pypi
asttokens                 3.0.0                    pypi_0    pypi
async-lru                 2.0.5                    pypi_0    pypi
attrs                     25.3.0                   pypi_0    pypi
av                        15.1.0                   pypi_0    pypi
babel                     2.17.0                   pypi_0    pypi
beautifulsoup4            4.14.2                   pypi_0    pypi
bleach                    6.2.0                    pypi_0    pypi
bzip2                     1.0.8                h2bbff1b_6
ca-certificates           2025.9.9             haa95532_0
cattrs                    25.2.0                   pypi_0    pypi
certifi                   2025.10.5                pypi_0    pypi
cffi                      2.0.0                    pypi_0    pypi
charset-normalizer        3.4.3                    pypi_0    pypi
click                     8.3.0                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
comm                      0.2.3                    pypi_0    pypi
contourpy                 1.3.3                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
debugpy                   1.8.17                   pypi_0    pypi
decorator                 5.2.1                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
executing                 2.2.1                    pypi_0    pypi
expat                     2.7.1                h8ddb27b_0
fastjsonschema            2.21.2                   pypi_0    pypi
filelock                  3.19.1                   pypi_0    pypi
fonttools                 4.60.1                   pypi_0    pypi
fqdn                      1.5.1                    pypi_0    pypi
frozenlist                1.8.0                    pypi_0    pypi
fsspec                    2025.9.0                 pypi_0    pypi
gitdb                     4.0.12                   pypi_0    pypi
gitpython                 3.1.45                   pypi_0    pypi
h11                       0.16.0                   pypi_0    pypi
h5py                      3.14.0                   pypi_0    pypi
hdmf                      4.1.0                    pypi_0    pypi
httpcore                  1.0.9                    pypi_0    pypi
httpx                     0.28.1                   pypi_0    pypi
hydra-core                1.3.2                    pypi_0    pypi
idna                      3.10                     pypi_0    pypi
imageio                   2.37.0                   pypi_0    pypi
imageio-ffmpeg            0.6.0                    pypi_0    pypi
imgstore                  0.3.7                    pypi_0    pypi
ipykernel                 6.30.1                   pypi_0    pypi
ipython                   9.6.0                    pypi_0    pypi
ipython-pygments-lexers   1.1.1                    pypi_0    pypi
ipywidgets                8.1.7                    pypi_0    pypi
isoduration               20.11.0                  pypi_0    pypi
jedi                      0.19.2                   pypi_0    pypi
jinja2                    3.1.6                    pypi_0    pypi
joblib                    1.5.2                    pypi_0    pypi
jsmin                     3.0.1                    pypi_0    pypi
json5                     0.12.1                   pypi_0    pypi
jsonpickle                4.1.1                    pypi_0    pypi
jsonpointer               3.0.0                    pypi_0    pypi
jsonschema                4.25.1                   pypi_0    pypi
jsonschema-specifications 2025.9.1                 pypi_0    pypi
jupyter                   1.1.1                    pypi_0    pypi
jupyter-client            8.6.3                    pypi_0    pypi
jupyter-console           6.6.3                    pypi_0    pypi
jupyter-core              5.8.1                    pypi_0    pypi
jupyter-events            0.12.0                   pypi_0    pypi
jupyter-lsp               2.3.0                    pypi_0    pypi
jupyter-server            2.17.0                   pypi_0    pypi
jupyter-server-terminals  0.5.3                    pypi_0    pypi
jupyterlab                4.4.9                    pypi_0    pypi
jupyterlab-pygments       0.3.0                    pypi_0    pypi
jupyterlab-server         2.27.3                   pypi_0    pypi
jupyterlab-widgets        3.0.15                   pypi_0    pypi
kiwisolver                1.4.9                    pypi_0    pypi
kornia                    0.8.1                    pypi_0    pypi
kornia-rs                 0.1.9                    pypi_0    pypi
lark                      1.3.0                    pypi_0    pypi
lazy-loader               0.4                      pypi_0    pypi
libffi                    3.4.4                hd77b12b_1
libzlib                   1.3.1                h02ab6af_0
lightning                 2.5.5                    pypi_0    pypi
lightning-utilities       0.15.2                   pypi_0    pypi
loguru                    0.7.3                    pypi_0    pypi
markdown-it-py            4.0.0                    pypi_0    pypi
markupsafe                3.0.3                    pypi_0    pypi
mat                       1.0.2                    pypi_0    pypi
matplotlib                3.10.6                   pypi_0    pypi
matplotlib-inline         0.1.7                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mistune                   3.1.4                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
multidict                 6.6.4                    pypi_0    pypi
mysql-connector-python    9.4.0                    pypi_0    pypi
nbclient                  0.10.2                   pypi_0    pypi
nbconvert                 7.16.6                   pypi_0    pypi
nbformat                  5.10.4                   pypi_0    pypi
ndx-pose                  0.2.2                    pypi_0    pypi
nest-asyncio              1.6.0                    pypi_0    pypi
networkx                  3.5                      pypi_0    pypi
nixio                     1.5.4                    pypi_0    pypi
notebook                  7.4.7                    pypi_0    pypi
notebook-shim             0.2.4                    pypi_0    pypi
numpy                     2.2.6                    pypi_0    pypi
omegaconf                 2.3.0                    pypi_0    pypi
opencv-python             4.12.0.88                pypi_0    pypi
opencv-python-headless    4.12.0.88                pypi_0    pypi
openssl                   3.0.18               h543e019_0
packaging                 25.0                     pypi_0    pypi
pandas                    2.3.3                    pypi_0    pypi
pandocfilters             1.5.1                    pypi_0    pypi
parso                     0.8.5                    pypi_0    pypi
pillow                    11.3.0                   pypi_0    pypi
pip                       25.2               pyhc872135_0
platformdirs              4.4.0                    pypi_0    pypi
prometheus-client         0.23.1                   pypi_0    pypi
prompt-toolkit            3.0.52                   pypi_0    pypi
propcache                 0.4.0                    pypi_0    pypi
protobuf                  6.32.1                   pypi_0    pypi
psutil                    7.1.0                    pypi_0    pypi
pure-eval                 0.2.3                    pypi_0    pypi
pycparser                 2.23                     pypi_0    pypi
pydantic                  2.11.10                  pypi_0    pypi
pydantic-core             2.33.2                   pypi_0    pypi
pygments                  2.19.2                   pypi_0    pypi
pykalman                  0.10.2                   pypi_0    pypi
pynwb                     3.1.2                    pypi_0    pypi
pyparsing                 3.2.5                    pypi_0    pypi
pyside6                   6.9.3                    pypi_0    pypi
pyside6-addons            6.9.3                    pypi_0    pypi
pyside6-essentials        6.9.3                    pypi_0    pypi
python                    3.12.11              h716150d_0
python-dateutil           2.9.0.post0              pypi_0    pypi
python-json-logger        4.0.0                    pypi_0    pypi
python-rapidjson          1.21                     pypi_0    pypi
pytorch-lightning         2.5.5                    pypi_0    pypi
pytz                      2025.2                   pypi_0    pypi
pywin32                   311                      pypi_0    pypi
pywinpty                  3.0.2                    pypi_0    pypi
pyyaml                    6.0.3                    pypi_0    pypi
pyzmq                     27.1.0                   pypi_0    pypi
qtpy                      2.4.3                    pypi_0    pypi
referencing               0.36.2                   pypi_0    pypi
requests                  2.32.5                   pypi_0    pypi
rfc3339-validator         0.1.4                    pypi_0    pypi
rfc3986-validator         0.1.1                    pypi_0    pypi
rfc3987-syntax            1.1.0                    pypi_0    pypi
rich                      14.1.0                   pypi_0    pypi
rpds-py                   0.27.1                   pypi_0    pypi
ruamel-yaml               0.18.15                  pypi_0    pypi
ruamel-yaml-clib          0.2.14                   pypi_0    pypi
scikit-base               0.12.6                   pypi_0    pypi
scikit-image              0.25.2                   pypi_0    pypi
scikit-learn              1.7.2                    pypi_0    pypi
scipy                     1.16.2                   pypi_0    pypi
seaborn                   0.13.2                   pypi_0    pypi
send2trash                1.8.3                    pypi_0    pypi
sentry-sdk                2.39.0                   pypi_0    pypi
setuptools                78.1.1          py312haa95532_0
shiboken6                 6.9.3                    pypi_0    pypi
simplejson                3.20.2                   pypi_0    pypi
simsimd                   6.5.3                    pypi_0    pypi
six                       1.17.0                   pypi_0    pypi
sleap                     1.5.1                    pypi_0    pypi
sleap-io                  0.5.5                    pypi_0    pypi
sleap-nn                  0.0.2                    pypi_0    pypi
smmap                     5.0.2                    pypi_0    pypi
sniffio                   1.3.1                    pypi_0    pypi
soupsieve                 2.8                      pypi_0    pypi
sqlite                    3.50.2               hda9a48d_1
stack-data                0.6.3                    pypi_0    pypi
stringzilla               4.1.0                    pypi_0    pypi
sympy                     1.14.0                   pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
terminado                 0.18.1                   pypi_0    pypi
threadpoolctl             3.6.0                    pypi_0    pypi
tifffile                  2025.10.4                pypi_0    pypi
tinycss2                  1.4.0                    pypi_0    pypi
tk                        8.6.15               hf199647_0
torch                     2.8.0+cu128              pypi_0    pypi
torchmetrics              1.8.2                    pypi_0    pypi
torchvision               0.23.0+cu128             pypi_0    pypi
tornado                   6.5.2                    pypi_0    pypi
tqdm                      4.67.1                   pypi_0    pypi
traitlets                 5.14.3                   pypi_0    pypi
types-python-dateutil     2.9.0.20250822           pypi_0    pypi
typing-extensions         4.15.0                   pypi_0    pypi
typing-inspection         0.4.2                    pypi_0    pypi
tzdata                    2025.2                   pypi_0    pypi
tzlocal                   5.3.1                    pypi_0    pypi
ucrt                      10.0.22621.0         haa95532_0
uri-template              1.3.0                    pypi_0    pypi
urllib3                   2.5.0                    pypi_0    pypi
vc                        14.3                h2df5915_10
vc14_runtime              14.44.35208         h4927774_10
vs2015_runtime            14.44.35208         ha6b5a95_10
wandb                     0.22.1                   pypi_0    pypi
wcwidth                   0.2.14                   pypi_0    pypi
webcolors                 24.11.1                  pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.8.0                    pypi_0    pypi
wheel                     0.45.1          py312haa95532_0
widgetsnbextension        4.0.14                   pypi_0    pypi
win32-setctime            1.2.0                    pypi_0    pypi
xz                        5.6.4                h4754444_1
yarl                      1.21.0                   pypi_0    pypi
zlib                      1.3.1                h02ab6af_0


==========PIP==========

aiohappyeyeballs==2.6.1
aiohttp==3.12.15
aiosignal==1.4.0
albucore==0.0.24
albumentations==2.0.8
annotated-types==0.7.0
antlr4-python3-runtime==4.9.3
anyio==4.11.0
argon2-cffi==25.1.0
argon2-cffi-bindings==25.1.0
arrow==1.3.0
asttokens==3.0.0
async-lru==2.0.5
attrs==25.3.0
av==15.1.0
babel==2.17.0
beautifulsoup4==4.14.2
bleach==6.2.0
cattrs==25.2.0
certifi==2025.10.5
cffi==2.0.0
charset-normalizer==3.4.3
click==8.3.0
colorama==0.4.6
comm==0.2.3
contourpy==1.3.3
cycler==0.12.1
debugpy==1.8.17
decorator==5.2.1
defusedxml==0.7.1
executing==2.2.1
fastjsonschema==2.21.2
filelock==3.19.1
fonttools==4.60.1
fqdn==1.5.1
frozenlist==1.8.0
fsspec==2025.9.0
gitdb==4.0.12
GitPython==3.1.45
h11==0.16.0
h5py==3.14.0
hdmf==4.1.0
httpcore==1.0.9
httpx==0.28.1
hydra-core==1.3.2
idna==3.10
imageio==2.37.0
imageio-ffmpeg==0.6.0
imgstore==0.3.7
ipykernel==6.30.1
ipython==9.6.0
ipython_pygments_lexers==1.1.1
ipywidgets==8.1.7
isoduration==20.11.0
jedi==0.19.2
Jinja2==3.1.6
joblib==1.5.2
jsmin==3.0.1
json5==0.12.1
jsonpickle==4.1.1
jsonpointer==3.0.0
jsonschema==4.25.1
jsonschema-specifications==2025.9.1
jupyter==1.1.1
jupyter-console==6.6.3
jupyter-events==0.12.0
jupyter-lsp==2.3.0
jupyter_client==8.6.3
jupyter_core==5.8.1
jupyter_server==2.17.0
jupyter_server_terminals==0.5.3
jupyterlab==4.4.9
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.3
jupyterlab_widgets==3.0.15
kiwisolver==1.4.9
kornia==0.8.1
kornia_rs==0.1.9
lark==1.3.0
lazy_loader==0.4
lightning==2.5.5
lightning-utilities==0.15.2
loguru==0.7.3
markdown-it-py==4.0.0
MarkupSafe==3.0.3
mat==1.0.2
matplotlib==3.10.6
matplotlib-inline==0.1.7
mdurl==0.1.2
mistune==3.1.4
mpmath==1.3.0
multidict==6.6.4
mysql-connector-python==9.4.0
nbclient==0.10.2
nbconvert==7.16.6
nbformat==5.10.4
ndx-pose==0.2.2
nest-asyncio==1.6.0
networkx==3.5
nixio==1.5.4
notebook==7.4.7
notebook_shim==0.2.4
numpy==2.2.6
omegaconf==2.3.0
opencv-python==4.12.0.88
opencv-python-headless==4.12.0.88
packaging==25.0
pandas==2.3.3
pandocfilters==1.5.1
parso==0.8.5
pillow==11.3.0
platformdirs==4.4.0
prometheus_client==0.23.1
prompt_toolkit==3.0.52
propcache==0.4.0
protobuf==6.32.1
psutil==7.1.0
pure_eval==0.2.3
pycparser==2.23
pydantic==2.11.10
pydantic_core==2.33.2
Pygments==2.19.2
pykalman==0.10.2
pynwb==3.1.2
pyparsing==3.2.5
PySide6==6.9.3
PySide6_Addons==6.9.3
PySide6_Essentials==6.9.3
python-dateutil==2.9.0.post0
python-json-logger==4.0.0
python-rapidjson==1.21
pytorch-lightning==2.5.5
pytz==2025.2
pywin32==311
pywinpty==3.0.2
PyYAML==6.0.3
pyzmq==27.1.0
QtPy==2.4.3
referencing==0.36.2
requests==2.32.5
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rfc3987-syntax==1.1.0
rich==14.1.0
rpds-py==0.27.1
ruamel.yaml==0.18.15
ruamel.yaml.clib==0.2.14
scikit-base==0.12.6
scikit-image==0.25.2
scikit-learn==1.7.2
scipy==1.16.2
seaborn==0.13.2
Send2Trash==1.8.3
sentry-sdk==2.39.0
setuptools==78.1.1
shiboken6==6.9.3
simplejson==3.20.2
simsimd==6.5.3
six==1.17.0
sleap==1.5.1
sleap-io==0.5.5
sleap-nn==0.0.2
smmap==5.0.2
sniffio==1.3.1
soupsieve==2.8
stack-data==0.6.3
stringzilla==4.1.0
sympy==1.14.0
tabulate==0.9.0
terminado==0.18.1
threadpoolctl==3.6.0
tifffile==2025.10.4
tinycss2==1.4.0
torch==2.8.0+cu128
torchmetrics==1.8.2
torchvision==0.23.0+cu128
tornado==6.5.2
tqdm==4.67.1
traitlets==5.14.3
types-python-dateutil==2.9.0.20250822
typing-inspection==0.4.2
typing_extensions==4.15.0
tzdata==2025.2
tzlocal==5.3.1
uri-template==1.3.0
urllib3==2.5.0
wandb==0.22.1
wcwidth==0.2.14
webcolors==24.11.1
webencodings==0.5.1
websocket-client==1.8.0
wheel==0.45.1
widgetsnbextension==4.0.14
win32_setctime==1.2.0
yarl==1.21.0


==========NVIDIA==========

Tue Oct 14 12:47:13 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.94                 Driver Version: 560.94         CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3070 Ti   WDDM  |   00000000:09:00.0  On |                  N/A |
|  0%   46C    P8             27W /  310W |     946MiB /   8192MiB |      3%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      3376    C+G   ...nt.CBS_cw5n1h2txyewy\SearchHost.exe      N/A      |
|    0   N/A  N/A      4236    C+G   ...on\140.0.3485.94\msedgewebview2.exe      N/A      |
|    0   N/A  N/A      6380    C+G   ...oogle\Chrome\Application\chrome.exe      N/A      |
|    0   N/A  N/A      6708    C+G   ...les\Microsoft OneDrive\OneDrive.exe      N/A      |
|    0   N/A  N/A      9624    C+G   ...cw5n1h2txyewy\CrossDeviceResume.exe      N/A      |
|    0   N/A  N/A      9800    C+G   C:\Windows\explorer.exe                     N/A      |
|    0   N/A  N/A     10016    C+G   ...siveControlPanel\SystemSettings.exe      N/A      |
|    0   N/A  N/A     10148    C+G   ...on\140.0.3485.94\msedgewebview2.exe      N/A      |
|    0   N/A  N/A     10768    C+G   ...ekyb3d8bbwe\PhoneExperienceHost.exe      N/A      |
|    0   N/A  N/A     10924    C+G   ...1.0_x64__8wekyb3d8bbwe\Video.UI.exe      N/A      |
|    0   N/A  N/A     11004    C+G   ...s\TeamViewer\TeamViewer_Desktop.exe      N/A      |
|    0   N/A  N/A     11616    C+G   C:\Windows\System32\dwm.exe                 N/A      |
|    0   N/A  N/A     13128    C+G   ...444_x64__8wekyb3d8bbwe\ms-teams.exe      N/A      |
|    0   N/A  N/A     13356    C+G   ...5n1h2txyewy\ShellExperienceHost.exe      N/A      |
|    0   N/A  N/A     13612    C+G   ...on\140.0.3485.94\msedgewebview2.exe      N/A      |
|    0   N/A  N/A     14668    C+G   ...444_x64__8wekyb3d8bbwe\ms-teams.exe      N/A      |
|    0   N/A  N/A     16784    C+G   ...2txyewy\StartMenuExperienceHost.exe      N/A      |
|    0   N/A  N/A     17496    C+G   C:\Windows\explorer.exe                     N/A      |
|    0   N/A  N/A     17736    C+G   ...s\System32\ApplicationFrameHost.exe      N/A      |
|    0   N/A  N/A     17780    C+G   C:\Windows\System32\ShellHost.exe           N/A      |
|    0   N/A  N/A     18144    C+G   C:\Windows\explorer.exe                     N/A      |
+-----------------------------------------------------------------------------------------+

Only thing I can think of is if the labels need to be in the same folder as the videos.

Any help/advice is appreciated!

Thanks,
A

[eberrigan]

Hello @abailey20,

It looks like you are predicting on one frame right now. Can you try doing this from the GUI or changing your frames to a list of frames?

Also, do you know if at the end of training you were able to get metrics from your model, implying it was able to get predictions at the end of training?

Thank you!

Elizabeth

https://docs.sleap.ai/v1.5.1/api/cli/#sleap.cli.track

@click.option(
    "--frames",
    type=str,
    default="",
    help="List of frames to predict when running on a video. Can be specified as a "
    "comma separated list (e.g. 1,2,3) or a range separated by hyphen "
    "(e.g., 1-3, for 1,2,3). If not provided, defaults to predicting on entire "
    "video.",
)

[abailey20]

Hi @eberrigan,

I only selected to run inference on current frame as an example of the output of such an attempt. Right now I'm working from the GUI. Is it better to run inference from the CLI?

I cannot get model evaluation metrics, any attempt to give me this output:
Traceback (most recent call last):

File "C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\sleap\gui\app.py", line 1610, in _show_metrics_dialog
   self._child_windows["metrics"] = MetricsTableDialog(self.state["filename"])
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\sleap\gui\dialogs\metrics.py", line 77, in __init__
   self._update_cfgs()
 File "C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\sleap\gui\dialogs\metrics.py", line 92, in _update_cfgs
   self.table_model.items = cfgs
   ^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\sleap\gui\dataviews.py", line 105, in items
   item_data = self.item_to_data(obj, item)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\sleap\gui\dialogs\metrics.py", line 217, in item_to_data
   metrics = cfg.metrics
             ^^^^^^^^^^^
 File "C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\sleap\gui\learning\configs.py", line 104, in metrics
   return self._get_metrics("val")
          ^^^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\sleap\gui\learning\configs.py", line 187, in _get_metrics
   with np.load(metrics_path, allow_pickle=True) as data:
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\numpy\lib\_npyio_impl.py", line 451, in load
   fid = stack.enter_context(open(os.fspath(file), "rb"))
                                  ^^^^^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not NoneType

Thanks,
A

[abailey20]

Hi @eberrigan

Here is an example of the parameters for training I am running.

Despite in my first post SLEAP detected one cuda device. When I try to actually train a model it detects 0. There is also some warnings about not having dataloggers configured, I am not sure how to rectify that but to my knowledge it doesn't explain why my model can't predict anything. Below I've attached some images of the training set up menu to show why I've chosen the parameters for the receptive field. Input scaling for the centroid model might be too high but again I can only see how this might make training slower but not how it affects trainability. I'm training on 171 labelled frames. The skeleton I am using only has 4 nodes: nose, both eyes and the bum so there is no real candidate for an anchor node for the centroid model my questions:

1. Why when I train do the logs show that I don't have any cuda device available? What has it got to do with `torch.set_float32_matmul_precision('medium' | 'high')?
2. Should I configure a data logger?
3. Why does my model have so many parameters? What is this number determined by?
4. Are the max stride and input scaling parameters I've chosen for each model appropriate? (I've attached pictures below)
5. Would having an anchor for the centroid model make a difference even if there are no real features on the back of the animal that I can use to identify a point reliably?

I will post the training log as soo as I have it. But would appreciate any clarity on the points above.

Thank you for any and all help!

Best,
A

Start training centroid...
['sleap-nn-train', '--config-name', '251017_101005_config', '--config-dir', 'C:\\Users\\Andrea\\AppData\\Local\\Temp\\tmpachlfoet']
Input config:

data_config:
  train_labels_path:
  - C:/Users/Andrea/SLEAP_PoseEstimation/Z_arboreus/Daylight_Conditions/labels.v001.slp
  val_labels_path: null
  validation_fraction: 0.1
  test_file_path: null
  provider: LabelsReader
  user_instances_only: true
  data_pipeline_fw: torch_dataset
  cache_img_path: null
  use_existing_imgs: false
  delete_cache_imgs_after_training: true
  preprocessing:
    ensure_rgb: false
    ensure_grayscale: false
    max_height: null
    max_width: null
    scale: 0.7
    crop_size: null
    min_crop_size: 100
  use_augmentations_train: false
  augmentation_config:
    intensity:
      uniform_noise_min: 0.0
      uniform_noise_max: 1.0
      uniform_noise_p: 0.0
      gaussian_noise_mean: 5.0
      gaussian_noise_std: 1.0
      gaussian_noise_p: 0.0
      contrast_min: 0.5
      contrast_max: 2.0
      contrast_p: 0.0
      brightness_min: 0.0
      brightness_max: 2.0
      brightness_p: 0.0
    geometric:
      rotation_min: -180.0
      rotation_max: 180.0
      scale_min: 0.9
      scale_max: 1.1
      translate_width: 0.0
      translate_height: 0.0
      affine_p: 1.0
      erase_scale_min: 0.0001
      erase_scale_max: 0.01
      erase_ratio_min: 1.0
      erase_ratio_max: 1.0
      erase_p: 0.0
      mixup_lambda_min: 0.01
      mixup_lambda_max: 0.05
      mixup_p: 0.0
  skeletons: null
model_config:
  init_weights: default
  pretrained_backbone_weights: null
  pretrained_head_weights: null
  backbone_config:
    unet:
      in_channels: 1
      kernel_size: 3
      filters: 16
      filters_rate: 2.0
      max_stride: 64
      stem_stride: null
      middle_block: true
      up_interpolate: true
      stacks: 1
      convs_per_block: 2
      output_stride: 2
    convnext: null
    swint: null
  head_configs:
    single_instance: null
    centroid:
      confmaps:
        anchor_part: null
        sigma: 3.5
        output_stride: 2
    centered_instance: null
    bottomup: null
    multi_class_bottomup: null
    multi_class_topdown: null
  total_params: null
trainer_config:
  train_data_loader:
    batch_size: 4
    shuffle: false
    num_workers: 0
  val_data_loader:
    batch_size: 4
    shuffle: false
    num_workers: 0
  model_ckpt:
    save_top_k: 1
    save_last: false
  trainer_devices: null
  trainer_device_indices: null
  trainer_accelerator: auto
  profiler: null
  trainer_strategy: auto
  enable_progress_bar: true
  min_train_steps_per_epoch: 200
  train_steps_per_epoch: null
  visualize_preds_during_training: true
  keep_viz: false
  max_epochs: 200
  seed: null
  use_wandb: false
  save_ckpt: true
  ckpt_dir: C:/Users/Andrea/SLEAP_PoseEstimation/Z_arboreus/Daylight_Conditions/models
  run_name: None_251017_101004.centroid.n=171
  resume_ckpt_path: null
  wandb:
    entity: ''
    project: ''
    name: ''
    save_viz_imgs_wandb: false
    api_key: ''
    wandb_mode: null
    prv_runid: ''
    group: ''
    current_run_id: null
  optimizer_name: Adam
  optimizer:
    lr: 0.0001
    amsgrad: false
  lr_scheduler: null
  early_stopping:
    min_delta: 1.0e-08
    patience: 20
    stop_training_on_plateau: true
  online_hard_keypoint_mining:
    online_mining: false
    hard_to_easy_ratio: 2.0
    min_hard_keypoints: 2
    max_hard_keypoints: null
    loss_scale: 5.0
  zmq:
    controller_port: 9000
    controller_polling_timeout: 10
    publish_port: 9001
name: ''
description: ''
sleap_nn_version: 0.0.2
filename: ''

2025-10-17 10:10:13 | INFO | sleap_nn.train:run_training:27 | Started training at: 2025-10-17 10:10:13.168916
2025-10-17 10:10:13 | INFO | sleap_nn.training.model_trainer:_setup_train_val_labels:216 | Creating train-val split...
2025-10-17 10:10:17 | INFO | sleap_nn.training.model_trainer:_setup_train_val_labels:261 | # Train Labeled frames: 153
2025-10-17 10:10:17 | INFO | sleap_nn.training.model_trainer:_setup_train_val_labels:262 | # Val Labeled frames: 17
2025-10-17 10:10:17 | INFO | sleap_nn.training.model_trainer:setup_config:512 | Setting up config...
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:_verify_model_input_channels:417 | Updating backbone in_channels to 3 based on the input image channels.
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:train:849 | Setting up for training...
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:_setup_model_ckpt_dir:575 | Setting up model ckpt dir: `C:/Users/Andrea/SLEAP_PoseEstimation/Z_arboreus/Daylight_Conditions/models/None_251017_101004.centroid.n=171`...
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:train:864 | Setting up visualization train and val datasets...
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:train:868 | Setting up Trainer...
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:_setup_loggers_callbacks:647 | Setting up callbacks and loggers...
2025-10-17 10:10:18 | INFO | sleap_nn.training.callbacks:__init__:228 | Training controller subscribed to: tcp://127.0.0.1:9000 (topic: )
2025-10-17 10:10:18 | INFO | sleap_nn.training.callbacks:__init__:287 | ProgressReporterZMQ publishing to tcp://127.0.0.1:9001 for ''
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:train:897 | Trainer devices: auto
INFO: GPU available: True (cuda), used: True
INFO:lightning.pytorch.utilities.rank_zero:GPU available: True (cuda), used: True
INFO: TPU available: False, using: 0 TPU cores
INFO:lightning.pytorch.utilities.rank_zero:TPU available: False, using: 0 TPU cores
INFO: HPU available: False, using: 0 HPUs
INFO:lightning.pytorch.utilities.rank_zero:HPU available: False, using: 0 HPUs
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:train:950 | Training on 1 device(s)
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:train:951 | Training on cuda:0 accelerator
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:train:955 | Setting up lightning module for centroid model...
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:train:959 | Backbone model: UNet(
  (encoders): ModuleList(
    (0): Encoder(
      (encoder_stack): ModuleList(
        (0): SimpleConvBlock(
          (blocks): Sequential(
            (stack0_enc0_conv0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc0_act0_relu): ReLU()
            (stack0_enc0_conv1): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc0_act1_relu): ReLU()
          )
        )
        (1): SimpleConvBlock(
          (blocks): Sequential(
            (stack0_enc1_pool): MaxPool2dWithSamePadding(kernel_size=2, stride=2, padding=same, dilation=1, ceil_mode=False)
            (stack0_enc1_conv0): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc1_act0_relu): ReLU()
            (stack0_enc1_conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc1_act1_relu): ReLU()
          )
        )
        (2): SimpleConvBlock(
          (blocks): Sequential(
            (stack0_enc2_pool): MaxPool2dWithSamePadding(kernel_size=2, stride=2, padding=same, dilation=1, ceil_mode=False)
            (stack0_enc2_conv0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc2_act0_relu): ReLU()
            (stack0_enc2_conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc2_act1_relu): ReLU()
          )
        )
        (3): SimpleConvBlock(
          (blocks): Sequential(
            (stack0_enc3_pool): MaxPool2dWithSamePadding(kernel_size=2, stride=2, padding=same, dilation=1, ceil_mode=False)
            (stack0_enc3_conv0): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc3_act0_relu): ReLU()
            (stack0_enc3_conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc3_act1_relu): ReLU()
          )
        )
        (4): SimpleConvBlock(
          (blocks): Sequential(
            (stack0_enc4_pool): MaxPool2dWithSamePadding(kernel_size=2, stride=2, padding=same, dilation=1, ceil_mode=False)
            (stack0_enc4_conv0): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc4_act0_relu): ReLU()
            (stack0_enc4_conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc4_act1_relu): ReLU()
          )
        )
        (5): SimpleConvBlock(
          (blocks): Sequential(
            (stack0_enc5_pool): MaxPool2dWithSamePadding(kernel_size=2, stride=2, padding=same, dilation=1, ceil_mode=False)
            (stack0_enc5_conv0): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc5_act0_relu): ReLU()
            (stack0_enc5_conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_enc5_act1_relu): ReLU()
          )
        )
        (6): Sequential(
          (stack0_enc6_last_pool): MaxPool2dWithSamePadding(kernel_size=2, stride=2, padding=same, dilation=1, ceil_mode=False)
        )
      )
    )
  )
  (decoders): ModuleList(
    (0): Decoder(
      (decoder_stack): ModuleList(
        (0): SimpleUpsamplingBlock(
          (blocks): Sequential(
            (stack0_dec0_s64_to_s32_interp_bilinear): Upsample(scale_factor=2.0, mode='bilinear')
            (stack0_dec0_s64_to_s32_refine_conv0): Conv2d(1536, 512, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_dec0_s64_to_s32_refine_conv0_act_relu): ReLU()
            (stack0_dec0_s64_to_s32_refine_conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_dec0_s64_to_s32_refine_conv1_act_relu): ReLU()
          )
        )
        (1): SimpleUpsamplingBlock(
          (blocks): Sequential(
            (stack0_dec1_s32_to_s16_interp_bilinear): Upsample(scale_factor=2.0, mode='bilinear')
            (stack0_dec1_s32_to_s16_refine_conv0): Conv2d(768, 256, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_dec1_s32_to_s16_refine_conv0_act_relu): ReLU()
            (stack0_dec1_s32_to_s16_refine_conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_dec1_s32_to_s16_refine_conv1_act_relu): ReLU()
          )
        )
        (2): SimpleUpsamplingBlock(
          (blocks): Sequential(
            (stack0_dec2_s16_to_s8_interp_bilinear): Upsample(scale_factor=2.0, mode='bilinear')
            (stack0_dec2_s16_to_s8_refine_conv0): Conv2d(384, 128, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_dec2_s16_to_s8_refine_conv0_act_relu): ReLU()
            (stack0_dec2_s16_to_s8_refine_conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_dec2_s16_to_s8_refine_conv1_act_relu): ReLU()
          )
        )
        (3): SimpleUpsamplingBlock(
          (blocks): Sequential(
            (stack0_dec3_s8_to_s4_interp_bilinear): Upsample(scale_factor=2.0, mode='bilinear')
            (stack0_dec3_s8_to_s4_refine_conv0): Conv2d(192, 64, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_dec3_s8_to_s4_refine_conv0_act_relu): ReLU()
            (stack0_dec3_s8_to_s4_refine_conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_dec3_s8_to_s4_refine_conv1_act_relu): ReLU()
          )
        )
        (4): SimpleUpsamplingBlock(
          (blocks): Sequential(
            (stack0_dec4_s4_to_s2_interp_bilinear): Upsample(scale_factor=2.0, mode='bilinear')
            (stack0_dec4_s4_to_s2_refine_conv0): Conv2d(96, 32, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_dec4_s4_to_s2_refine_conv0_act_relu): ReLU()
            (stack0_dec4_s4_to_s2_refine_conv1): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=same)
            (stack0_dec4_s4_to_s2_refine_conv1_act_relu): ReLU()
          )
        )
      )
    )
  )
  (middle_blocks): ModuleList(
    (0): SimpleConvBlock(
      (blocks): Sequential(
        (stack0_enc7_middle_expand_conv0): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=same)
        (stack0_enc7_middle_expand_act0_relu): ReLU()
      )
    )
    (1): SimpleConvBlock(
      (blocks): Sequential(
        (stack0_enc8_middle_contract_conv0): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=same)
        (stack0_enc8_middle_contract_act0_relu): ReLU()
      )
    )
  )
)
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:train:960 | Head model: ModuleList(
  (0): Sequential(
    (CentroidConfmapsHead): Sequential(
      (0): Conv2d(32, 1, kernel_size=(1, 1), stride=(1, 1), padding=same)
      (1): Identity()
    )
  )
)
2025-10-17 10:10:18 | INFO | sleap_nn.training.model_trainer:train:962 | Total model parameters: 31449201
2025-10-17 10:10:19 | INFO | sleap_nn.training.model_trainer:train:967 | Input image shape: torch.Size([1, 3, 1536, 2688])
2025-10-17 10:10:19 | INFO | sleap_nn.training.model_trainer:train:1021 | Finished trainer set up. [1.6s]
2025-10-17 10:10:19 | INFO | sleap_nn.training.model_trainer:train:1024 | Starting training loop...
INFO: You are using a CUDA device ('NVIDIA GeForce RTX 3070 Ti') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
INFO:lightning.pytorch.utilities.rank_zero:You are using a CUDA device ('NVIDIA GeForce RTX 3070 Ti') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\lightning\pytorch\callbacks\model_checkpoint.py:751: Checkpoint directory C:\Users\Andrea\SLEAP_PoseEstimation\Z_arboreus\Daylight_Conditions\models\None_251017_101004.centroid.n=171 exists and is not empty.
INFO: LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
INFO:lightning.pytorch.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
INFO:
  | Name               | Type         | Params | Mode
------------------------------------------------------------
0 | model              | Model        | 31.4 M | train
1 | centroid_inf_layer | CentroidCrop | 0      | train
------------------------------------------------------------
31.4 M    Trainable params
0         Non-trainable params
31.4 M    Total params
125.797   Total estimated model params size (MB)
101       Modules in train mode
0         Modules in eval mode
INFO:lightning.pytorch.callbacks.model_summary:
  | Name               | Type         | Params | Mode
------------------------------------------------------------
0 | model              | Model        | 31.4 M | train
1 | centroid_inf_layer | CentroidCrop | 0      | train
------------------------------------------------------------
31.4 M    Trainable params
0         Non-trainable params
31.4 M    Total params
125.797   Total estimated model params size (MB)
101       Modules in train mode
0         Modules in eval mode
Sanity Checking: |                                                                                               | 0/? [00:00<?, ?it/s]C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\lightning\pytorch\trainer\connectors\data_connector.py:433: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=15` in the `DataLoader` to improve performance.
Sanity Checking DataLoader 0:   0%|                                                                              | 0/2 [00:00<?, ?it/s]C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\lightning\pytorch\core\module.py:520: You called `self.log('learning_rate', ..., logger=True)` but have no logger configured. You can enable one by doing `Trainer(logger=ALogger(...))`
C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\lightning\pytorch\core\module.py:520: You called `self.log('val_loss', ..., logger=True)` but have no logger configured. You can enable one by doing `Trainer(logger=ALogger(...))`
Sanity Checking DataLoader 0: 100%|██████████████████████████████████████████████████████████████████████| 2/2 [00:21<00:00,  0.09it/s]C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\lightning\pytorch\core\module.py:520: You called `self.log('val_time', ..., logger=True)` but have no logger configured. You can enable one by doing `Trainer(logger=ALogger(...))`
C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\lightning\pytorch\trainer\connectors\data_connector.py:433: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=15` in the `DataLoader` to improve performance.
Epoch 0:   0%|                                                                                                 | 0/200 [00:00<?, ?it/s]C:\Users\Andrea\Anaconda\envs\sleap\Lib\site-packages\lightning\pytorch\core\module.py:520: You called `self.log('train_loss', ..., logger=True)` but have no logger configured. You can enable one by doing `Trainer(logger=ALogger(...))`
Epoch 0:   2%|█▏                                                            | 4/200 [02:31<2:03:39,  0.03it/s, train_loss_step=3.79e-5]