Issues with mixed-size animal tracking

[Jiaming-He]

Hi everyone,

I’m using SLEAP (v1.5.1) for multi-animal tracking in a homecage setup with a Dam and prewean pups, and I have a few questions:

Animals moving out of frame / becoming invisible
– How should we label frames where the rat moves under objects (e.g., under the food area) and becomes partly or completely occluded? Should we skip these frames, or make a random guess and label the invisible animal under the food area?

Different-sized animals in the same video
– How should we set the centroid receptive field size when animals differ greatly in size (Dam vs pups)?
– Is it better to train one model for all animals or separate models per size group?

• Aside from the size group, we also have different age groups. Is it possible to generalize the model accross all age groups? I referred to this thread (Multi-age Three Mouse Tracking #1625), but our animals change a lot in size and appearance.

Animal changing apparent size (moving close to camera)
– Is there good way to tackle this problem aside from setting the receptive field as large as possible?

Overfitting after recursive labeling
– After several rounds of labeling and retraining, the multi-animal top-down model overfits more: training loss drops but validation loss stays high, and tracking performance worsens. Any suggestions for avoiding this?

Thanks for any advice or experience with similar homecage setups!

[alicup29]

Hey @Jiaming-He !
To answer your questions:

Animals moving out of frame / becoming invisible: How should we label frames where the rat moves under objects (e.g., under the food area) and becomes partly or completely occluded? Should we skip these frames, or make a random guess and label the invisible animal under the food area?

When an animal moves partly out of frame / becomes "invisible", you don't have to skip these frames, as you can right-click on the occluded nodes in that frame in order to mark them as "not visible" when they leave frame. However, if the animal is completely not there, then there is no need to label. This also prevents making over-confident predictions on occluded parts. It is better to just label visible nodes and mark non-visible as occluded (set visibility to False). See more about proof-reading here.

Different-sized animals in the same video:
– How should we set the centroid receptive field size when animals differ greatly in size (Dam vs pups)?
– Is it better to train one model for all animals or separate models per size group?

For when animals differ greatly in size, the centroid receptive field size should be set larger to accommodate a size as large as your biggest animal in order to give the model enough context. To increase receptive field, it is important to control both max stride (increase to increase receptive field) and input scaling (decrease to increase receptive field). For example, relying on solely aggressive scaling reduces the resolution and can degrade performance. Additionally, scale augmentations would be better since the animals all vary in size.

In your case, it is better to just train one model for all animals instead of separating per size group. Separate models are usually only needed if one group is extremely underrepresented or looks basically like a different species (very different shape/appearance across ages).

Animal changing apparent size (moving close to camera)
– Is there good way to tackle this problem aside from setting the receptive field as large as possible?

Here, we wouldn't recommend top-down here since using top-down would detect the centroid & crop the animal before sending the crop to the centered instance model and the big animals are going to be cropped. Instead try using bottom-up and label representative instances with all sizes while making sure your receptive field size is large enough so that the model knows what it is looking at without being so large that the features per animal are not discernible. Additionally, you could try lens correction to assist in visibility. For example, a pinhole model correction can help restore the original of shape of your equipment/mice.

However ideally, for more consistent data, it may be worth considering change to experimental design such as moving the camera further up to accommodate for these situations.

Overfitting after recursive labeling
– After several rounds of labeling and retraining, the multi-animal top-down model overfits more: training loss drops but validation loss stays high, and tracking performance worsens. Any suggestions for avoiding this?

Adding augmentations (i.e. scale, rotation, etc.) would help and enabling early stopping might help (if not already enabled).

Sorry about the late response... Let us know if you have further questions!

Best,
Amick

[Jiaming-He]

Dear Amick,

Thank you for the clear and helpful guidance! The tips on training a single model for entire homecage and all age groups, labeling occlusions and using scale augmentations are very useful. We're changing our experimental design for better video qualities and thus better inference performance. Really appreciate your support which gives us a nice path forward.

Best,
Jiaming