Segment Anything Model 2 is a foundation model for interactive instance segmentation in images and videos. It is based on transformer architecture with streaming memory for real-time video processing. SAM 2 is a generalization of the first version of SAM to the video domain, it processes video frame-by-frame and uses a memory attention module to attend to the previous memories of the target object. When SAM 2 is applied to images, the memory is empty and the model behaves like usual SAM.
Unlike the first version of Segment Anything, the frame embedding used by the SAM 2 decoder is conditioned on memories of past predictions and prompted frames (instead of being taken directly from an image decoder). Memory encoder creates "memories" of frames based on the current prediction, these "memories" are stored in model's memory bank for use in subsequent frames. The memory attention operation takes the per-frame embedding from the image encoder and conditions it on the memory bank to produce an embedding that is then passed to the mask decoder.
Step 1 Select pretrained model architecture and press the Serve button
Alternatively, you can load your custom SAM 2 checkpoint:
Step 2. Wait for the model to deploy
Usage of Segment Anything 2 as a Smart Tool for image labeling:
Video object segmentation and tracking:
Automatic image mask generation without any prompts via NN Image Labeling app:
Applying model to object in bounding box:
Fast labeling of images batch via Batched Smart Tool:
| Key | Description |
|---|---|
| Left Mouse Button | Place a positive click |
| Shift + Left Mouse Button | Place a negative click |
| Scroll Wheel | Zoom an image in and out |
| Right Mouse Button + Move Mouse |
Move an image |
| Space | Finish the current object mask |
| Shift + H | Higlight instances with random colors |
| Ctrl + H | Hide all labels |
— SmartTool selector button, switch between SmartTool apps and models
This app is based on the great work Segment Anything 2: github.
