Investigate faster SAM implementations and consider changing the default model

[GenevieveBuckley]

I know there are a whole bunch of projects that popped up after segment-anything was released, but another one we might like to look at and potentially include is FastSAM: https://github.com/CASIA-IVA-Lab/FastSAM

The model checkpoints are linked from the README here: https://github.com/CASIA-IVA-Lab/FastSAM#model-checkpoints

Cross-reference: #174

[constantinpape]

Thanks for the pointer! We're aware of that one already and it's on the list for looking into faster SAM implementations ;).
I just went ahead with the vit_t in #174 because it is so easy to integrate.

[constantinpape]

I changed the title to indicate the broader scope. As mentioned, a student will start to work on this mid of October, and @anwai98 and me are already working on better benchmarks for the models (that reflect interactive segmentation quality).
See this doc for an overview of our (preliminary) plans for evaluating the faster models.

[GenevieveBuckley]

Just came across another potentially interesting project: https://github.com/SysCV/sam-hq

I haven't looked at this yet, but it might be worth figuring out if it could be useful for microscopy datasets (which are often high resolution images with a lot of fine-grained detail).

[constantinpape]

ALso relevant:
https://pytorch.org/blog/accelerating-generative-ai/
https://github.com/pytorch-labs/segment-anything-fast

[constantinpape]

Another candidate: https://arxiv.org/pdf/2312.13789.pdf (similar to MobileSAM, but claims better performance).

[constantinpape]

I tried TinySAM (the method from my previous reply), but it does not work for our data; the bounding box prompt is somehow broken.

However, we now have MobileSAM, which also uses a vit_tiny backbone, integrated nicely. Will summarize this soon.

[constantinpape]

Just as a follow up here:

• We investigated different vit-tiny implementations, including all the ones from the links above. It turns out the original one from the mobile sam publication works best for our use-cases.
• We looked into FastSAM (using a conv backbone), but integrating their code would be a bit more difficult, so we will not do this in the short term.
• We also did much more thorough investigations regarding different models (both the original models and our finetuned ones).

Based on this I think the best strategy is:

• Setting vit_l as the default model. (It's faster than vit_h without any penalty in performance, or even slightly better performance).
• Giving a concise recommendation for the best model for a given situation (i.e. when to use our finetuned models).
• We will go ahead with this for 0.5 (goal is to get this out within the next 2 weeks), but it's also pending modelzoo integration (because we will upload our finetuned models there), see Bioimageio Model Creation #227

[constantinpape]

For the next release we will go with the strategy outlined above, so I removed this issue from the milestone.
Will leave it open though for future discussions.

[psobolewskiPhD]

For 3D, SAM2 seems promising:
https://https//arxiv.org/abs/2408.00756

[constantinpape]

Thanks @psobolewskiPhD , we are already looking into SAM2 :).