Finetuning

Is there any plans to release scripts for finetuning the model?
Also you did such a great work! Thank you very much!

Information on fine tuning would be great.

+1, I'd love to be able to fine tune to improve performance on extremely difficult tiny-object tasks, for example segmenting cars in this image:

this thread is referenced as the answer for similar questions, but I don't think there is an answer here for transfer learning?

Look forward to finetuning

I would love to be able to fine tune the model for specific datasets as well.

Do we wait for Meta to provide a training/fine-tuning script? Or should the open source hivemind write it?

Has anyone tried the idea of what may be called "point prompt engineering"? i.e. training a model that learns how to put positive prompt points and negative prompt points, such that these points prompt SAM to select target objects from a custom dataset.
Or we can just summarize strategies and best practices in terms of placing positive and negative prompt points/prompt boxes, similar to how GPT/DALLE users summarize the best ways to write prompts.
Wonder if this could be one way to fine-tune the model when only a limited amount of annotations are available. Happy to discuss more if anyone want to work together to try it out.

+1, Looking forward to fine-tuning the SAM model on the custom dataset.:)

After some messing around I have gotten preliminary fine-tuning to work on my fork. The code is still super messy and early, but perhaps people can find use in it. The biggest thing I figured out is that you have to break up the Sam model into its components in order for there to be a gradient path for fine-tuning.

After some messing around I have gotten preliminary fine-tuning to work on my fork. The code is still super messy and early, but perhaps people can find use in it. The biggest thing I figured out is that you have to break up the Sam model into its components in order for there to be a gradient path for fine-tuning.

Could you please recommend the minimum hardware configuration for fine-tuning the SAM? eg. 4090 x 4?

I can get the smallest pre-trained model (vit_b) with a batch size of 1 in <5GB of GPU memory, but I think fine tuning with those settings would take forever.

I have access to a 4 x A100 /w 80G if you want me to test something.

hi @hu-po ,
Thanks for sharing the fine-tuning code very much. Would it be possible for you to give guidance on how to prepare the customized dataset (e.g., data format and folder structures)?

hi @hu-po ,
Thanks for sharing the fine-tuning code very much. Would it be possible for you to give guidance on how to prepare the customized dataset (e.g., data format and folder structures)?

Thank me when I get it to work 😭 this is more complicated than anticipated.

+1, interested in fine-tuning it for coral reef images.

+1 interested in fine-tuning it for cracking on roads.

+1 🙌

+1 interested in fine-tuning!

+1, I'd like to do some vehicle detection on low quality images!

+1 interested in fine tunning prompt encoder or mask decoder!

+1! I would be interested in fine-tuning the model for medical image analysis

I'm curious that is it possible to point out an unknown object have not been learned (like anomaly detection) by text prompt if I fine-tune with custom data.

+1!

CC: @ericmintun @nikhilaravi

@hu-po
hi, nice work for sharing finetune script , is "FragmentDataset" the datasets that released by official datasets https://segment-anything.com/dataset/index.html

I have a finetune starter code for COCO format data with some basic functionality at this repo. Hope it would help!

Hey, we wrote a blog post outlining some of the key steps to fine tune SAM using the mask decoder, particularly linking which functions from SAM to use to pre/post process the data.

@hu-po
hi, nice work for sharing finetune script , is "FragmentDataset" the datasets that released by official datasets https://segment-anything.com/dataset/index.html

No, it's a custom dataset for x-ray data of scroll fragments: https://www.kaggle.com/competitions/vesuvius-challenge-ink-detection/data

I know those guys!

I am a student and I am also looking forward to the release of the fine-tuning to complete my academic paper, and I would be very grateful if it is released

+1 thanks for finetuning

@codybum @750563720 @ZhuJD-China I tried this tutorial @alex-encord posted, seems to be working well. Would be easier with a Colab Notebook though.

hi @ulsha ,
Where is the colab?

^ @alex000kim Check this.

It's intersting if SAM has been finetuned on autonomouse driving dataset.

I'm trying to do a fine-tuning on the semantic segmentation task without bbox, but only labels. I resized the image and label to be 1024x1024, and use the following codes:
    # Assign data
    img, label, o_img_size, n_img_size = batch['image'], batch['label'], batch['original_image_size'], batch['image_size']

    # Map to variables
    img = Variable(img)
    label = Variable(label)

    # Get embeddings
    with torch.no_grad():
        image_embedding = self.model.image_encoder(img)
        sparse_embeddings, dense_embeddings = self.model.prompt_encoder(
            points=None,boxes=None,masks=label.float())

    # Get predictions
    low_res_masks, iou_predictions = self.model.mask_decoder(
        image_embeddings=image_embedding,
        image_pe=self.model.prompt_encoder.get_dense_pe(),
        sparse_prompt_embeddings=sparse_embeddings,
        dense_prompt_embeddings=dense_embeddings,
        multimask_output=True,
        )
    upscaled_masks = self.model.postprocess_masks(low_res_masks, n_img_size, o_img_size)

`
I modify the decoder head like this:
    net = sam_model_registry[args.sam_model_type](checkpoint=args.ckpt)
    d = net.mask_decoder
    net.mask_decoder = MaskDecoder(transformer_dim=d.transformer_dim, transformer=d.transformer, num_multimask_outputs=args.num_classes)

    return net

However, I got errors from the decoder embeddings fusions:
    from segment_anything/modeling/mask_decoder.py", line 127, in predict_masks
    src = src + dense_prompt_embeddings
    RuntimeError: The size of tensor a (64) must match the size of tensor b (256) at non-singleton dimension 3

In this case, my sparse_embeddings has shape [B,0,256], and dense_embeddings has shape [B, 256, 256, 256]. Anyone has ideas?

I'm trying to do a fine-tuning on the semantic segmentation task without bbox, but only labels. I resized the image and label to be 1024x1024, and use the following codes:
    # Assign data
    img, label, o_img_size, n_img_size = batch['image'], batch['label'], batch['original_image_size'], batch['image_size']

    # Map to variables
    img = Variable(img)
    label = Variable(label)

    # Get embeddings
    with torch.no_grad():
        image_embedding = self.model.image_encoder(img)
        sparse_embeddings, dense_embeddings = self.model.prompt_encoder(
            points=None,boxes=None,masks=label.float())

    # Get predictions
    low_res_masks, iou_predictions = self.model.mask_decoder(
        image_embeddings=image_embedding,
        image_pe=self.model.prompt_encoder.get_dense_pe(),
        sparse_prompt_embeddings=sparse_embeddings,
        dense_prompt_embeddings=dense_embeddings,
        multimask_output=True,
        )
    upscaled_masks = self.model.postprocess_masks(low_res_masks, n_img_size, o_img_size)

`
I modify the decoder head like this:
    net = sam_model_registry[args.sam_model_type](checkpoint=args.ckpt)
    d = net.mask_decoder
    net.mask_decoder = MaskDecoder(transformer_dim=d.transformer_dim, transformer=d.transformer, num_multimask_outputs=args.num_classes)

    return net

However, I got errors from the decoder embeddings fusions:
    from segment_anything/modeling/mask_decoder.py", line 127, in predict_masks
    src = src + dense_prompt_embeddings
    RuntimeError: The size of tensor a (64) must match the size of tensor b (256) at non-singleton dimension 3

In this case, my sparse_embeddings has shape [B,0,256], and dense_embeddings has shape [B, 256, 256, 256]. Anyone has ideas?

I opened a new issue #277

我正在尝试在没有 bbox 的情况下对语义分割任务进行微调，但只有标签。我将图像和标签调整为 1024x1024，并使用以下代码：
    # Assign data
    img, label, o_img_size, n_img_size = batch['image'], batch['label'], batch['original_image_size'], batch['image_size']

    # Map to variables
    img = Variable(img)
    label = Variable(label)

    # Get embeddings
    with torch.no_grad():
        image_embedding = self.model.image_encoder(img)
        sparse_embeddings, dense_embeddings = self.model.prompt_encoder(
            points=None,boxes=None,masks=label.float())

    # Get predictions
    low_res_masks, iou_predictions = self.model.mask_decoder(
        image_embeddings=image_embedding,
        image_pe=self.model.prompt_encoder.get_dense_pe(),
        sparse_prompt_embeddings=sparse_embeddings,
        dense_prompt_embeddings=dense_embeddings,
        multimask_output=True,
        )
    upscaled_masks = self.model.postprocess_masks(low_res_masks, n_img_size, o_img_size)

`
我这样修改解码器头：
    net = sam_model_registry[args.sam_model_type](checkpoint=args.ckpt)
    d = net.mask_decoder
    net.mask_decoder = MaskDecoder(transformer_dim=d.transformer_dim, transformer=d.transformer, num_multimask_outputs=args.num_classes)

    return net

但是，我从解码器嵌入融合中得到了错误：
    from segment_anything/modeling/mask_decoder.py", line 127, in predict_masks
    src = src + dense_prompt_embeddings
    RuntimeError: The size of tensor a (64) must match the size of tensor b (256) at non-singleton dimension 3

在本例中，我的sparse_embeddings形状为 [B,0,256]，并且dense_embeddings形状为 [B, 256, 256, 256]。有人有想法吗？

I use the same way to fine_tune sam, label as mask prompt input, and then get predict output.I compute  loss(ground_truth, output) but the loss no decline ,remain invariable，can you give me some advice?I would appreciate it！

Made a repo for fine-tuning SAM.
A PoC to see if fine-tuning SAM using bounding boxes as prompts would increase the IoU or improve the quality of the masks in general. One can use a COCO format dataset to fine-tune SAM for a specific task where SAM does not perform well (e.g., segmenting text on documents) and then use that model with interactive prompts just like SAM.
https://github.com/luca-medeiros/lightning-sam

I'm trying to do a fine-tuning on the semantic segmentation task without bbox, but only labels. I resized the image and label to be 1024x1024, and use the following codes:
    # Assign data
    img, label, o_img_size, n_img_size = batch['image'], batch['label'], batch['original_image_size'], batch['image_size']

    # Map to variables
    img = Variable(img)
    label = Variable(label)

    # Get embeddings
    with torch.no_grad():
        image_embedding = self.model.image_encoder(img)
        sparse_embeddings, dense_embeddings = self.model.prompt_encoder(
            points=None,boxes=None,masks=label.float())

    # Get predictions
    low_res_masks, iou_predictions = self.model.mask_decoder(
        image_embeddings=image_embedding,
        image_pe=self.model.prompt_encoder.get_dense_pe(),
        sparse_prompt_embeddings=sparse_embeddings,
        dense_prompt_embeddings=dense_embeddings,
        multimask_output=True,
        )
    upscaled_masks = self.model.postprocess_masks(low_res_masks, n_img_size, o_img_size)

`
I modify the decoder head like this:
    net = sam_model_registry[args.sam_model_type](checkpoint=args.ckpt)
    d = net.mask_decoder
    net.mask_decoder = MaskDecoder(transformer_dim=d.transformer_dim, transformer=d.transformer, num_multimask_outputs=args.num_classes)

    return net

However, I got errors from the decoder embeddings fusions:
    from segment_anything/modeling/mask_decoder.py", line 127, in predict_masks
    src = src + dense_prompt_embeddings
    RuntimeError: The size of tensor a (64) must match the size of tensor b (256) at non-singleton dimension 3

In this case, my sparse_embeddings has shape [B,0,256], and dense_embeddings has shape [B, 256, 256, 256]. Anyone has ideas?

I'm able to solve the problem by not passing any arguments to the prompt_encoder, where I mistakenly input the labels into masks variable. However, I keep getting out of memory problems as the training moves on, reducing the number of batche size will only postpone the errors in later epochs, why is this happenning?

following

+1

+1

Guys you can just hit the subscribe button on the right to get notifications for just this thread without annoying everybody who already subscribed :angry:

Hi @kdcd  @codybum @austinmw @penguingiraffe2 @jindameias @BenSpex @hu-po @openvino-book @dlod-openvino
@shakesBeardZ @AMInnovationTeam @maskani-moh @ariannaravera @harry-s-grewal @javiermcebrian @francescodisalvo05 @travishsu @imandrealombardo @Kenneth-X @bhpfelix @alex-encord @750563720 @ZhuJD-China @RCfun @Tobyzai @MaxMatti
We provide a step-by-step tutorial on fine-tuning SAM on 2D and 3D medical image datasets. It requires less than 10G GPU memory.
Hope that it could be useful.
https://github.com/bowang-lab/MedSAM#model-training-video-tutorial

also found this one using adaptors for efficient fine tuning: https://github.com/tianrun-chen/SAM-Adapter-PyTorch

thank you all for sharing

@kdcd @codybum @austinmw @penguingiraffe2 @jindameias @BenSpex @hu-po @openvino-book @dlod-openvino
@shakesBeardZ @AMInnovationTeam @maskani-moh @ariannaravera @harry-s-grewal @javiermcebrian @francescodisalvo05 @travishsu @imandrealombardo @Kenneth-X @bhpfelix @alex-encord @750563720 @ZhuJD-China @RCfun @Tobyzai @MaxMatti @dshlai @dorbodwolf
Hey guys, 👋
Built a quick app to finetune SAM (https://app.instalabel.ai) on custom data (for those who don't wanna go through the hassle of changing and debugging the code 😅). The app allows users to directly train a fine-tuned SAM model by following a few simple steps:

Log in or create an account
Upload around 15-20 samples from your dataset to the platform
Upload individual ground truth segmentation masks for each sample
Click 'Train annotator'. This will finetune SAM to the data you sent. May need to wait for 15-20 minutes for it to train. ⏳
Go to the 'Annotation' tab and upload any other images you want to try the finetuned SAM on. You can test it out with a few bounding box prompts. Otherwise, you can save the finetuned '.pth' file yourself to use in your projects by pressing the 'Download Trained Checkpoint' button in the annotation tab!
The app is still a little buggy and I'm working to fix them (like refreshing the page after uploading all ground truth masks before you can press the 'Train annotator' button). However, it should still be largely usable! 👍

Hoping this platform can help developers have a simple UI to get finetuned checkpoints for SAM and annotate their data with a fine-tuned Segment Anything Model 🎯
Hope it helps! 🙌

@aadilmehdis The model worked pretty well when I was able to figure out the UI. Was not very intuitive to work through though(took a while to train model, and encountered some bugs). Is the finetuning code open source? Would be interested to know what losses/hyperparameters you used.

Following up on this, I found the method section of @JunMa11 's manuscript https://arxiv.org/pdf/2304.12306.pdf useful to get an idea of how to fine-tune SAM.

Hi @kdcd @codybum @austinmw @penguingiraffe2 @jindameias @BenSpex @hu-po @openvino-book @dlod-openvino @shakesBeardZ @AMInnovationTeam @maskani-moh @ariannaravera @harry-s-grewal @javiermcebrian @francescodisalvo05 @travishsu @imandrealombardo @Kenneth-X @bhpfelix @alex-encord @750563720 @ZhuJD-China @RCfun @Tobyzai @MaxMatti
We provide a step-by-step tutorial on fine-tuning SAM on 2D and 3D medical image datasets. It requires less than 10G GPU memory. Hope that it could be useful.
https://github.com/bowang-lab/MedSAM#model-training-video-tutorial

@kdcd @codybum @austinmw @penguingiraffe2 @jindameias @BenSpex @hu-po @openvino-book @dlod-openvino @shakesBeardZ @AMInnovationTeam @maskani-moh @ariannaravera @harry-s-grewal @javiermcebrian @francescodisalvo05 @travishsu @imandrealombardo @Kenneth-X @bhpfelix @alex-encord @750563720 @ZhuJD-China @RCfun @Tobyzai @MaxMatti @dshlai @dorbodwolf
Hey guys, 👋
Built a quick app to finetune SAM (https://app.instalabel.ai) on custom data (for those who don't wanna go through the hassle of changing and debugging the code 😅). The app allows users to directly train a fine-tuned SAM model by following a few simple steps:

Log in or create an account
Upload around 15-20 samples from your dataset to the platform
Upload individual ground truth segmentation masks for each sample
Click 'Train annotator'. This will finetune SAM to the data you sent. May need to wait for 15-20 minutes for it to train. ⏳
Go to the 'Annotation' tab and upload any other images you want to try the finetuned SAM on. You can test it out with a few bounding box prompts. Otherwise, you can save the finetuned '.pth' file yourself to use in your projects by pressing the 'Download Trained Checkpoint' button in the annotation tab!
The app is still a little buggy and I'm working to fix them (like refreshing the page after uploading all ground truth masks before you can press the 'Train annotator' button). However, it should still be largely usable! 👍

Hoping this platform can help developers have a simple UI to get finetuned checkpoints for SAM and annotate their data with a fine-tuned Segment Anything Model 🎯
Hope it helps! 🙌
📢 Update Note:

Most of the code used were from open-source repos already available earlier in this thread! Just whipped up a platform really quick to make it possible for users to directly download fine-tuned SAM checkpoints without having to deal with the hassle of setup.
Also received a few issues about how users can directly upload their individual ground truth segmentation masks for each sample. I used LabelMe for this. An example of a groundtruth mask is shown here.

📢 Update Note:

Thank you for the positive feedback and interest in our platform! I really appreciate the support. However, I want to address a concern regarding scalability and multiple users. While the platform initially worked well with a few users, the sudden increase in the number of users has exposed some limitations. I am actively working on scaling up the platform to accommodate the more users.

📢 Update Note:
Hello everyone! :wave:
I've scaled up the platform to handle a larger number of users and made sure that the platform is robust enough for a smoother experience. I've also added a feature that sends you an email notification when a job has started and when it has finished fine-tuning. 🔔 It might take a couple of minutes for the email to come through, so please hang tight!
Furthermore, in order to manage GPU load more efficiently, I am currently resizing all images to a uniform 256x256 resolution. I am working to support varied dimensions and will incorporate this feature soon.
I'd really appreciate it if you could take it for a spin! Any feedback or issues, big or small, can be addressed directly to me at<EMAIL_ADDRESS>Your input is extremely invaluable as I continue to improve the platform!
Thanks

+1! I would be interested in fine-tuning the model for medical image analysis

I would love to apply it for medical image segmentation.

+1, interested in fine-tuning it for some domain special datasets.

@EnchiridionHero Hi,
I want to know how to upload the image on the https://app.instalabel.ai,
Should I upload both the original and annotated images at the same time? I have read the operating instructions, but I still don't understand.

Thanks everyone for the guideline on fine tune the SAM decoder.
I have used this approach to train on a small set of masked data and it worked like charm. However, after I doubled the training size, the fine-tuned models performed worse due to the dice loss + focal loss did not decrease during the training.
Anyone can give an advice in this case? Really appreciate!

Looks like HF is working on it https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SAM/README.md

To reproduce the training (and eventunally fintetuning) you could also use:
https://github.com/UX-Decoder/Semantic-SAM

To reproduce the training (and eventunally fintetuning) you could also use:
https://github.com/UX-Decoder/Semantic-SAM

Thanks for the sharing.
From the code in the link, what you mean is in order to further improve the fine-tuned SAM we need to retrain the model on SAM dataset with the architecture close to the original SAM right? Currently, is there anyway to fine tune the encoder in the original SAM?

The blog that @alex-encord shared works fine if you have 1 bounding box per image. But in reality you will have multiple bounding box in an image. Below code fails if you have multiple bounding box
with torch.no_grad():
      sparse_embeddings, dense_embeddings = sam_model.prompt_encoder(
          points=None,
          boxes=box_torch,
          masks=None,
      )

Am I missing something?

Can you explain how fail @nahidalam? What is the error or stacktrace?

@nhhung1810 @alex-encord this is the error I get
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 1 but got size 19 for tensor number 1 in the list.

I get this error because I have 19 bounding boxes in my first image
Full stack trace
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[38], line 27
     24   box_torch = torch.as_tensor(box, dtype=torch.float, device=device)
     25   box_torch = box_torch[None, :]
---> 27   sparse_embeddings, dense_embeddings = sam_model.prompt_encoder(
     28       points=None,
     29       boxes=box_torch,
     30       masks=None,
     31   )
     32 low_res_masks, iou_predictions = sam_model.mask_decoder(
     33   image_embeddings=image_embedding,
     34   image_pe=sam_model.prompt_encoder.get_dense_pe(),
   (...)
     37   multimask_output=False,
     38 )
     40 upscaled_masks = sam_model.postprocess_masks(low_res_masks, input_size, original_image_size).to(device)

File ~/SAM-experiments/env/lib/python3.8/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File ~/SAM-experiments/env/lib/python3.8/site-packages/segment_anything/modeling/prompt_encoder.py:159, in PromptEncoder.forward(self, points, boxes, masks)
    157 if boxes is not None:
    158     box_embeddings = self._embed_boxes(boxes)
--> 159     sparse_embeddings = torch.cat([sparse_embeddings, box_embeddings], dim=1)
    161 if masks is not None:
    162     dense_embeddings = self._embed_masks(masks)

RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 1 but got size 19 for tensor number 1 in the list.

Hey @nahidalam
Maybe the problem is box_torch = box_torch[None, :]. I assume that your input box_torch have the batch-size as 19. But then you run box_torch = box_torch[None, :] => which make the batch-size become (as box_torch.shape = [1, ...]).
Check this line and this line, you will see that your bs become 1.
I suggest that you remove that  box_torch = box_torch[None, :] line and run again. If there is some problem, try putting a breakpoint and use debug to see the shape change, it may help a lot.

The blog that @alex-encord shared works fine if you have 1 bounding box per image. But in reality you will have multiple bounding box in an image. Below code fails if you have multiple bounding box
with torch.no_grad():
      sparse_embeddings, dense_embeddings = sam_model.prompt_encoder(
          points=None,
          boxes=box_torch,
          masks=None,
      )

Am I missing something?

Hi @nahidalam
I have same question as you, i.e., reality image will have multiple object bounding box. Should we put the multiple box together? Thanks in advance.

Question: Can we fine-tune the decoder to infer without prompt ?
Possible Solution (Just thinking out loud): Fine-Tuning without Prompt or fixed static prompt >> Hence infer without prompt using fine-tuned model?

Every tutorial I saw has one thing in common: The fine-tuning SAM model for specific object also requires the prompt as input. During fine-tuning it is easier because we can take bounding-box of ground-truth mask and provide that as prompt-input.

But during inference, we don't have any such bounding box already available

Hence, are there any experiments around where prompt was set to some pre-fixed static value or skipped completely and SAM decoder was fine-tuned to segment only the object of interest.

The end-result I want, is a solution where during inference, I should not be needed to pass any prompt as input.

Question: Can we fine-tune the decoder to infer without prompt ?
Possible Solution (Just thinking out loud): Fine-Tuning without Prompt or fixed static prompt >> Hence infer without prompt using fine-tuned model?

Every tutorial I saw has one thing in common: The fine-tuning SAM model for specific object also requires the prompt as input. During fine-tuning it is easier because we can take bounding-box of ground-truth mask and provide that as prompt-input.
But during inference, we don't have any such bounding box already available

Hence, are there any experiments around where prompt was set to some pre-fixed static value or skipped completely and SAM decoder was fine-tuned to segment only the object of interest.

The end-result I want, is a solution where during inference, I should not be needed to pass any prompt as input.

It is what I did. Finetune without any prompt because I do not have during inference time.
Therefore, I have to predict specific classes so I recreate a decoder with n_classes masks as output. So, I train the decoder from scratch which is very long to converge. It seems to not be the best model for this task.
Imo, it's powerful for automatic segmentation task in an annotation tool or trimming object in PS etc..

@bertinma Thanks for your inputs. I was actually planning to try this out over the weekend.
If possible, can you please share some insights on how you fine-tuned the decoder ? As in:

Did you give a static point as input ? (example: center of image or something like that )
Or did you change the decoder architecture and removed all parts where prompt encoding was getting used ?
Or did you completely wrote a custom decoder which upsampled 256x64x64 encoder embedding size to original image size using convTranspose2d or something else ?

Thanks in advance and thanks again for sharing your initial inputs .

Yes, as you said, the SAM models take N point prompts and converts it to shape: Nx256. Later, the encoder embedding and prompt embedding have 2 cross-attention blocks in the decoder.
So since your model did not converge and during fine-tuning did not throw an error as well, guessing from my current shallow knowledge:

when we not pass any prompt, it might have selected a default prompt (usually a grid of points) and trained on it ?
Not sure how this will go, but I will try to write a custom decoder and train it from scratch (directly from encoder embedding)

Thanks again for your insights, this will surely help me.

@yogendra-yatnalkar I plan to do something similar. May you share with me your result ? Did you succeed ? If you did, may I ask how have you implemented that ? Thank you very much.

Hi @TKPhuong, yes, I tried it on a very small sample and the results are very promising. I currently tried it on some data which I cannot share. I will surely replicate it on some kaggle or open-source dataset over the weekend and share that over here.

Hi @yogendra-yatnalkar, interested to hear if you ended up replicating it with an open-source dataset? Thanks.

@yogendra-yatnalkar @AdrienneBerghAMAT @TKPhuong @bertinma @wm-Githuber @nhhung1810 @nahidalam @DuongTSon @bhack @Jeff1933 @andyoung009 @mikiotada @EnchiridionHero @dorbodwolf @JunMa11 @austinmw @hu-po @cnsystem
We've actually built out a platform that finetunes a research variant of SAM for labeling purposes at InstaLabel. All you need to do is upload your dataset, select and label a few samples you'd like to finetune the model on in our platform, and hit the 'Train annotator' button. You can then head over to the 'Annotate' tab, click 'Instant Annotation' and have the finetuned model label the rest of your dataset for you. Should be useful for a lot of your custom computer vision use cases. Hope this helps!

1k images
2 x h100 sxm 80

Yes, as you said, the SAM models take N point prompts and converts it to shape: Nx256. Later, the encoder embedding and prompt embedding have 2 cross-attention blocks in the decoder.
So since your model did not converge and during fine-tuning did not throw an error as well, guessing from my current shallow knowledge:

when we not pass any prompt, it might have selected a default prompt (usually a grid of points) and trained on it ?
Not sure how this will go, but I will try to write a custom decoder and train it from scratch (directly from encoder embedding)

Thanks again for your insights, this will surely help me.

This is very useful, thanks @yogendra-yatnalkar

Has anyone tried the idea of what may be called "point prompt engineering"? i.e. training a separate model that learns how to put positive prompt points and negative prompt points, such that these points prompt SAM to select target objects from a custom dataset.
Or we can just summarize strategies and best practices in terms of placing positive and negative prompt points/prompt boxes, similar to how GPT/DALLE users summarize the best ways to write prompts.
Wonder if this could be one way to fine-tune the SAM model when only a limited amount of annotations are available. Happy to discuss more if anyone wants to work together and try it out.

https://github.com/vignywang/SAMFeat

The blog that @alex-encord shared works fine if you have 1 bounding box per image. But in reality you will have multiple bounding box in an image. Below code fails if you have multiple bounding box
with torch.no_grad():
      sparse_embeddings, dense_embeddings = sam_model.prompt_encoder(
          points=None,
          boxes=box_torch,
          masks=None,
      )

Am I missing something?

Hi @nahidalam I have same question as you, i.e., reality image will have multiple object bounding box. Should we put the multiple box together? Thanks in advance.

Hi @wm-Githuber , @nahidalam ,
Did you find an answer for this? Can one fine tune with a mask containing multiple objects and multiple boxes or how should one do this?
I created a training entry for each mask with one box and seems to work fine, but I wonder if using multiple boxes / objects is possible cause it would make training faster. Until now, nobody has done this.

Hi @TKPhuong @bertinma @AdrienneBerghAMAT Sorry for the late response. I actually had to opensource this NB with a small blog but I did not find time for that hence sharing the kaggle NB right away:

https://www.kaggle.com/code/yogendrayatnalkar/finetuning-segment-anything/notebook
(For readers having less time, please jump straight on the Results section)

How does SAM work (high-level):

Sam Encoder --> ViT + Neck-Module (Consisting of 2 Conv2D layers used for downsampling the channels of the ViT output)
Input: 1024x1024x3
Output of Encoder: 256x64x64
This output goes into the decoder with Prompt Input and generates the output

What I tried with model:

Removed the decoder
Freeze the ViT part of encoder and un-freeze the Conv2d Neck
Add a custom decoder having multiple blocks of: Conv2d-Transpose + LayerNorm2D + Relu + Dropout --> Added 4 such blocks
The input to the decoder will be of shape: 256x64x64 and the output will be of shape: 1024x1024x1

Training:

I trained this SAM+Custom-Decoder model on a open kaggle dataset consisting of binary segmentation
Dataset has 1620 images.
To prove SAM's capability, I trained this model only on 135 images, ie around 8.3% of the total data just for 11 epochs

Results:

The results are very promising as on a completely random image of test-set, have a look at the output:
(Left-most image is the ground-truth, middle image is the model prediction, right-most image is the input)

 I really don't think that if we train any other model just for 11 epochs and 135 images, it would be this good in generalization on a test-set image.
Future:

If this Encoder is combined with a model like: Mask2Former, according to my current understanding, this would show very good results.
The Decoder is very basic model currently but just proves that this can act as a good starting point for further experimentations

Will just convert this long text to a blog soon. Please let me know your thoughts on this experiments as well.

Hi readers, just 2 small updates from my previous message.

Update-1:
- My code had a small issue. My logic of un-freezing the 2 conv2d blocks was not getting implemented. Some good person on kaggle corrected it and the performance has increased slightly
- Earlier the IoU score on my test-set was around 89%, now its 91% ++ The edges of segmentation maps have improved.

Update-2:
- I have added visual explanation on what I did (PS: By no means my way of doing this is very unique)

The kaggle NB: https://www.kaggle.com/code/yogendrayatnalkar/promptless-taskspecific-finetuning-of-metaai-sam

@yogendra-yatnalkar Thanks for the nice work.
Can you specify the GPU memory usage of fine tuning either SAM original mask decoder or your custom decoder without using prompts.
I will be fine tuning SAM mask decoder only on my few shot medical images (maximum 50). But I have only 8GB GPU memory allocated from my university. Hence I am confused that 8GB would be enough or not to fine tune only mask decoder with few medical images.
Thanks!

Hi, I have finetuned a Sam model on a custom dataset, however i want to use the AutomaticMaskGenerator. When loading my finetuned model i get the following error: AttributeError: 'SamModel' object has no attribute 'image_encoder'
Does anyone know why this is and are there any solutions ?

@williamhoole I do not know the answer, but encounter the same error

I'm trying to do a fine-tuning on the semantic segmentation task without bbox, but only labels. I resized the image and label to be 1024x1024, and use the following codes:
    # Assign data
    img, label, o_img_size, n_img_size = batch['image'], batch['label'], batch['original_image_size'], batch['image_size']

    # Map to variables
    img = Variable(img)
    label = Variable(label)

    # Get embeddings
    with torch.no_grad():
        image_embedding = self.model.image_encoder(img)
        sparse_embeddings, dense_embeddings = self.model.prompt_encoder(
            points=None,boxes=None,masks=label.float())

    # Get predictions
    low_res_masks, iou_predictions = self.model.mask_decoder(
        image_embeddings=image_embedding,
        image_pe=self.model.prompt_encoder.get_dense_pe(),
        sparse_prompt_embeddings=sparse_embeddings,
        dense_prompt_embeddings=dense_embeddings,
        multimask_output=True,
        )
    upscaled_masks = self.model.postprocess_masks(low_res_masks, n_img_size, o_img_size)

`
I modify the decoder head like this:
    net = sam_model_registry[args.sam_model_type](checkpoint=args.ckpt)
    d = net.mask_decoder
    net.mask_decoder = MaskDecoder(transformer_dim=d.transformer_dim, transformer=d.transformer, num_multimask_outputs=args.num_classes)

    return net

However, I got errors from the decoder embeddings fusions:
    from segment_anything/modeling/mask_decoder.py", line 127, in predict_masks
    src = src + dense_prompt_embeddings
    RuntimeError: The size of tensor a (64) must match the size of tensor b (256) at non-singleton dimension 3

In this case, my sparse_embeddings has shape [B,0,256], and dense_embeddings has shape [B, 256, 256, 256]. Anyone has ideas?

Where do you import the MaskDecoder from? Because I get this error when using from segment_anything.modeling import MaskDecoder:
Traceback (most recent call last):
  File "/home/raminta/PROJECTS/Raminta/farmvibes/main.py", line 94, in <module>
    transformer_dim=d.transformer_dim,
  File "/home/raminta/anaconda3/envs/farmvibes/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1688, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'SamMaskDecoder' object has no attribute 'transformer_dim'

I'm trying to do a fine-tuning on the semantic segmentation task without bbox, but only labels. I resized the image and label to be 1024x1024, and use the following codes:
    # Assign data
    img, label, o_img_size, n_img_size = batch['image'], batch['label'], batch['original_image_size'], batch['image_size']

    # Map to variables
    img = Variable(img)
    label = Variable(label)

    # Get embeddings
    with torch.no_grad():
        image_embedding = self.model.image_encoder(img)
        sparse_embeddings, dense_embeddings = self.model.prompt_encoder(
            points=None,boxes=None,masks=label.float())

    # Get predictions
    low_res_masks, iou_predictions = self.model.mask_decoder(
        image_embeddings=image_embedding,
        image_pe=self.model.prompt_encoder.get_dense_pe(),
        sparse_prompt_embeddings=sparse_embeddings,
        dense_prompt_embeddings=dense_embeddings,
        multimask_output=True,
        )
    upscaled_masks = self.model.postprocess_masks(low_res_masks, n_img_size, o_img_size)

I modify the decoder head like this:
    net = sam_model_registry[args.sam_model_type](checkpoint=args.ckpt)
    d = net.mask_decoder
    net.mask_decoder = MaskDecoder(transformer_dim=d.transformer_dim, transformer=d.transformer, num_multimask_outputs=args.num_classes)

    return net

However, I got errors from the decoder embeddings fusions:
    from segment_anything/modeling/mask_decoder.py", line 127, in predict_masks
    src = src + dense_prompt_embeddings
    RuntimeError: The size of tensor a (64) must match the size of tensor b (256) at non-singleton dimension 3

In this case, my sparse_embeddings has shape [B,0,256], and dense_embeddings has shape [B, 256, 256, 256]. Anyone has ideas?

Where do you import the MaskDecoder from? Because I get this error when using from segment_anything.modeling import MaskDecoder:
Traceback (most recent call last):
  File "/home/raminta/PROJECTS/Raminta/farmvibes/main.py", line 94, in <module>
    transformer_dim=d.transformer_dim,
  File "/home/raminta/anaconda3/envs/farmvibes/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1688, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'SamMaskDecoder' object has no attribute 'transformer_dim'

I am currently out of office, but I figured out what caused my error.
I fine tuned my model using the segment anything model from hugging face. The issue is that hugging face models mask decoder is named differently. So when fine tuning the model using hugging faces SAM model and weights you cannot use Metas SAM automatic mask generator.
I resolved my issue by using hugging faces automatic mask generation pipeline
beginning of next week I am back in office and can take a closer look at your issue. Hopefully this helps you further for now.

I'm trying to do a fine-tuning on the semantic segmentation task without bbox, but only labels. I resized the image and label to be 1024x1024, and use the following codes:
    # Assign data
    img, label, o_img_size, n_img_size = batch['image'], batch['label'], batch['original_image_size'], batch['image_size']

    # Map to variables
    img = Variable(img)
    label = Variable(label)

    # Get embeddings
    with torch.no_grad():
        image_embedding = self.model.image_encoder(img)
        sparse_embeddings, dense_embeddings = self.model.prompt_encoder(
            points=None,boxes=None,masks=label.float())

    # Get predictions
    low_res_masks, iou_predictions = self.model.mask_decoder(
        image_embeddings=image_embedding,
        image_pe=self.model.prompt_encoder.get_dense_pe(),
        sparse_prompt_embeddings=sparse_embeddings,
        dense_prompt_embeddings=dense_embeddings,
        multimask_output=True,
        )
    upscaled_masks = self.model.postprocess_masks(low_res_masks, n_img_size, o_img_size)

I modify the decoder head like this:
    net = sam_model_registry[args.sam_model_type](checkpoint=args.ckpt)
    d = net.mask_decoder
    net.mask_decoder = MaskDecoder(transformer_dim=d.transformer_dim, transformer=d.transformer, num_multimask_outputs=args.num_classes)

    return net

However, I got errors from the decoder embeddings fusions:
    from segment_anything/modeling/mask_decoder.py", line 127, in predict_masks
    src = src + dense_prompt_embeddings
    RuntimeError: The size of tensor a (64) must match the size of tensor b (256) at non-singleton dimension 3

In this case, my sparse_embeddings has shape [B,0,256], and dense_embeddings has shape [B, 256, 256, 256]. Anyone has ideas?

Where do you import the MaskDecoder from? Because I get this error when using from segment_anything.modeling import MaskDecoder:
Traceback (most recent call last):
  File "/home/raminta/PROJECTS/Raminta/farmvibes/main.py", line 94, in <module>
    transformer_dim=d.transformer_dim,
  File "/home/raminta/anaconda3/envs/farmvibes/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1688, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'SamMaskDecoder' object has no attribute 'transformer_dim'

I squeezed the mask variable to have size of  (# of batches, 1,size of mask,size of  mask) rather than (# of batches, 1,1 ,size of mask,size of  mask)
and it works

Hey, my friend and I have been working on a fine-Turing of SAM (automatic predictor) for our thesis. It might be useful for your problem. Check it out https://github.com/Marchisceddu/finestSAM

Hey, my friend and I have been working on a fine-Turing of SAM (automatic predictor) for our thesis. It might be useful for your problem. Check it out https://github.com/Marchisceddu/finestSAM

This looks great, I look forward to trying it out. Merry Christmas 🎄


mudselect with multiselection and lines more than 1

Bug type
Component
Component name
MudSelect
What happened?
Multiple lines do not appear in mudselect component when multiselection is true and property lines are set to more than 1. Here is one simple example:
https://try.mudblazor.com/snippet/wumRasPeVCetUtWf
Expected behavior
The multiple-line MudSelect component.
Reproduction link
https://try.mudblazor.com/snippet/wumRasPeVCetUtWf
Reproduction steps
/
Relevant log output
No response
Version (bug)
6.8.0
Version (working)
No response
What browsers are you seeing the problem on?
Chrome, Microsoft Edge
On what operating system are you experiencing the issue?
Windows
Pull Request

[ ] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi there @Senso4sLab
at the moment the Lines property is not passed down to the underlying input of a MudSelect.
You are right, this is a bug in the docs as this property is listed there as inherited, but i am not sure, if it is possible to remove the property from being shown there.
Do you use/need this property ?

In the case of multiple select mode, it will be good if the MudSelect element allows an automatic update of its height or at least the user has a property, just as lines, to define the initial height of the mudSelect element.


I want to use keystore,truststore

I want to use keystore,truststore

Hi, @naphattharawat. Thanks for creating this request. As I understand you, you are interested in configuring client  keystore and trust store configs. It's possible now, you could pass these params like this:
KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_LOCATION=
KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_PASSWORD=

Please let me know, if you your request was not matched with this info.


Folder name inside WindowsApps isn't necessarily the appid

I need to find a way to populate AUMIDS from windows packages to make things actually work, not based on simple guess

Fixed, finding proper aumids with powershell.


Add r5.8xlarge instance runs for concurrent search baseline

Description
Add r5.8xlarge instance runs for concurrent search baseline.
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Please rebase on main for link checker fix in #4581.
Thanks.