How to depict an image in canvas using segment anything result data

Question

In the example below, I get the generated data. I want to pass this masks to the web client to draw the mask image in the canvas, but I don't know which data inside the masks can be used to draw the image. Can you give me an example?

from segment_anything import SamAutomaticMaskGenerator, sam_model_registry
import numpy as np
from PIL import Image
sam = sam_model_registry["vit_b"] (checkpoint="/Users/jiangxingshang/Downloads/sam_vit_b_01ec64.pth")
sam.to(device='cuda')
mask_generator = SamAutomaticMaskGenerator(sam)

img = Image.open("/Users/jiangxingshang/Pictures/8435e5dde71190ef699eced5cc1b9d16fdfa6056.jpg")
imgnp = np.array(img)
masks = mask_generator.generate(imgnp)

Answer 1

You can follow this example.

Briefly, masks is a list, in which each mask is a dictionary. For example,

first_mask = masks[0]['segmentation'] 

will assign the first mask to first_mask variable, assuming the list masks is not empty.

The dictionary keys are the following:

• segmentation : the mask
• area : the area of the mask in pixels
• bbox : the boundary box of the mask in XYWH format
• predicted_iou : the model's own prediction for the quality of the mask
• point_coords : the sampled input point that generated this mask
• stability_score : an additional measure of mask quality
• crop_box : the crop of the image used to generate this mask in XYWH format