Unable to get contour when bounding boxes are not overlapping

Question

I have some sprite sheets. In some cases, the bounding boxes of the sprites are overlapping, even though the sprites themselves are not overlapping:

In other cases, the bounding boxes are not overlapping:

To extract the individual sprites, I am doing the following:

im = cv2.imread("trees.png") # read image
imGray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY) # convert to gray
contours, _ = cv2.findContours(imGray, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) # contouring
sortedContours = sorted(contours, key=cv2.contourArea, reverse=True) # sorting, not necessary...
for contourIdx in range(0,len(sortedContours)-1): # loop with index for easier saving
    contouredImage = im.copy() # copy image
    contouredImage = cv2.drawContours(contouredImage, sortedContours, contourIdx, (255,255,255), -1) # fill contour with white
    extractedImage = cv2.inRange(contouredImage, (254,254,254), (255,255,255)) # extract white from image
    resultImage = cv2.bitwise_and(im, im, mask=extractedImage) # AND operator to get only one filled contour at a time
    x, y, w, h = cv2.boundingRect(sortedContours[contourIdx]) # get bounding box
    croppedImage = resultImage[y:y + h, x:x + w] # crop
    cv2.imwrite("contour_"+str(contourIdx)+".png", croppedImage) # save

This works great for the former image where the bounding boxes are overlapping, but fails in the latter case where the bounding boxes are not overlapping. Why is that and how can I fix it ?

EDIT: In the former case, as expected, it detects the individual contours and outputs each separately. But in the latter case, it doesn't seem to detect any individual contours, or rather the whole image is output.

Answer 1

The problem is that your images have not been created in the same way. If you run ExifTool on the overlap image, it has this:

Bit Depth                       : 8
Color Type                      : RGB with Alpha
Compression                     : Deflate/Inflate
Filter                          : Adaptive
Interlace                       : Adam7 Interlace
SRGB Rendering                  : Perceptual
Exif Byte Order                 : Big-endian (Motorola, MM)
Orientation                     : Horizontal (normal)
X Resolution                    : 72
Y Resolution                    : 72
Resolution Unit                 : inches
Software                        : Pixelmator 3.9.11
Modify Date                     : 2024:01:21 12:01:94
Color Space                     : sRGB
Exif Image Width                : 526
Exif Image Height               : 244
Pixels Per Unit X               : 2835
Pixels Per Unit Y               : 2835
Pixel Units                     : meters
XMP Toolkit                     : XMP Core 6.0.0
...
Creator Tool                    : Pixelmator 3.9.11

And the no-overlap image:

Bit Depth                       : 8
Color Type                      : RGB with Alpha
Compression                     : Deflate/Inflate
Filter                          : Adaptive
Interlace                       : Adam7 Interlace
Software                        : Adobe ImageReady
XMP Toolkit                     : Adobe XMP Core 5.0-c060 61.134777, 2010/02/12-17:32:00
Creator Tool                    : Adobe Photoshop CS5 Windows

Which of those aspects is causing the exact issue doesn't really matter, but if you examine the image after loading it with OpenCV, you'll find that the transparent pixels in the no-overlap image are green, and the grey-scale conversion turns all the pixels into the same colour.

However, this code works for both images:

# read the image as is, no interpretation
im = cv2.imread("trees.png", cv2.IMREAD_UNCHANGED)
# get the first three channels as RGB
rgb = im[:, :, :3]
# replace all the transparent pixels (alpha = 0) with black
rgb[im[:, :, 3] == 0] = [0, 0, 0]
# generate a greyscale image from the resulting RGB channels
imGray = cv2.cvtColor(rgb, cv2.COLOR_RGB2GRAY)

Instead of picking black, you might pick white, depending on what you expect to see in the image. The change above works, but it doesn't means it's the best way to approach this. I would imagine the alpha channel (im[:, :, 3]) actually has everything you need and perhaps you should work with that instead of the RGB channel, to find the contours.

Here's a rewrite of your code that's a bit easier to understand and uses the alpha channel for the contouring:

import cv2


def load_image(im_filename):
    return cv2.imread(im_filename, cv2.IMREAD_UNCHANGED)


def extract_alpha_channel(im):
    return im[:, :, 3]


def find_contours(channel):
    return cv2.findContours(channel, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[0]


def crop_and_save_contoured(im, contours):
    for idx, contour in enumerate(contours):
        x, y, w, h = cv2.boundingRect(contour)
        cropped_image = im[y:y + h, x:x + w]
        cv2.imwrite(f"output/contour_{idx}.png", cropped_image)


def main():
    original_image = load_image("no_overlap.png")
    alpha_channel = extract_alpha_channel(original_image)
    contours = find_contours(alpha_channel)
    crop_and_save_contoured(original_image, contours)


if __name__ == "__main__":
    main()