Adding Images Together With A Mask

Question

So I currently have 2 images below. These were obtained using a neural style transfer network on the foreground and background parts of the image.

The code used was the following:

add = cv2.add(image1, image2)
cv2.imshow('Addition', add)
cv2.waitKey(0)
cv2.destroyAllWindows()

I have merged them to create the following resultant image:

However, I want to use the salience mask on the obtained picture. The salience mask is below. I have tried inputting the mask argument to the add function but had no luck. I've tried looking at this article but wasn't able to work out what to do Add 2 images together based on a mask

EDIT:

Found the issue was that I was not converting the data types. The updated code is:

saliencyMap = cv2.resize(saliencyMap, (384,384))
newmap = saliencyMap.astype(np.uint8)
add = cv2.add(image1, image2, mask=newmap)
cv2.imshow('addition', add)
cv2.waitKey(0)

However, the resultant image is and I'm not sure why:

Answer 1

I think what you might want is to simply resize the two images to the same size and then add them. Then use the mask to do hard light compositing in Python/OpenCV.

img1:

img2:

mask (saliency):

import cv2
import numpy as np

# read image 1
img1 = cv2.imread('img1.png')
hh, ww = img1.shape[:2]

# read image 2 and resize to same size as img1
img2 = cv2.imread('img2.png')
img2 = cv2.resize(img2, (ww,hh))

# read saliency mask as grayscale and resize to same size as img1
mask = cv2.imread('mask.png')
mask = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
mask = cv2.resize(mask, (ww,hh))

# add img1 and img2
img12 = cv2.add(img1, img2)

# get mean of mask and shift mean to mid-gray
# desirable for hard light compositing
# add bias as needed
mean = np.mean(mask)
bias = -32
shift = 128 - mean + bias
mask = cv2.add(mask, shift)
mask = cv2.merge([mask,mask,mask])

# threshold mask at mid gray and convert to 3 channels
# (needed to use as src < 0.5 "if" condition in hard light)
thresh = cv2.threshold(mask, 128, 255, cv2.THRESH_BINARY)[1]

# do hard light composite of img12 and mask
# see CSS specs at https://www.w3.org/TR/compositing-1/#blendinghardlight
img12f = img12.astype(np.uint8)/255
maskf =  mask.astype(np.uint8)/255
threshf =  thresh.astype(np.uint8)/255
threshf_inv = 1 - threshf
low = 2.0 * img12f * maskf
high = 1 - 2.0 * (1-img12f) * (1-maskf)
result = ( 255 * (low * threshf_inv + high * threshf) ).clip(0, 255).astype(np.uint8)

# save results
cv2.imwrite('img12_hardlight.png', result)

# show results
cv2.imshow('img12', img12)
cv2.imshow('mask', mask)
cv2.imshow('thresh', thresh)
cv2.imshow('result', result)
cv2.waitKey(0)
cv2.destroyAllWindows()

Result: