Detect Graphical Regions in an Image on a Web Page

Question

I've been beating around the bush a lot, so I'll explain my problem here and hope with the whole picture, somebody has some ideas. With the following image:

I need to detect a mouseover on the blobs over her eyes and mouth, and solve this problem in a general form. The model and blobs are on two different layers, so I can produce one image with only the blobs, and one with only the model, and somehow synchronise a virtual cursor over the blobs while it actually hovers over the model.

I can also make the blobs polygons, for hit testing, but I think a colour hit test would be much easier. If I hit blue, I am on her mouth and I show lipstick images; if I hit pink, I'm over her eyes, and display eye makeup images.

What are the suggestions and conversation of the learned ones here?

Answer 1

The simpler way to do it would be to load the layer image in a canvas, then get all its pixel data. When the mouse is hovering the model image, find out what color is currently selected and if it is different from a previous one, trigger an event to indicate that the selection has changed.

Here is an example, feel free to toy with it; but be aware that it doesn't handle all cases:

• what if the layer and the model image are not the same size
• what if the layer and the model image are not the same width/height ratio
• what if you want to use some alpha channel (the example doesn't take it into account)

$(function() {

  /* we load all the image data first */
  var imageData = null;
  var layerImage = new Image();
  layerImage.onload = function() {
    var canvas = document.createElement("canvas");
    canvas.width = this.width;
    canvas.height = this.height;
    context = canvas.getContext('2d');
    context.drawImage(this, 0, 0, canvas.width, canvas.height);
    imageData = context.getImageData(0, 0, canvas.width, canvas.height).data;
  };

  /* it's easier to set the image data for example as base64 data */
  layerImage.src = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAYAAACqaXHeAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAOwgAADsIBFShKgAAAABp0RVh0U29mdHdhcmUAUGFpbnQuTkVUIHYzLjUuMTFH80I3AAAA5klEQVR4Xu3WMQ7CMAwF0IxcgCtw/xsGdWApItWHKorLQ8qESe2HbWjNiwABAgQIECBAgAABAgQIECBAgMAUgd4e/ehMSeTMh4wK2j/nqPjt/TNzm3IXgEFb64CdgBG44hKcsmg8pKDAvbf+7SlY7nvK3xb/+lx5BAA/jMCGpwOqC/z9CFT/ApfP/9Z6T87yBaUJJsVvsen9y8cDMAJ2gCWY7IHll1qaYFL84a9AelnF+CFwxYLSnAGMBFLNivE6QAcMBCq2dJqzETACRuCzQDpPFePTv9riCRAgQIAAAQIECBC4nsATagY67TVyuhAAAAAASUVORK5CYII=";

  var pColor = null;

  /* on mouse over the model image */
  $("#model").mousemove(function(event) {
    /* we correct the offset */
    var offset = $(this).offset();
    var relX = event.pageX - offset.left;
    var relY = event.pageY - Math.round(offset.top);

    /* and get the pixel values at this place (note we are not keeping the alpha channel; it's your decision whether or not it is valuable */
    var pixelIndex = relY * layerImage.width + relX;
    var dataIndex = pixelIndex * 4;
    var color = [imageData[dataIndex], imageData[dataIndex + 1], imageData[dataIndex + 2]];

    if (pColor == null) {
      /* we trigger when first entering the image */
      $(this).trigger("newColor", {
        message: "Initial layer color",
        data: color
      });
    } else if (pColor[0] != color[0] || pColor[1] != color[1] || pColor[2] != color[2]) {
      /* we trigger if the new position is a new color in the layer image */
      $(this).trigger("newColor", {
        message: "Changed layer color",
        data: color
      });
    }
    pColor = color;
  });

  /* some small help to convert rgb to css colors */
  function rgb2hex(red, green, blue) {
    var rgb = blue | (green << 8) | (red << 16);
    return '#' + (0x1000000 + rgb).toString(16).slice(1)
  }

  /* there you have the new layer color event management; for the example sake we change the color of some text */
  $("#model").on("newColor", function(event, eventData) {
    $("#selector").css("color", rgb2hex(eventData.data[0], eventData.data[1], eventData.data[2]));
  });
});

img {
  border: 1px solid silver
}

<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.0/jquery.min.js"></script>

<body>
  <h4>Model image</h4>

  <img id="model" src="data:image/png;base64, iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAIAAAAlC+aJAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAadEVYdFNvZnR3YXJlAFBhaW50Lk5FVCB2My41LjExR/NCNwAAAtdJREFUaEPtmS2PwkAQhhGQEASEBIECFAJBgoCAxpDgsWBxSP4lPwGJRCLv5m4uk6Xt7s5He9dLFst2+z7vfG3bxsc//zX+uf6PBPDXEUwRSBEwOpBSyGig+fIUAbOFxg3KjMD1em3EfqvV6vl8GkW7l5cJsF6vY/p//i8RQwbwer22221U5W63K/T4fr93u1263LdMFB8ZAEc96BsOhwER5WIIAMB+NO92u4lMKlw8m80oFJaMEgCg/ZADdvW0A2GMRiPdtgKAEu13tUJG4c667sQFoBap8yl8FdiPDIqyZgFQ9oerU83mlrV0ExaAJfshdKfTiSMLg8BZKR5k4ewPS4TpNplMOLKqAqD88YngSwxjVAVgyR+O8bSmKoCKumeerVoAkZe6xQnA45vCGH7rfGuIFbVRBUDgvO2bCefzWXEj4I8PDtyXOYzQ0cVi4WMonAmWSR8HIDWbzUZ33grXNKifTqcA3Ov1FNUfB3AdhdOv4h6+S0D6fr/HWLVaLd1jBgsAFNChV5RLAVpKelDfbrd16lk1QCLczLZjoPEW6SiMG4FAdUphXO/tCSkD8GEwi9uVDvZLny6gZg6HQ8YvDYBr23g8DnR9319S6XjH5XIJGw4Gg7fxZwwiFDfsyGfQSQeRvnci1ghE+XXz1d3W7bb5WVF3ABpzvpZVdwDM+8CYqyOAmzPRx6l6ARyPx3w/CNd9jQDgNEHq+RO6LgCgvt/vA8Dlcol2tjLnQPRmhW00n+W4rNPpPB6P6J5cAHofqnhlSffIAIDT8/k8n+jNZhO8l6qPHOYKPxlJ3+Wj1swpqJRzKPc06n6JIOf4GBmz1U778kpWxJmvQ9EjEOQergEX1KegcEnIAHAvPgaIRgDpMwO/jjUA/N1hJT3Hia7iL64c4KtRfP/4mkQrq9r3rVUngEBMUgQYCZtqIGRSSqGUQgwHUgoZTSrrQ3KhjN8oYiN/+PJPqpb83Htu7qcAAAAASUVORK5CYII="
  />

  <p>You are pointing at some <strong><span id="selector">color</span></strong>
  </p>

  <hr/>
  <h4>Layer image (reference only, not displayed in page)</h4>

  <img id="layer" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAYAAACqaXHeAAAABGdBTUEAALGPC/xhBQAAAAlwSFlzAAAOwgAADsIBFShKgAAAABp0RVh0U29mdHdhcmUAUGFpbnQuTkVUIHYzLjUuMTFH80I3AAAA5klEQVR4Xu3WMQ7CMAwF0IxcgCtw/xsGdWApItWHKorLQ8qESe2HbWjNiwABAgQIECBAgAABAgQIECBAgMAUgd4e/ehMSeTMh4wK2j/nqPjt/TNzm3IXgEFb64CdgBG44hKcsmg8pKDAvbf+7SlY7nvK3xb/+lx5BAA/jMCGpwOqC/z9CFT/ApfP/9Z6T87yBaUJJsVvsen9y8cDMAJ2gCWY7IHll1qaYFL84a9AelnF+CFwxYLSnAGMBFLNivE6QAcMBCq2dJqzETACRuCzQDpPFePTv9riCRAgQIAAAQIECBC4nsATagY67TVyuhAAAAAASUVORK5CYII="
  />
</body>

Answer 2

You could use SVG graphics to layer over the image. My example uses an ellipse but you could use a polygons just as easily. You could use the colour like you stated in your question or add an extra property to the svg element. The example uses onclick but mouseover works as well.

example js:

function svg_clicked(objSVG)
{
    alert(objSVG.style.fill);
    alert(objSVG.getAttribute('data-category'));
}

example svg:

<svg xmlns="http://www.w3.org/2000/svg" version="1.1">
    <ellipse cx="110" cy="80" rx="100" ry="50" style="fill:red;" onclick="svg_clicked(this);" data-category="lipstick" />
</svg>

Here's a fiddle (move the mouse over the O's in the picture)

It still works if you make the svg element transparent (using fill:transparent). You can change the overlay to a colour or outline quickly for testing.

Answer 3

If you can have both images (the one with the blob and the one without), I think you can do this using HTML5 canvas.

• draw the image normally
• draw the blob image beneath the master image so it is invisible
• copy the blob to a Canvas
• onMouseOver, retrieve pixel data (R,G,B and alpha) for the Canvas at the appropriate coordinates
• profit

Twist: you might be able to do this with only one image and its alpha channel, if you don't need it for anything else - give the pixels a full opacity (A=255) everywhere except in blobs 1, 2 and 3, which will have opacity equal to 255-(1,2,3...). You can't have too many different blobs or the transparency will become noticeable. Haven't tried, but it should work. Given the likely compressibility of a "blob-only" image, a pair of images (one without transparency, one also without transparency and with only N+1 colours, PNG compressed) should yield better results.

More-or-less-pseudo code with two images, using jQuery (can be done without):

var image  = document.getElementById('mainImage')
var blobs  = document.getElementById('blobImage');

// Create a canvas
canvas = $('<canvas/>')[0];
canvas.width = image.width;
canvas.height = image.height;
// IMPORTANT: for this to work, this script and blobImage.src must be both
// in the same security domain, or you'll get "this operation is insecure"
canvas.getContext('2d').drawImage(blobs, 0, 0, image.width, image.height);

// Now wait for it.
$('#mainImage').mouseover(function(event) {
    // TO DO: offset clientX, clientY by margin on mainImage
    var ctx = canvas.getContext('2d');
    // Get one pixel
    var pix = ctx.getImageData(event.clientX, event.clientY, 1, 1);
    // Retrieve the red component
    var red = pix.data[0];
    if (red > 128) {
        // ... do something for red
    }
});

Answer 4

I highly recommend the time tested method.

The easiest way to both create blobs and to detect if the mouse is over them is to use svg graphics on top of the other image. SVG supports mouseover events and allows vector shapes which are going to give you far greater precision than using <map> or <area>.

I found this question that might also shed some light on where I am coming from: Hover only on non-transparent part of image. Read the second answer down because it will likely be prefered in your situation.

The svg elements on your image would be transparent (or whatever you would want), and you could easily detect the mouse over events.

The library from that question is called raphael. Hopefully this proves to be useful.