Detecting square in image

Question

At a school we are preparing artwork which we have scanned and want automatically crop to the correct size. The kids (attempt) to draw within a rectangle:

I want to detect the inner rectangle borders, so I have applied a few filters with accord.net:

var newImage = new Bitmap(@"C:\Temp\temp.jpg");
var g = Graphics.FromImage(newImage);
var pen = new Pen(Color.Purple, 10);

var grayScaleFilter = new Grayscale(1, 0, 0);
var image = grayScaleFilter.Apply(newImage);
image.Save(@"C:\temp\grey.jpg");

var skewChecker = new DocumentSkewChecker();
var angle = skewChecker.GetSkewAngle(image);
var rotationFilter = new RotateBilinear(-angle);
rotationFilter.FillColor = Color.White;
var rotatedImage = rotationFilter.Apply(image);
rotatedImage.Save(@"C:\Temp\rotated.jpg");

var thresholdFilter = new IterativeThreshold(10, 128);
thresholdFilter.ApplyInPlace(rotatedImage);
rotatedImage.Save(@"C:\temp\threshold.jpg");

var invertFilter = new Invert();
invertFilter.ApplyInPlace(rotatedImage);
rotatedImage.Save(@"C:\temp\inverted.jpg");

var bc = new BlobCounter
{
    BackgroundThreshold = Color.Black,
    FilterBlobs = true,
    MinWidth = 1000,
    MinHeight = 1000
};

bc.ProcessImage(rotatedImage);
foreach (var rect in bc.GetObjectsRectangles())
{
    g.DrawRectangle(pen, rect);
}

newImage.Save(@"C:\Temp\test.jpg");

This produces the following inverted image that the BlobCounter uses as input:

But the result of the blobcounter isn't super accurate, the purple lines indicate what the BC has detected.

Would there be a better alternative to the BlobCounter in accord.net or are there other C# library better suited for this kind of computer vision?

Answer 1

Here is a simple solution while I was bored on my lunch break.

Basically it just scans all the dimensions from outside to inside for a given color threshold (black), then takes the most prominent result.

Given

[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static unsafe bool IsValid(int* scan0Ptr, int x, int y,int stride, double thresh)
{
   var c = *(scan0Ptr + x + y * stride);
   var r = ((c >> 16) & 255);
   var g = ((c >> 8) & 255);
   var b = ((c >> 0) & 255);

   // compare it against the threshold
   return r * r + g * g + b * b < thresh;
}

private static int GetBest(IEnumerable<int> array)
   => array.Where(x => x != 0)
      .GroupBy(i => i)
      .OrderByDescending(grp => grp.Count())
      .Select(grp => grp.Key)
      .First();

Example

private static unsafe Rectangle ConvertImage(string path, Color source,  double threshold)
{

   var thresh = threshold * threshold;

   using var bmp = new Bitmap(path);

   // lock the array for direct access
   var bitmapData = bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height), ImageLockMode.ReadWrite, PixelFormat.Format32bppPArgb);
   int left, top, bottom, right;

   try
   {
      // get the pointer
      var scan0Ptr = (int*)bitmapData.Scan0;    
      // get the stride
      var stride = bitmapData.Stride / 4;

      var array = new int[bmp.Height];

      for (var y = 0; y < bmp.Height; y++)
      for (var x = 0; x < bmp.Width; x++)
         if (IsValid(scan0Ptr, x, y, stride, thresh))
         {
            array[y] = x;
            break;
         }

      left = GetBest(array);

      array = new int[bmp.Height];

      for (var y = 0; y < bmp.Height; y++)
      for (var x = bmp.Width-1; x > 0; x--)
         if (IsValid(scan0Ptr, x, y, stride, thresh))
         {
            array[y] = x;
            break;
         }

      right = GetBest(array);

      array = new int[bmp.Width];

      for (var x = 0; x < bmp.Width; x++)
      for (var y = 0; y < bmp.Height; y++)
         if (IsValid(scan0Ptr, x, y, stride, thresh))
         {
            array[x] = y;
            break;
         }

      top = GetBest(array);

      array = new int[bmp.Width];

      for (var x = 0; x < bmp.Width; x++)
      for (var y = bmp.Height-1; y > 0; y--)
         if (IsValid(scan0Ptr, x, y, stride, thresh))
         {
            array[x] = y;
            break;
         }

      bottom = GetBest(array);


   }
   finally
   {
      // unlock the bitmap
      bmp.UnlockBits(bitmapData);
   }

   return new Rectangle(left,top,right-left,bottom-top);

}

Usage

var fileName = @"D:\7548p.jpg";

var rect = ConvertImage(fileName, Color.Black, 50);

using var src = new Bitmap(fileName);
using var target = new Bitmap(rect.Width, rect.Height);
using var g = Graphics.FromImage(target);

g.DrawImage(src, new Rectangle(0, 0, target.Width, target.Height), rect, GraphicsUnit.Pixel); 

target.Save(@"D:\Test.Bmp");

Output

Notes :

• This is not meant to be bulletproof or the best solution. Just a fast simple one.
• There are many approaches to this, even machine learning ones that are likely better and more robust.
• There is a lot of code repetition here, basically I just copied, pasted and tweaked for each side
• I have just picked an arbitrary threshold that seems to work. Play with it
• Getting the most common occurrence for the side is likely not the best approach, maybe you would want to bucket the results.
• You could probably sanity limit the amount a side needs to scan in.