This project has moved and is read-only. For the latest updates, please go here.

Image Compare in C# is returning erroneous results for matching .png images

Aug 20, 2015 at 11:26 PM
I'm using Magick.NET with C# and using Selenium to take browser screenshots and then compare them to reference images (also taken by Selenium) and return the percent match between the two. If they are not close, it's fine. If they are exact duplicates (I make file1 and file2 point at the same file) then it shows 100%. When the images are not exactly duplicate, it often returns a match percentage that is...very large:

"PASS: Images are duplicate: 27,529,786,086.48 % - (275297860.864805)"

The second value is the same, just not converted to percent. This is for 2 .png screenshots of the same screen taken in succession and which thus have a very high similarity but are not binary equal. Is this a known issue? Also, if I change the ErrorMetric from Undefined to Absolute, I still get a hellaciously large value:

PASS: Images are duplicate: 78,402,600.00 % - (784026)

Here's my code, I've carved it down to the important bits, so I might be plus or minus a curly brace, and WriteToFile is our in-house logging:

void compareScreenshotImages(string file1, string file2, double accuracy)
{

TakeScreenshot(driver, file2);
        using (MagickImage image1 = new MagickImage(file1))
        {
            MagickImage compare = new MagickImage();
            MagickImage image2 = new MagickImage(file2);

            Double result = image1.Compare(image2, ErrorMetric.Undefined, compare);

            compare.Write(compareresult);

            if (result > accuracy) {
                WriteToFile(String.Format("  PASS: Images are duplicate: {0:P2} - ({0})", result));
            }
            else {
                WriteToFile(String.Format("ERROR: Result similarity: {0:P2} - ({0})", result));
            }
   }
}
Aug 21, 2015 at 7:45 PM
Edited Aug 21, 2015 at 8:20 PM
When you use ErrorMetric.Undefined you are actually using ErrorMetric.NormalizedCrossCorrelation. You can find an explanation about the different error metrics here: http://www.imagemagick.org/Usage/compare/#statistics.

I don't understand why you get such an odd value, this looks like a bug to me. I cannot reproduce this with my local development version. Are you using the latest version of Magick.NET? And can you tell me the full version you are using? (this includes Q8/16 x86/x64).

Can you add a link to the two images that you are comparing that are showing these odd numbers so I can try it for myself? Feel free to contact me through CodePlex if you don't want to publicly share the images.
Aug 24, 2015 at 9:44 PM
Edited Aug 24, 2015 at 9:44 PM
Thank you for sending me those images. You are getting this odd value because you are comparing images that have different dimensions. This means that the smaller image will get so called 'virtual pixels' that will have the value of the background color. You will need to add an extra check to make sure that the images are equal and I would also advise you to use a different ErrorMetric. I use ErrorMetric.RootMeanSquared most of the time. The closer the value is to zero the better the match. Below is an example of what you could do instead:
void compareScreenshotImages(string file1, string file2, double accuracy)
{
  TakeScreenshot(driver, file2);

  using (MagickImage image1 = new MagickImage(file1))
  {
    using (MagickImage image2 = new MagickImage(file2))
    {
      if (image1->Width != image2->Width || image1->Height != image2->Height)
        WriteToFile("ERROR: Invalid dimensions");
      else
      {
        using (MagickImage compare= new MagickImage())
        {
          Double result = image1.Compare(image2, ErrorMetric.RootMeanSquared, compare);
          compare.Write(compareresult);

          if (result > accuracy)
            WriteToFile(String.Format("  PASS: Images are duplicate: {0:P2} - ({0})", result));
          else
            WriteToFile(String.Format("ERROR: Result similarity: {0:P2} - ({0})", result));
        }
      }
   }
}