Monthly Archives: October 2013

Final Year Project Log – Image Matching Experiment using SURF Feature Extraction

This post is regarding my Final Year Project I’m working on for my degree in Software Engineering at APIIT, which I’m working on an Augmented Reality based Product Identification and Advertising System.

Through this post I’m keeping a log book of the experiments I’m running for my utilized Image Matching algorithm based on SURF Feature Extraction and Matching algorithm.
This post is no educational or knowledge source, just some post that I’m keeping online to mark the progress of my experiment.

Language – C#
API – EmguCV (C# Wrapper for OpenCV)

Log 1 – Successfully installed and configured EmguCV in the Laptop running Window 8 Pro and imported all the required dll files for the project solution in Visual Studio 2012.

Log 2 – Implemented Simple Image Matching code and was able to find the matching points based on Extracted Feature points of a given Source Image and a Model Image.

Model Image – the Model Image that you want to look for in other image sceneries.
Source Image – the actual scenery to look up for the given image.

Log 3 – Started SURF Experiment project solution and implemented the following functionality,

– Load the given set of images to the memory at Run time and their feature points will be extracted and saved in the Memory. This was done in order to make the execution, as in the matching process faster.

– Look up for the brand logo of a given Product Image by extracting the Feature points of the given image and matching them with the feature points of the existing brand logos.

Log 4 – Oki now the process does not seem to be working for images that are from other sources, rather than a model image captured from the actual scenery image.

The model Image or the brand logo image in this case has to be an image which is being captured from the same exact product image that we are giving in to the system. The matching process does not work for custom logo images which are from other sources. Now lets say if we download a logo image from the internet and add it to the logo image library and load it to the program, and I give the actual product image in order to find the matching brand logo, it would not give accurate result and catch the above given logo as the result.

Concluded Reason based on Knowledge – I guess this is because of the essence of the SURF Algorithm, frankly speaking its capable of detecting a given object from a scenery where the object actually includes in the scenery. And now if we apply it to this scenario, the given logo is from another source and its feature points, colors, pixels are different from the product image that we are going to identify, therefore the algorithm would not detect it as the result logo.

This could be a constraint, but it would not matter if I implement an option to let the admin user capture the exact logo from the given product image and store that logo in the Database ! 😀

Log 5 – The Important attributes in extracting the matching results from the Algorithm in Code,

Matrix<int> indices – Still have no idea, but it stores some X,Y points based on the matched points I suppose.
Matrix<byte> mask – Store the results of filtering the matched features and votes for the points given by the algorithm. This matrix exactly includes the number of matched feature points. 

The actual voting for matched feature points happens in here –

using (Matrix<float> dist = new Matrix<float>(observedDescriptors.Rows, k))
 {
 matcher.KnnMatch(observedDescriptors, indices, dist, k, null);
 mask = new Matrix<byte>(dist.Rows, 1);
 mask.SetValue(255);
 Features2DToolbox.VoteForUniqueness(dist, uniquenessThreshold, mask);
 }

And I have discovered that using those values, specially “mask ” matrix value, I could determine something like the percentage of the matching between two images which actually stores the voting results of the Feature point matching. And then I could take the highest matching points count and get the best match for identifying the product logo.

But there is a drawback in that when I actually executed the process, that is the SURF detector detects several matching feature points in a given Product Logo image and a Product Image, where sometimes it detects randomly more than the actual supposed result, which leads to incorrect results. Therefore I need to look for a better approach.

Looking for a Better Approach… Rather than counting the number of matching points…

Log 6 – I just figured out that In order to solve this I need to consider some factors regarding the area of the detected results, as in when the algorithm look for the product logo in the given product image, it specifically pinpoints to a specific area of the logo in the product image. If I could calculate that that determine it, BOOM ! its done !

Here are some of the factors I found and their samples,

Test Samples –

This is how the algorithm actually detects the logo in the given product Image,
Capture

As you can see in the above Image in the right side the UI displays the Logo image and in the left side it displays the given product Image. The dots represents the detected feature points of the both images and those lines in between the two images determines and maps the matching feature points of the two images. As in how it displays where the logo is detected in the given product Image.

Pretty cool eh ! 😀

Then when I started digging deeper I noticed that those mapped lines actually forms some kind of a polygon around the detected logo area in the product image. So I edited the code to draw a line around it.
Capture2

The for a clear view I painted the formed polygon, just for the sake of viewing ease.Capture3
Looks perfect eh ! 😉 Now this should lead me to something useful while determining the exact accurate result.

Chasing the Polygon………..

Log 7 – I noticed that the formed Polygon on incorrect results or i n those results where the logo can not be detected, takes a malformed shape. Such as below,
Capture4

Capture6

And on some of the matching it does not even draw out the polygon as an example,
Capture5

Now this seems to provide a great opportunity, as in if I could determine whether its a properly formed polygon or not and whether it is actually a polygon in the given matching result, then I could easily figure out the perfect accurate match.

This is where the Polygon is initiated from the Homography matrix,

#region draw the projected region on the image
 if (homography != null)
 { //draw a rectangle along the projected model
 Rectangle rect = modelImage.ROI;
 PointF[] pts = new PointF[] { 
 new PointF(rect.Left, rect.Bottom),
 new PointF(rect.Right, rect.Bottom),
 new PointF(rect.Right, rect.Top),
 new PointF(rect.Left, rect.Top)};
 homography.ProjectPoints(pts);
}

Log 8 – Right now at this point I am calculating the below values in order to determine the perfect match result of the product image’s logo,

  • Number of Matching points count
  • Area of the Polygon
  • Number of Matching points within the Polygon Area

based on these values of each and every matching circle i was able to determine the best matching result.

Log 9 – Right now at this point I’m going through a crisis issue where there is no possible way to determine a matching instance of when a user inserts a product image which is not available in the database to find the product logo. As in lets say the user inserts a product image where it’s product logo isn’t available in the database, right now there is no way to determine whether it could be found or not.

This is because of the fact that, even though the number of matching counts goes down, some other factor such as area of the polygon, or may be if the area of the polygon goes down the number of matching points could increase. This situation changes from one image to another unpredictably and of course I tried my best to track down a pattern for this, but it was also unsuccessful as it differs from one image to another ! 😦

 Suggested Solutions at the point –

Get rid of the process of Identifying the Brand Logo of a given product Image and directly move on to visual product identification. This could take a long time to give a result.

As you can see the reason why I came up with this, “brand Logo Recognition process” was to increase the speed of the product identification. There is a huge set of product images in the database which are related to each and every brand type. So rather than search through all the product images, what if I identify the brand type first and then search through only the products that falls under that brand type ? Which will definitely increase the speed of matching reduce unwanted time consumption in matching.

Right at this point I’m pretty confused about what I should do, but I may have to go through the above solution ! :\ Must contact the supervisor and assessor immediately.

Log 10 – I contacted my Assessor lecturer of my FYP and presented him this issue. He came up with an amazing idea of taking the percentage of detected feature points between the Model, Logo Image and the Scenery, Product Image. This was a lil confusing for me at once but then later only it triggered me ! 😀

Therefore following that lead, I took the percentage of the difference between the Logo and the Given Product Image at every circulation of the matching and kept that value for later use in order to determine which gives the best percentage.

 percentage = (( (double)result.modelDescriptors.Rows) / (double)result.observedDescriptors.Rows) * 100;

In the above code I get the percentage and store it with the index number of the corresponding Logo and Product.

Then at the end, the matching percentage has to be over 20% and the Polygon Area should not be 0 and the Number of the matches inside the polygon area has to be over 0, otherwise it would be identified as unidentifiable.

if (currentLargestPercentage <= 20)
 {
    if ((currentlargestArea == 0 && currentlargestMatchesInPolygonArea == 0))
    {
       MessageBox.Show("Sorry ! This can not be Identified ! :( ");
    }
    else
    {

    }
 }
 else
 {
    if (currentlargestMatchesCount > 10)
    {

    }
 }

Log 11 – Using the above found method I was able to successfully identify the Product images which can not be identified with the Logo Images in the database. But the problem is sometimes it doesn’t identify the products which can even be identifies with the product logo in the database 😦 ! I tried changing the matching percentage but nothing seems to be working, when I change it, the product images which can be identified doesn’t get identified and  the ones that can not be identified comes up with wrong matches. 

So now I’m in need of a new approach for this, something similar to the same method.

Log 12 – The I used “descriptors” of the model and the observed images, which has the computed values of the given image and the points locations because of its convenience, as it appeared to be it was holding some significant set of values. But later I found out there is not much of use of it in take the percentage so i thought of switching to the “vector of feature points” of both logo and product image, where as those vectors holds the values for the detected SURF Feature points. Therefore according to that,

 percentage = (((double)result.modelKeypoints.Size) / ((double)result.observedKeypoints.Size)) * 100;

So now this seems like a perfect match by getting the ration between the keypoints of the model image and observed image. And then I did some tuning in matching identification condition.

if ((currentlargestArea == 0 && 
currentlargestMatchesInPolygonArea == 0) ||
currentLargestPercentage <= 5)
 {
         MessageBox.Show("Sorry ! This can not be Identified ! :( ");
 }

Now this appears to be giving fair results even though it is not 100% accurate, but according to the time constraints I should not waste anymore time with the matching process.

Screw this torture, I finished developing and successfully Implemented ! PS – Based on the above experimentations and results ! 😉
Hope it may help any of you !
Cheers ! 😀

When I was a Kid…

When I was a Kid, I used to stare at the Sky and Observe the night sky in the middle of the night till dawn. It was more of a life long dream of mine Astronomy Scientist. Well the reason should be the fact that ever since when I was a kid, I was extremely curious about the environment around me, what ever I could get my eye on, I got curious and start thinking about them, specially it was my mom who had to go through that headache when I was asking her 1000s of questions about each one of them. The Night Sky, Later which I later learned to call The Universe, is something which took my brain pretty much melted where as I got so obsessed with it.  I was so fascinated about Universe ever since I was a kid, as it was such an amusement for me even to think, read or even talk with someone about it.

Amazing-Night-Sky-Photography-image

My mom who was really supportive of this behavior of mine, bought me tons of books about Space and Universe. At this same age my own sister also got pretty much interested in astrology. I recall those days, we always used to talk about Astrology whenever possible which became more of a habit of ours. She took me to Science fairs and exhibitions whenever possible even though dad was away from us fighting the 30 years old war against LTTE terrorists at the front lines as an Army Officer. I’m still grateful for all her effort she did towards my every interest even though she had to manage everything by herself with no one to help around.

I used to build up my own simple Telescopes and star maps, where I would use every single night. I would just lie down on a mattress laid on the ground and stare at the sky for hours. I looked up for star patterns, stars, plants, and galaxies across. Something which was really fun was counting the number of asteroids I see everyday and whenever I saw any moving object in the sky I would just jump up and down shouting out, “Hello there mysterious space object, please come and take me to see the universe with you !”. Which was really funny when I later learned they were actually satellites.

I was so fond of the concept of Extra Terrestrial Intelligence, to be frank, Aliens. I recall my mom used to tell me stories of how it would be out there in the other planets and their living beings, their advanced intelligence, and specially how it would be like if they visit us. Sometimes we made up stories of what we would do if some aliens visits our house and so on which were pretty much our bed time stories throughout our childhood.

Those days, I would stare at the sky all by my self thinking how many worlds would be out there with beings of greater intelligence than us and are we actually alone in the Universe, why don’t they visit us in public. Sometimes I would secretly wish for a visit by Aliens to our me and take me to their planets to see the universe.

Thinking about the Universe when I was a kid, made me realize how small We are as Humans, having millions of planets, stars, galaxies out there. A huge part inside always whispered me, We are not Alone ! They will visit us Someday !

When the other kids were talking about Super heroes, or Cars, or their expensive Lame-ass toys, I was wondering and trying to solve this puzzle about Universe being a lonely silent kid.

How to Install Flashtool Drivers on Windows 8 (64bit)

Recently, well actually two days back when I installed Flashtool on my Windows 8 Laptop in order to flash my cutie pie, Xperia Ray phone 😛 I ran into a massive issue. After successfully installing Flashtool setup I headed up to installing flashtool drivers. I selected the needed drivers from the list and executed with the process, but then suddenly the driver installation results had failed at the end. I tried running as Admin, restarting the Lap and I even downloaded the whole set up once again, which was pretty much retarded until i realized the issue and reinstalled the whole thingy. But still NO ! -_-

I was really frustrated and thought of making up my mind to use Flashtool on my Windows 7 boot, where as I have my laptop dual booted to Windows 8 and Windows 7. Then suddenly it occurred to me could this be a incompatibility of the drivers or some security issue, and i went through all the forums available on XDA developers and finally found some solution.  Here is the link to that thread – http://forum.xda-developers.com/showthread.php?t=1913245

You may have to go to the end of the thread to find the solution over there, but i though of leaving a small note about How I solved this issue.

The only solution left is by disabling the Driver Signature Enforcement, which is a security feature that comes with Windws 8 where it does not allow unofficial drivers to be installed in your PC.

And below is how I managed to solve this issue, and please be cautions that you have to restart the PC in the middle ! 😛

Step 1 – Install the Flashtool set up as usual

Step 2 – Hover over the mouse in your top right hand side corner of the screen and go to,

Settings -> Change PC Settings -> General

a

Over there Click on “Restart Now”, this is to restart the PC and edit the settings in order to disable the Driver Signature Enforcement.

1

Once you Restart, the above screen will Appear, from there select on “Troubleshoot”.

2

Next select “Advanced Options”,

3

Then select “Startup Settings”,

4

At the end Click on “Restart” button in order to go to next step.

Step 3 –

5

At the of that long process this screen will pop up and from here you need to select the option you want to execute with. So in our case we need to disable the “Driver Signature Enforcement”, go ahead and press “F7” in your keyboard. 🙂

Step 4 – Then the PC will start up and you can go ahead with the usual  Flashtool Driver installation which is located in C:\Flashtool\drivers folder.

And if you want to re-enable Driver Signature Enforcement feature, you can go through the same process once again and enable it as you wish. But as a careless experimental IT Geek I would rather not go for it, because I don’t wanna go through the same burden process again and again ! 😛

Aaaaaand that was it ! BOOM ! xD

Oh oh I almost forgot, here is some video I found on Youtube of how to Disable DSE. 😀 Check it out if you are interested.

http://www.youtube.com/watch?v=NM1MN8QZhnk