Reverse Image Search

What is Reverse Image Search?

Reverse image search is a function found in some search engines that lets you find webpages that contain an image by uploading the image to the search engine. In some cases, it's also possible to find other things besides webpages by their images.

The reason reverse image search is called "reverse" image search is because in normal image search you type a text description to find an image, while in reverse image search you already have the image, and what you're looking for is information about the image instead. For example, if you type "tulip" in an image search engine, you'll find images of tulips, while if you upload an image of a tulip, even if you don't know the name of the plant, a reverse image search engine will try to find webpages that contain that image, and in those webpages you'll learn what is the name of the plant you took a photo of, as well as other information.

What are some Reverse Image Search Engines?

Many search engines that offer web search and normal image search also offer reverse image search, for example: Google, Bing, and Yandex all offer reverse image search. Kagi claims to offer it, but I personally couldn't get it to work.

There are also specialized reverse image search engines, such as TinEye (https://tineye.com/).

How to Use It

In general, reverse image search engines let you upload images by clicking on a little camera icon button in the image search page, that is typically at the right side of the search box where you would type a text to search for images normally. It's also typically possible to draw and drop an image into image search page to search for an image. In particular, you don't need to drag a file, you can also drag an image from the web browser, from a webpage opened in a window to the reverse image search page in another window. It's also possible to paste the URL of an image you can get by right clicking on an image in a web browser. Lastly, it's possible to paste an image to search for by pressing Ctrl+V while the the field to paste a URL has keyboard focus.

How it Works

There are many ways to implement reverse image search, so the results you get depend on the algorithm implemented as well as the size of the index of the search engine. If you're really trying to find an image, it doesn't hurt to just try every single reverse image search engine available.

The simplest method would be to just store a MD5 hash of every image file and then compare that hash to the hash of any uploaded file. In this case, the only way for a match to occur is if both images are exactly the same image file. If you rotate the image or change its size, or even saved it as JPG again, it would change the bytes of the file and the colors slightly, so it wouldn't get a match.

A more complex method is using a perceptive hash instead of a data hash. In this case, instead of looking at the bytes of the image, we look at the colors of the pixels and create a very tiny 1 bit per pixel image of fixed size (e.g. 32x32 pixels) that represents the colors of the whole thing. We convert all images, regardless of size, to that 32x32 pixel size. 32 times 32 is 1024, so all we need to do is get the hamming distance between two 1024-bit sequences to figure how different two images are from each other. A factor in this case is the hashing algorithm, there are various: phash, ahash, dhash, etc., each with its own strengths and weaknesses, and they're capable of handling images being rotated and mirrored as well. This works well enough for many applications, but it's not enough for some cases.

With the method above, if you uploaded an image that's mostly blue, you would get results that are mostly blue images, regardless of what is contained in the image. This is good if you're trying to find images that are visually similar to each other. For example, if you find an image of a webcomic and you want to find what's the official website of that webcomic, you can just upload it to a reverse image search engine and it will be able to find everything that looks visually similar to that image.

However, this won't work if you just take your phone and take a photo of a random thing and try to find what it is. Because you took the photo, there is no photo exactly like it on the Internet yet for the search engine to have indexed it. To make this work, so you can just take a photo of whatever and let Google tell you what it is, something more sophisticate is necessary.

The more complex reverse image search engines use computer vision (i.e. "AI") to classify images. Essentially, they try to figure out what is actually contained in the image based on the shapes of objects and changes in the colors of the pixels. While this is good for finding things, it's bad for finding specific images on the web.

Notably, Google decided to add this to Google Chrome, changing the old right click menu that let you search for image, to opening a weird sidebar that uses Google Lens. This Google Lens feature uses AI, so it's very good when you want to find a product in a shopping website from a completely different photo of the same product, but it's not as good when you want to find the source of an image on the Internet.