Modern Software Experience

2008-08-18

visual search

Some people call SearchMe, KartOO, Quintura or Viewzi visual search engines, but these are just text search engines with a more or less graphical user interface. SearchMe’s Cover Flow interface is nice, but it isn’t real visual search, it is just a visual browse of your text search results.
Pixsy is a text-based search engine for images, but Google Image Search gives more results.

Real visual search is hard, but a few companies are now claiming to be up to it. Here’s a quick overview of a bunch of search engines that represents the state of the art.

like.com

like.com has "visual search" as a tag line, and it is visual, but it seems to be shopping portal instead of a search engine. Well, it clearly is a shopping portal, and it’s probably great if you are looking to buy a particular pair of shoes. It is clear how like.com makes its money. Posts on the like.com blog are not technical, but frivolous. A typical post is some recent celeb snapshot followed by an itemised list of links to get same ugly white bag as Britney Spears.

Like does not offer visual search of the web. Your visual browsing is limited to the inventory of participating e-tail partners. I wondered what phones Like considers to be similar to the Apple iPhone, but my search for "Apple iPhone" returned zero results.

Like’s best feature is Like Upload, the ability to upload your own photo of your favourite item and then find the same or similar products, but alas, that feature is still "coming soon". Until Like features Like Upload, it does not offer visual search, but merely visual browse.

riya.com

Like.com, announced late in 2006, was started by riya.com, up until then known as photo search company focussed on facial recognition. If you visit riya.com, you get to see a search engine page with a huge add for like.com. I do not consider Like to be a true search engine, but one search engine advertising another one on their home page feels pretty weird. If you decided to stick around regardless, you find out that it does not work at all until you allow JavaScript.

When you allow JavaScript and search for "Paris Hilton" you get a whole bunch of pictures, and menu along the left that lets you focus the search on people, album, time, location and tags. Problem is, when you click any of these picture, you get to see a description, a star rating, tags, anything but the website it was taken from. You may add tags, comments, mark it as adult comment, add it to your blog or download it, but not visit the website. That’s because Riya is not a regular search engine. It does not link to websites at all.

Riya let’s you upload your photos and tag them. It will autotag some things, and after presenting it with some examples, it will recognise the faces in your photos and autotag these as well. You remain in control, and Riya lets you correct its mistakes by removing false positives from the tag set. Once you’ve tagged all your pictures, you can browse them by people, albums, time, location and text.

All this is free, but your pictures are now part of Riya’s database. Riya is not a search engine, but a web-hosted photo album with good search features. You can keep photos private. The search is not really visual, but based on tags you create assisted by image recognition routines - and that seems to be how like.com works too.

picitup

Picitup is in public beta since May. Picitup does compare pictures, but you do not need to provide a picture to get started. You start your search textually. Picitup then retrieves image from Yahoo! or Google, so it is actually piggy-backing on their search capabilities. Each picture has link to the original site under it, and that is the only part of the interface that simply works. Only when you enable JavaScript does picitup respond sensibly to the "Similar Picture" link it shows below each picture.

The "Similar Picture" search is very slow, and picitup attempts to run cross-site scripts from shopping.com, perhaps you let pick it up there. My search for pictures similar its first result for the Palm Treo 650 took about five to return only that picture itself. Taking five minutes to produce nothing is not impressive. The entire menu of visual filters on the left is rather superfluous for just one photo.

I tried something else entirely. I searched for "squirrel". I immediately noticed two results that really show the same picture of a squirrel apparently using a straw to drink from one of two beer cans, and asked the picitup to find similar pictures. It was faster this time, did return both pictures - and two others that do not have beer cans in them at all.

I searched for "sunflowers". The one but last result on the first page was a picture of Van Gogh’s painting, and the last result a tie printed with that painting. I clicked to get results similar to that. It does not return other printed ties, but many paintings instead. If you follow the link for that tie, you find links to the "Starry Night", "Outdoor Cafe at Night", and "Wheatfield & Cypress" neckties right there, and one click shows that they sell neckties with paintings from Claude Monet, Leonardo da Vinci, Sandro Botticelli, Auguste Rodin, Edvard Munch, Toulouse Lautrec, Salvador Dali, Paul Cezanne and Rembrandt van Rijn. And it is not just other Van Gogh ties, or more artist ties, there is a Van Gogh "Sunflowers" pin, and they sell Van Gogh umbrellas too. All clothing items with Van Gogh paintings on them, and picitup did not return any of this.

On a search for Van Gogh ties, like.com does a lot better. Just type "Van Gogh", and you get loads of Van Gogh ties, posters, perfumes, and even a handbag on the first page. It looks like every Van Gogh painting is available as a tie, but picitup didn’t pick that up.

TinEye

TinEye is a visual search engine by idée. Idée offers two other products, PixID and Piximilar, that are also based on its image recognition technology. This is so good, that Adobe is using Piximilar in Adobe PhotoShop Elements, digg uses its image identification to prevent duplicate image submissions, and several news agencies use PixID to track where their images appear in print or on the web.

TinEye became available on 2008 May 6 as an invitation-only service. TinEye is still in beta, and you need to register and login to use it, but the beta opened last Friday (2008 Aug 15), and it is direct registration now. It is worth the trouble, because TinEye is a real visual search engine.

TinEye does not search by text. You search for images by providing an image. You can upload an image or provide an URL, and it then searches for similar images. TinEye does not rely on text at all. It does not care about the pictures title or any metadata such as a title or tags. It analyses the pictures, stores some kind of digital fingerprint, and whenever you search, it returns all images with identical or similar fingerprints.

I pointed it to the Van Gogh "sunflowers" necktie, and it found no matches at all. The results page admits that their index is still relatively small. Well, no match is better than false matches. So, how about the beer drinking squirrel? Google finds multiple matches. TinEye finds 140 matches, all the exact same image, including copies that have been resized or cropped, or overlayed with some text or a logo. I browsed through the entire results set and I did not encounter a single false match. This is pretty impressive.

I decided to present something more challenging, the Donald Duck adaptation of Rembrandt’s Night watch, the Duckwatch. I found one through google and asked TinEye to find similar images. It found one other copy, and, most importantly, did not swamp me with results for Rembrandt’s original.
There is a photo of the 3D Night watch on the same page. TinEye did not turn up other results for that.

I tried the Internet explorer logo. The results start with a massive amount of exact matches in various sizes, followed by the logo without text, the logo with "Beta" on it, the logo without text and a "download" button over it, the logo as part of bigger picture, the logo in faded colours, the logo with a darker background, and so on.

Today, no search engine test is complete without a Paris Hilton search. I submitted the Go Away Paris background image to TinEye, and it found several copies, including one on a T-shirt, but the logo in the masthead of the Go Away Paris site was apparently either not in the index or too dissimilar to warrant a match. I wondered if any search might turn up the original picture the Go Away Paris logo is based on, but TinEye really looks for the logo, and picitup just isn’t smart enough.

TinEye has several Cool Searches canned for you, such as the Beatles’ famous Abbey Road cover. With all those searches, the most interesting and most impressive matches are near the end of each result set. The compare image feature, that let’s you toggle between two images is a nice gimmick.

TinEye seems so good, I installed their Firefox plug-in. There is one for Internet Explorer too.

updates

2008-09-13 GazoPa

Yet another visual search engine: GazoPa.

2011-04-23 broken links

The Go Away Paris: logo background is no more. The broken link has been removed.
The SearchMe search engine is defunct. The broken link has been removed.

2011-05-31 GazoPa

GazoPa will shut down on 2011-06-08.

2011-06-18 Pixsy

Pixsy seems to have been discontinued.

links

search engines

search examples