Visual Search: The Future of Search

Author: Pat Lasserre

Introduction

If you’ve ever gone through the frustration of trying different ways to describe an item through text-based search and still couldn’t find what you were looking for, then visual search might be for you.

With visual search, finding exactly what you’re looking for is as easy as taking a picture on your phone.

According to a new report by the market research company Global Market Insights, Inc., visual search is forecasted to grow at a CAGR of over 45% during the period of 2018- 2024.

This blog post is an introduction to a series of posts that will cover visual search and the machine learning application of similarity search in general.

Making Search Easy
Visual search allows you to replace your keyboard with your camera phone by using images instead of text to search for things.

For example, as seen in the picture below from Target, if you see a particular vase that you like, all you need to do is take a picture of the vase and visual search will help you find the same vase or a similar vase from Target’s catalog.

Figure 1. An example of using visual search to find a particular vase on the Target website.

Provides More Relevant Results
In addition to simplifying search, visual search also returns more accurate results. Oftentimes, products are difficult to describe using traditional text-based search, and visual search provides a very explicit way to search, allowing for more accurate results.

Visual Search can also help in cases where a product has been mislabeled or if a set of keywords you’re using to search were not assigned to the product description. If it was mislabeled, a text-base search would return the wrong product.

Visual Search in Retail
Visual search is ideal for retail segments, such as fashion and home design because they are largely driven by visual content, and style is often difficult to describe using text search.

Wayfair is an example of a home design company that makes it easy for you to search their catalog by using a picture that you have taken.

Discovering New Products and Ideas
In addition to visually searching their catalog for similar items, the Wayfair visual search application also allows for discovery of cross-category products. The application looks at your browse history (context) to provide recommendations for items in different categories. For example, it might recommend a nightstand with a similar style to the style of a dresser you recently browsed.

Figure 2. An example from Wayfair of cross-category discovery where rustic dressers and headboards (left) serve as the browse context
and a cross-class recommendation for rustic nightstands (right) is returned.

Pinterest is another company that lets you make discoveries using pictures. They mention an example where you can use their Lens visual search application for discovery around the topic of a blueberry. In their example, if you take a picture of a blueberry, in addition to providing similar images of blueberries, Lens provides more diverse results, for example, returning recipes for blueberry scones and smoothies.

Performance and Scale
Visual search clearly holds a lot of promise, but there are a few challenges. The highly relevant content needs to be returned with low latency in order to provide a good user experience, and this needs to be done at scale.

For example, companies such as eBay, Microsoft, and Wayfair mention the need to provide visual search results with latency on the order of 100ms.

In terms of scale, eBay has over a billion items in their catalog, Microsoft Bingmentions the need for web-scale visual search for tens of billions of images, and Pinterest states that they have over 100 billion ideas to search from.

Conclusion
With visual search, you don’t have to think of the right words to describe an item; you can just use pictures to make it easy to find what you’re looking for.

Many people believe that visual search will change the way we search, as evidenced by the following quote from Pinterest co-founder and CEO Ben Silbermann in a CNBC interview, “A lot of the future of search is going to be about pictures instead of keywords.”

Some of the challenges facing visual search solutions are scaling to billions of items and providing latencies of around 100ms. Future posts in this series will introduce machine learning methods, such as similarity search, that address these challenges.