Visualizing Embeddings Using t-SNE

Author: Shashank Iyer

In this blog I will talk about using Tensorboard to view image embeddings and provide some visual examples of clustering using Clarifai’s embeddings.

Installing Tensorboard

Tensorboard is installed along with Tensorflow. Run the following command to install Tensorflow:

pip install tensorflow

Visualize Embeddings with Tensorboard

Tensorboard provides the ability to view embeddings on it’s Projector. Users may select either PCA, t-SNE or provide a custom algorithm to visualize embeddings. A few steps are to be followed to create the right files needed by the Projector.

Create a Tensorflow variable to store the embeddings. Configure a Projector object as shown below:

Steps to setup a projector

A metadata file is to be created to store the values of the embedding’s corresponding labels. Calling the Projector object’s visualize embeddings method makes it ready for the operation with the appropriate parameters passes. Finally, initialize and save the Tensorflow session as follows:

Create and save a metadata file

Run Tensorboard from the terminal:

Running Tensorboard from the terminal

Viewing embeddings using Tensorboard

Getting Clarifai’s Embeddings

Clarifai’s ‘General’ model represents images as a vector of embeddings of size 1024. This model has a thousand labels to categorise images. Each image uploaded to a Clarifai database is assigned 20 labels based on it’s 20-nearest-neighbours in these 1024 dimensions.

I created an application to obtain embeddings for images using Clarifai’s API (To learn about more ways to use Clarifai’s API, please refer to my previous blog). I created a custom database of images using Clarifai’s explorer. The database had 500 images of wildlife, vehicles, crowded streets and historic buildings.

Clarifai enables users to not only create custom models but also use their pre-trained ones. My application uses the ‘embed’ type of Clarifai’s ‘General V1.3’ model. Every time the model predict API is queried, it forward propagates the query image through Clarifai’s ANN. In response to the query, the API returns an embedding vector that represents the query image.

A Python script running on my PC sent queries to obtain embedding vectors and the corresponding labels for each image in my custom database. I performed an experiment using t-SNE to check how well the embeddings represent the spatial distribution of the images. The following images represent these experiments:

Wildlife image clustering by t-SNE

The t-SNE algorithm groups images of wildlife together. However, it also accurately groups them into sub-categories such as birds and animals.

Image distinction by t-SNE

In the image above, a clear distinction has been made between images of historic buildings and those of people walking on a street.

In my next blog, I will talk about reducing the dimensions of the image vector using triplet-loss while attempting to preserve it’s nearest neighbours.