Visualizing Embeddings Using t-SNE

 


In this blog I will talk about using Tensorboard to view image embeddings and provide some visual examples of clustering using Clarifai’s embeddings.
 

Installing Tensorboard

Tensorboard is installed along with Tensorflow. Run the following command to install Tensorflow:
 

pip install tensorflow


Visualise Embeddings with Tensorboard

Tensorboard provides the ability to view embeddings on its Projector. Users may select either PCA, t-SNE, or provide a custom algorithm to visualize embeddings. A few steps need to be followed to create the right files needed by the Projector.
 

Create a Tensorflow variable to store the embeddings. Configure a Projector object as shown below:
 

 

Steps to setup a projector
 

A metadata file is created to store the values of the embedding’s corresponding labels. Calling the Projector object’s visualize embeddings method makes it ready for the operation with the appropriate parameters passes. Finally, initialize and save the Tensorflow session as follows:
 

 

Create and save a metadata file
 

Run Tensorboard from the terminal:
 

 

Running Tensorboard from the terminal
 

 

Viewing embeddings using Tensorboard
 

Getting Clarifai’s embeddings

Clarifai’s ‘General’ model represents images as a vector of embeddings of size 1024. This model has a thousand labels to categorize images. Each image uploaded to a Clarifai database is assigned 20 labels based on its 20-nearest-neighbors in these 1024 dimensions.
 

I created an application to obtain embeddings for images using Clarifai’s API (To learn about more ways to use Clarifai’s API, please refer to my previous blog). I created a custom database of images using Clarifai’s explorer. The database had 500 images of wildlife, vehicles, crowded streets, and historic buildings.
 

Clarifai enables users to not only create custom models, but also use their pre-trained ones. My application uses the ‘embed’ type of Clarifai’s ‘General V1.3’ model. Every time the model predict API is queried, it forward propagates the query image through Clarifai’s ANN. In response to the query, the API returns an embedding vector that represents the query image.
 

A Python script running on my PC sent queries to obtain embedding vectors and the corresponding labels for each image in my custom database. I performed an experiment using t-SNE to check how well the embeddings represent the spatial distribution of the images.

The following images represent these experiments:

 

Wildlife image clustering by t-SNE
 

The t-SNE algorithm groups images of wildlife together. However, it also accurately groups them into sub-categories such as birds and animals.

 

Image distinction by t-SNE
 

In the image above, a clear distinction has been made between images of historic buildings and those of people walking on a street.
 

In my next blog, I will talk about reducing the dimensions of the image vector using triplet-loss while attempting to preserve its nearest neighbors.

Author:  Shashank Iyer