Getting Started with Clarifai's SDK

 

What is Clarifai?

Clarifai is a company based in NY that develops powerful image and video recognition solutions. Their services include allowing users to train new models for image classification and efficiently search through a database of images. Their internally developed neural nets can scan a query image, analyze it, and identify objects/concepts within it.
 

Getting Started

I began by creating an account a Clarifai account and installing the Clarifai package for Python. An API key will be made available on account creation. The Clarifai Explorer application enables users to upload images to their Clarifai database and manage customized models.
 

You can access the full code on my GitHub repository.
 

Training a model

My first experiment with Calrifai was to create a model to classify Zener cards. I provided 2000 images to train this model and used five classes:
 

  • • O : Circle
  • • P : Plus
  • • S : Square
  • • S : Star
  • • W : Wave
 
 
Representational images of Zener cards
 

Random changes were made in rotation, orientation, and line width of the objects in the images. Ellipsoids were also added to them.
 

Clarifai uses K-fold cross-validation to train its models. I created a test set for the model and it achieved a 100% accuracy.
 

Make sure to upload the test images to the database for future reference and use.
 

Clarifai also provides a means to evaluate trained models. It tests the models on unlabelled data and generates confusion matrices, precision-recall, and ROC curves.
 

 
A screenshot of the Clarifai Explorer
 

My First Search App

I created an application to perform visual search using Clarifai’s API. I created a custom database of images using Clarifai’s explorer. The database had 500 images of wildlife, vehicles, crowded streets and historic buildings. Every time the API is queried, it looks up a user-defined database for similar images. In response to a query, the API returns a list of images that are similar to the query image. Accompanying each image is a score that represents the confidence with which the algorithm feels that the query and database images are similar. A Python script running on my PC accepted query images. I performed a few experiments to check the accuracy of Clarifai’s algorithm. The following images represent these experiments:
 

 
Similarity search using a butterfly as a query image
 

The butterfly and bird are most closely matched because of the similarity in colour and features(such as wings). However, the algorithm predicts that they have a low match.
 

 
Similarity search using a van as a query image
 

The algorithm matches the van mostly with images containing automobiles. However, the query image has some greenery in its surroundings, as does the image of the person walking on the road. They are, therefore, matched but with a low score.
 

 
Similarity search using Calcutta High Court as a query image
 

The High Court building is matched with a high score with similar historical buildings. The score drops for other categories of images.
 

Clarifai has pre-trained models available for image classification. It provides some very accurate details about many of the images uploaded to the database. The following are snippets of those details:
 

 
Visual description of a liquid being poured into a glass
 
 
Visual description of the Britist Parliament
 

Such information is probably indicative that Clarifai is using a ResNet model with Transfer learning.
 

However, there were anomalies present in some of the images. The results of the classifier are as follows:
 

 
Visual description of the Indian Parliament
 
 
Visual description of people crossing a road
 

Even though these anomalies exist, the results are promising and show that Clarifai’s Neural Network architectures are very robust and can detect even minute details. In my next blog I will talk more on the ResNet model and KNN algorithms used for similarity search.
 

Author:  Shashank Iyer