Home Getting Started with Clarifai’s SDK
Author: Shashank Iyer
Clarifai is a company based in NY that develops powerful image and video recognition solutions. Their services include allowing users to train new models for image classification and efficiently search through a database of images. Their internally developed neural nets can scan a query image, analyze it and identify objects/concepts within it.
I began by creating an account a Clarifai account and installing the Clarifai package for Python. An API key will be made available on account creation. The Clarifai Explorer application enables users to upload images to their Clarifai database and manage customised models.
You can access the full code on my GitHub repository.
My first experiment with Calrifai was to create a model to classify Zener cards. I provided 2000 images to train this model and used five classes:
Representational images of Zener cards
Random changes were made in rotation, orientation and line width of the objects in the images. Ellipsoids were also added to them.
Clarifai uses K-fold cross-validation to train its models. I created a test set for the model and it achieved a 100% accuracy.
Make sure to upload the test images to the database for future reference and use.
Clarifai also provides a means to evaluate trained models. It tests the models on unlabelled data and generates confusion matrices, precision-recall and ROC curves.
A screenshot of the Clarifai Explorer
I created an application to perform visual search using Clarifai’s API. I created a custom database of images using Clarifai’s explorer. The database had 500 images of wildlife, vehicles, crowded streets and historic buildings. Every time the API is queried, it looks up a user-defined database for similar images. In response to a query, the API returns a list of images that are similar to the query image. Accompanying each image is a score that represents the confidence with which the algorithm feels that the query and database images are similar. A Python script running on my PC accepted query images. I performed a few experiments to check the accuracy of Clarifai’s algorithm. The following images represent these experiments:
Similarity search using a butterfly as a query image
The butterfly and bird are most closely matched because of the similarity in colour and features(such as wings). However, the algorithm predicts that they have a low match.
Similarity search using a van as a query image
The algorithm matches the van mostly with images containing automobiles. However, the query image has some greenery in its surroundings, as does the image of the person walking on the road. They are therefore matched but with a low score.
Similarity search using Calcutta High Court as a query image
The High Court building is matched with a high score with similar historical buildings. The score drops for other categories of images.
Clarifai has pre-trained models available for image classification. It provides some very accurate details about many of the images uploaded to the database. The following are snippets of those details:
Visual description of a liquid being poured into a glass
Visual description of the British Parliament
Such information is probably indicative that Clarifai is using a ResNet model with Transfer learning.
However, there were anomalies present in some of the images. The results of the classifier are as follows:
Visual description of the Indian Parliament
Visual description of people crossing a road
Even though these anomalies exist, the results are promising and show that Clarifai’s Neural Network architectures are very robust and can detect even minute details. In my next blog I will talk more on the ResNet model and KNN algorithms used for similarity search.
