Tutorial: Pybrain Neural network for classifying Olivetti faces

After experimenting various machine learning algorithms like KNN, Support vector machines (SVM), i decided to take a look on neural networks.  The Feed forward neural network really captured my interest. I really dont like reading too many papers and mathematics, so just decided to play around with the neural network to see how it works on face recognition. In this tutorial i am going to feed images to neural network classifier and the overall score is as high as 97% by letting the neural network do everything.

Goal

We train FNN neural network classifier on olivetti faces dataset. We aim to get the highest score of over 95% with the neural network with unseen test data.  We feed the raw image data to the neural network without doing much of feature extraction or dimensionality reduction on the image data.  After training, we test the network with new test data and see how well it predicts on new data.

What do you need?

  • Scikit-learn
  • Pybrain
  • Python with numpy, scipy and matplotlib
  • Olivetti faces dataset (we load using scikit)

Olivetti Dataset

First before we start, lets analyze the olivetti faces dataset to get a picture. The dataset consists of 400 images with greyscale 64×64 pixels. There are 10 images for each person, so there is 40 persons (target) which make ist 40×100 equals 400 rows of data.  This means the dimensionality is huge 64×64=4096 features. Lets import the dataset using scikit-learn built in function. The dataset is already normalized so you dont have to do preprocessing.

The first 16 faces in the olivetti dataset look like this.

faces

We load the data into our variables. This code will download the data automatically if not found locally.

from sklearn import datasets
olivetti = datasets.fetch_olivetti_faces()
X, y = olivetti.data, olivetti.target

Lets print the data shape of faces, X is input feature vector and y is the target of faces.

>>> X.shape
400,4096

 

Lets get started

Neural networks can learn any complex functions and the key is hidden layers in the neural network. Hidden layers work like automatic feature extractors when fed new data and can work towards the output. We create a FFNN Feedforward neural network with 4096 input neurons, just 1 hidden layer with 64 neurons and 1 output.

nn

We use Softmax layer function in the output neuron, Logistic sigmoid [0,1] in hidden layer neurons. For classification problems, it is recommended to use the Softmax layer for output and this one output should be binarized (one neuron output for each face) which has to be done internally. For example for 40 faces the NN produces converts 1 output to 40 binary output (1 neuron output for each face). I will explain this in the later stages of the code.

Lets create a  neural network using pybrain code in python

First load all the python modules

from pybrain.datasets            import ClassificationDataSet
from pybrain.utilities           import percentError
from pybrain.tools.shortcuts     import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.structure.modules   import SoftmaxLayer
from pybrain.tools.xml.networkwriter import NetworkWriter
from pybrain.tools.xml.networkreader import NetworkReader

then load the dataset into the neural network. The key here is we flatten the 64×64 data to one dimensional 4096 and then we feed the data our NN classification dataset:

ds = ClassificationDataSet(4096, 1 , nb_classes=40)
for k in xrange(len(X)): 
 ds.addSample(ravel(X[k]),y[k])

Next we split the data into 75% training and 25% test data.

tstdata, trndata = ds.splitWithProportion( 0.25 )

and this code converts 1 output to 40 binary outputs

trndata._convertToOneOfMany( )
tstdata._convertToOneOfMany( )

You can check the data inside the neural network like this

>>> print trndata['input'], trndata['target'], tstdata.indim, tstdata.outdim

Now that all the data is loaded into the neural network, we build the network and backpropagation trainer.

 fnn = buildNetwork( trndata.indim, 64 , trndata.outdim, outclass=SoftmaxLayer )
trainer = BackpropTrainer( fnn, dataset=trndata, momentum=0.1, learningrate=0.01 , verbose=True, weightdecay=0.01) 

Sometimes when you train for hours, you might want to save the data. Here is how to do it using the built in pybrain function.  Replace the above fnn variable with this. Here we check if the file exists we resume from where the training stopped by loading the file.

if  os.path.isfile('oliv.xml'): 
 fnn = NetworkReader.readFrom('oliv.xml') 
else:
 fnn = buildNetwork( trndata.indim, 64 , trndata.outdim, outclass=SoftmaxLayer )

and then put this line to write a file when the script ends.

NetworkWriter.writeToFile(fnn, 'oliv.xml')

Our neural network is ready for training. Lets start training and compute the percentage error on test data. Remember the network learns from training data and predicts on the test data. This means our network does not know about test data set.

We train our network for 50 epochs and compute the percentage error on test data. I am not gonna show the error on training data because it is usually about less than 2% and this not important. Sometimes the network overfit and memorize the data. The key here is test data set.

trainer.trainEpochs (50)
print 'Percent Error on Test dataset: ' , percentError( trainer.testOnClassData (
           dataset=tstdata )
           , tstdata['class'] )

Results

After training, as you can see from the below graph, I have achieved only 3% error on test data after 950 epochs and then after the error rate jumps up. This means our neural network score is 97% accurate and pretty great. Also note that after training 350 epochs, we get 5% error (ie 95%) which is not bad!!

results