After experimenting various machine learning algorithms like KNN, Support vector machines (SVM), i decided to take a look on neural networks. The Feed forward neural network really captured my interest. I really dont like reading too many papers and mathematics, so just decided to play around with the neural network to see how it works on face recognition. In this tutorial i am going to feed images to neural network classifier and the overall score is as high as 97% by letting the neural network do everything.
Goal
We train FNN neural network classifier on olivetti faces dataset. We aim to get the highest score of over 95% with the neural network with unseen test data. We feed the raw image data to the neural network without doing much of feature extraction or dimensionality reduction on the image data. After training, we test the network with new test data and see how well it predicts on new data.
What do you need?
- Scikit-learn
- Pybrain
- Python with numpy, scipy and matplotlib
- Olivetti faces dataset (we load using scikit)
Olivetti Dataset
First before we start, lets analyze the olivetti faces dataset to get a picture. The dataset consists of 400 images with greyscale 64×64 pixels. There are 10 images for each person, so there is 40 persons (target) which make ist 40×100 equals 400 rows of data. This means the dimensionality is huge 64×64=4096 features. Lets import the dataset using scikit-learn built in function. The dataset is already normalized so you dont have to do preprocessing.
The first 16 faces in the olivetti dataset look like this.
We load the data into our variables. This code will download the data automatically if not found locally.
from sklearn import datasets olivetti = datasets.fetch_olivetti_faces() X, y = olivetti.data, olivetti.target
Lets print the data shape of faces, X is input feature vector and y is the target of faces.
>>> X.shape 400,4096
Lets get started
Neural networks can learn any complex functions and the key is hidden layers in the neural network. Hidden layers work like automatic feature extractors when fed new data and can work towards the output. We create a FFNN Feedforward neural network with 4096 input neurons, just 1 hidden layer with 64 neurons and 1 output.
We use Softmax layer function in the output neuron, Logistic sigmoid [0,1] in hidden layer neurons. For classification problems, it is recommended to use the Softmax layer for output and this one output should be binarized (one neuron output for each face) which has to be done internally. For example for 40 faces the NN produces converts 1 output to 40 binary output (1 neuron output for each face). I will explain this in the later stages of the code.
Lets create a neural network using pybrain code in python
First load all the python modules
from pybrain.datasets import ClassificationDataSet from pybrain.utilities import percentError from pybrain.tools.shortcuts import buildNetwork from pybrain.supervised.trainers import BackpropTrainer from pybrain.structure.modules import SoftmaxLayer from pybrain.tools.xml.networkwriter import NetworkWriter from pybrain.tools.xml.networkreader import NetworkReader
then load the dataset into the neural network. The key here is we flatten the 64×64 data to one dimensional 4096 and then we feed the data our NN classification dataset:
ds = ClassificationDataSet(4096, 1 , nb_classes=40) for k in xrange(len(X)): ds.addSample(ravel(X[k]),y[k])
Next we split the data into 75% training and 25% test data.
tstdata, trndata = ds.splitWithProportion( 0.25 )
and this code converts 1 output to 40 binary outputs
trndata._convertToOneOfMany( ) tstdata._convertToOneOfMany( )
You can check the data inside the neural network like this
>>> print trndata['input'], trndata['target'], tstdata.indim, tstdata.outdim
Now that all the data is loaded into the neural network, we build the network and backpropagation trainer.
fnn = buildNetwork( trndata.indim, 64 , trndata.outdim, outclass=SoftmaxLayer ) trainer = BackpropTrainer( fnn, dataset=trndata, momentum=0.1, learningrate=0.01 , verbose=True, weightdecay=0.01)
Sometimes when you train for hours, you might want to save the data. Here is how to do it using the built in pybrain function. Replace the above fnn variable with this. Here we check if the file exists we resume from where the training stopped by loading the file.
if os.path.isfile('oliv.xml'): fnn = NetworkReader.readFrom('oliv.xml') else: fnn = buildNetwork( trndata.indim, 64 , trndata.outdim, outclass=SoftmaxLayer )
and then put this line to write a file when the script ends.
NetworkWriter.writeToFile(fnn, 'oliv.xml')
Our neural network is ready for training. Lets start training and compute the percentage error on test data. Remember the network learns from training data and predicts on the test data. This means our network does not know about test data set.
We train our network for 50 epochs and compute the percentage error on test data. I am not gonna show the error on training data because it is usually about less than 2% and this not important. Sometimes the network overfit and memorize the data. The key here is test data set.
trainer.trainEpochs (50) print 'Percent Error on Test dataset: ' , percentError( trainer.testOnClassData ( dataset=tstdata ) , tstdata['class'] )
Results
After training, as you can see from the below graph, I have achieved only 3% error on test data after 950 epochs and then after the error rate jumps up. This means our neural network score is 97% accurate and pretty great. Also note that after training 350 epochs, we get 5% error (ie 95%) which is not bad!!