Running a neural network in GPU

I was so passionate about neural networks and always wanted a fast library. I found FANN library c is the fastest NN library, so that i could train a neural network in GPU. I finally managed to find libcudann library (extension of FANN) which runs on Nvidia GPU with CUDA. I tested on my ubuntu machine running intel i5 quad core processor and Nvidia 750Ti (620 cuda cores). with  CUDA and openCL.

From what i have tested, FANN library running on CPU is 10X faster than pybrain and 100X faster running in GPU. Caffe deep neural network library is the fastest i have seen 200X faster on MNIST when run on nvidia cuda architecture supported GPU.  I decided to run MNIST handwriting data on my neural network. After GPU training of about 10,000 epochs, I was able to get 0.003 error on test data within few minutes ~3min, which is pretty amazing. The inputs were normalized between -1 and 1

figure_1

The python file writes the MNIST images to be in desired format for fann library.

import numpy as np
from sklearn import datasets, preprocessing, cross_validation
from numpy import array_str

from sklearn.datasets import fetch_mldata
mnist = fetch_mldata('MNIST original')

X,y = mnist.data, mnist.target

X.astype("float32")

#Grayscale between -1 and 1
X = X/255.0*2 - 1

outputs = max(y) + 1

X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.4, random_state=0)

ytrain = preprocessing.LabelBinarizer().fit_transform(y_train)
ytest =  preprocessing.LabelBinarizer().fit_transform(y_test)


# Writing training data
f = open("mnist.train", "w")
f.write ( str(X_train.shape[0]) + ' ' + str(X_train.shape[1]) + ' ' + str(outputs) + '\n' )

for k in range(len(X_train)): 
 f.write( ' '.join(map(str, X_train[k] )) + '\n' )     
 f.write( ' '.join(map(str, ytrain[k] )) + '\n' ) 

f.close()

# Writing test data

f = open("mnist.test", "w")
f.write ( str(X_test.shape[0]) + ' ' + str(X_test.shape[1]) + ' ' + str(outputs) + '\n' )

for k in range(len(X)): 
 f.write( ' '.join(map(str, X_test[k] )) + '\n' )     
 f.write( ' '.join(map(str, ytest[k] )) + '\n' ) 
f.close()

This outputs mnis.train and mnist.test and the output below..

pbu@pbu-desktop:~/Desktop/libcudann$ ./execute
Network: 784x60x10
Activation functions: 2 1 2 
Training on: GPU
Algorithm: Batch
Desired Error: 0.000000
Max epochs: 10000
Epochs between reports: 100
Learning rate: 0.100000
Momentum: 0.000000
Shuffle: ON
Error function: TANH
Starting: Error on train set 0.0563584156 Error on test set 0.0563257895
Epoch 100 Error on train set 0.0158128124 Error on test set 0.0157885905
Epoch 200 Error on train set 0.0132353399 Error on test set 0.0132152764
Epoch 300 Error on train set 0.0119769620 Error on test set 0.0119561991
Epoch 400 Error on train set 0.0112424437 Error on test set 0.0112195630
Epoch 500 Error on train set 0.0107437009 Error on test set 0.0107200257
Epoch 600 Error on train set 0.0103605166 Error on test set 0.0103376042
Epoch 700 Error on train set 0.0100403074 Error on test set 0.0100191962
Epoch 800 Error on train set 0.0097571611 Error on test set 0.0097383494
Epoch 900 Error on train set 0.0094972840 Error on test set 0.0094809681
Epoch 1000 Error on train set 0.0092534767 Error on test set 0.0092397314
Epoch 1100 Error on train set 0.0090220291 Error on test set 0.0090108840
Epoch 1200 Error on train set 0.0088008856 Error on test set 0.0087923482
Epoch 1300 Error on train set 0.0085888570 Error on test set 0.0085829040
Epoch 1400 Error on train set 0.0083853239 Error on test set 0.0083819143
Epoch 1500 Error on train set 0.0081900833 Error on test set 0.0081891529
Epoch 1600 Error on train set 0.0080031529 Error on test set 0.0080046197
Epoch 1700 Error on train set 0.0078245914 Error on test set 0.0078283595
Epoch 1800 Error on train set 0.0076543707 Error on test set 0.0076603359
Epoch 1900 Error on train set 0.0074923150 Error on test set 0.0075003696
Epoch 2000 Error on train set 0.0073381159 Error on test set 0.0073481603
Epoch 2100 Error on train set 0.0071913525 Error on test set 0.0072032907
Epoch 2200 Error on train set 0.0070515368 Error on test set 0.0070652841
Epoch 2300 Error on train set 0.0069181491 Error on test set 0.0069336332
Epoch 2400 Error on train set 0.0067906706 Error on test set 0.0068078293
Epoch 2500 Error on train set 0.0066686082 Error on test set 0.0066873883
Epoch 2600 Error on train set 0.0065515204 Error on test set 0.0065718819
Epoch 2700 Error on train set 0.0064390311 Error on test set 0.0064609414
Epoch 2800 Error on train set 0.0063308338 Error on test set 0.0063542691
Epoch 2900 Error on train set 0.0062266998 Error on test set 0.0062516420
Epoch 3000 Error on train set 0.0061264569 Error on test set 0.0061528925
Epoch 3100 Error on train set 0.0060299933 Error on test set 0.0060579111
Epoch 3200 Error on train set 0.0059372275 Error on test set 0.0059666205
Epoch 3300 Error on train set 0.0058480934 Error on test set 0.0058789537
Epoch 3400 Error on train set 0.0057625249 Error on test set 0.0057948455
Epoch 3500 Error on train set 0.0056804456 Error on test set 0.0057142195
Epoch 3600 Error on train set 0.0056017553 Error on test set 0.0056369752
Epoch 3700 Error on train set 0.0055263452 Error on test set 0.0055630011
Epoch 3800 Error on train set 0.0054540779 Error on test set 0.0054921624
Epoch 3900 Error on train set 0.0053848140 Error on test set 0.0054243156
Epoch 4000 Error on train set 0.0053184042 Error on test set 0.0053593121
Epoch 4100 Error on train set 0.0052546998 Error on test set 0.0052970033
Epoch 4200 Error on train set 0.0051935427 Error on test set 0.0052372292
Epoch 4300 Error on train set 0.0051347865 Error on test set 0.0051798457
Epoch 4400 Error on train set 0.0050782855 Error on test set 0.0051247063
Epoch 4500 Error on train set 0.0050239097 Error on test set 0.0050716810
Epoch 4600 Error on train set 0.0049715303 Error on test set 0.0050206413
Epoch 4700 Error on train set 0.0049210321 Error on test set 0.0049714693
Epoch 4800 Error on train set 0.0048723132 Error on test set 0.0049240668
Epoch 4900 Error on train set 0.0048252800 Error on test set 0.0048783384
Epoch 5000 Error on train set 0.0047798492 Error on test set 0.0048341998
Epoch 5100 Error on train set 0.0047359481 Error on test set 0.0047915797
Epoch 5200 Error on train set 0.0046935058 Error on test set 0.0047504064
Epoch 5300 Error on train set 0.0046524643 Error on test set 0.0047106217
Epoch 5400 Error on train set 0.0046127713 Error on test set 0.0046721748
Epoch 5500 Error on train set 0.0045743682 Error on test set 0.0046350062
Epoch 5600 Error on train set 0.0045372057 Error on test set 0.0045990688
Epoch 5700 Error on train set 0.0045012361 Error on test set 0.0045643141
Epoch 5800 Error on train set 0.0044664103 Error on test set 0.0045306929
Epoch 5900 Error on train set 0.0044326847 Error on test set 0.0044981632
Epoch 6000 Error on train set 0.0044000074 Error on test set 0.0044666729
Epoch 6100 Error on train set 0.0043683355 Error on test set 0.0044361809
Epoch 6200 Error on train set 0.0043376205 Error on test set 0.0044066384
Epoch 6300 Error on train set 0.0043078279 Error on test set 0.0043780110
Epoch 6400 Error on train set 0.0042789057 Error on test set 0.0043502459
Epoch 6500 Error on train set 0.0042508165 Error on test set 0.0043233084
Epoch 6600 Error on train set 0.0042235283 Error on test set 0.0042971647
Epoch 6700 Error on train set 0.0041969963 Error on test set 0.0042717727
Epoch 6800 Error on train set 0.0041711926 Error on test set 0.0042471024
Epoch 6900 Error on train set 0.0041460851 Error on test set 0.0042231218
Epoch 7000 Error on train set 0.0041216379 Error on test set 0.0041997982
Epoch 7100 Error on train set 0.0040978277 Error on test set 0.0041771047
Epoch 7200 Error on train set 0.0040746210 Error on test set 0.0041550109
Epoch 7300 Error on train set 0.0040519978 Error on test set 0.0041334946
Epoch 7400 Error on train set 0.0040299348 Error on test set 0.0041125333
Epoch 7500 Error on train set 0.0040084026 Error on test set 0.0040920973
Epoch 7600 Error on train set 0.0039873808 Error on test set 0.0040721665
Epoch 7700 Error on train set 0.0039668498 Error on test set 0.0040527219
Epoch 7800 Error on train set 0.0039467923 Error on test set 0.0040337448
Epoch 7900 Error on train set 0.0039271922 Error on test set 0.0040152199
Epoch 8000 Error on train set 0.0039080293 Error on test set 0.0039971280
Epoch 8100 Error on train set 0.0038892890 Error on test set 0.0039794510
Epoch 8200 Error on train set 0.0038709559 Error on test set 0.0039621773
Epoch 8300 Error on train set 0.0038530135 Error on test set 0.0039452892
Epoch 8400 Error on train set 0.0038354506 Error on test set 0.0039287745
Epoch 8500 Error on train set 0.0038182531 Error on test set 0.0039126198
Epoch 8600 Error on train set 0.0038014064 Error on test set 0.0038968106
Epoch 8700 Error on train set 0.0037848980 Error on test set 0.0038813341
Epoch 8800 Error on train set 0.0037687204 Error on test set 0.0038661817
Epoch 8900 Error on train set 0.0037528600 Error on test set 0.0038513422
Epoch 9000 Error on train set 0.0037373099 Error on test set 0.0038368078
Epoch 9100 Error on train set 0.0037220509 Error on test set 0.0038225586
Epoch 9200 Error on train set 0.0037070781 Error on test set 0.0038085904
Epoch 9300 Error on train set 0.0036923850 Error on test set 0.0037948966
Epoch 9400 Error on train set 0.0036779642 Error on test set 0.0037814688
Epoch 9500 Error on train set 0.0036638069 Error on test set 0.0037683002
Epoch 9600 Error on train set 0.0036498983 Error on test set 0.0037553732
Epoch 9700 Error on train set 0.0036362333 Error on test set 0.0037426862
Epoch 9800 Error on train set 0.0036228085 Error on test set 0.0037302342
Epoch 9900 Error on train set 0.0036096165 Error on test set 0.0037180081
Epoch 10000 Error on train set 0.0035966476 Error on test set 0.0037060005
Training complete.

Similar Posts:

Balakrishnan Prabhu

Mr. Balakrishnan Prabhu is the founder of Corpocrat magazine. He is also the founder of Best Citizenships (BC), assisting wealthy individuals with with global citizenship and residency programs in Europe. His other interests are Linux, Machine learning, Wordpress, etc. You can contact him here