I was so passionate about neural networks and always wanted a fast library. I found FANN library c is the fastest NN library, so that i could train a neural network in GPU. I finally managed to find libcudann library (extension of FANN) which runs on Nvidia GPU with CUDA. I tested on my ubuntu machine running intel i5 quad core processor and Nvidia 750Ti (620 cuda cores). with CUDA and openCL.
From what i have tested, FANN library running on CPU is 10X faster than pybrain and 100X faster running in GPU. Caffe deep neural network library is the fastest i have seen 200X faster on MNIST when run on nvidia cuda architecture supported GPU. I decided to run MNIST handwriting data on my neural network. After GPU training of about 10,000 epochs, I was able to get 0.003 error on test data within few minutes ~3min, which is pretty amazing. The inputs were normalized between -1 and 1
The python file writes the MNIST images to be in desired format for fann library.
import numpy as np
from sklearn import datasets, preprocessing, cross_validation
from numpy import array_str
from sklearn.datasets import fetch_mldata
mnist = fetch_mldata('MNIST original')
X,y = mnist.data, mnist.target
X.astype("float32")
#Grayscale between -1 and 1
X = X/255.0*2 - 1
outputs = max(y) + 1
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.4, random_state=0)
ytrain = preprocessing.LabelBinarizer().fit_transform(y_train)
ytest = preprocessing.LabelBinarizer().fit_transform(y_test)
# Writing training data
f = open("mnist.train", "w")
f.write ( str(X_train.shape[0]) + ' ' + str(X_train.shape[1]) + ' ' + str(outputs) + '\n' )
for k in range(len(X_train)):
f.write( ' '.join(map(str, X_train[k] )) + '\n' )
f.write( ' '.join(map(str, ytrain[k] )) + '\n' )
f.close()
# Writing test data
f = open("mnist.test", "w")
f.write ( str(X_test.shape[0]) + ' ' + str(X_test.shape[1]) + ' ' + str(outputs) + '\n' )
for k in range(len(X)):
f.write( ' '.join(map(str, X_test[k] )) + '\n' )
f.write( ' '.join(map(str, ytest[k] )) + '\n' )
f.close()
This outputs mnis.train and mnist.test and the output below..
pbu@pbu-desktop:~/Desktop/libcudann$ ./execute Network: 784x60x10 Activation functions: 2 1 2 Training on: GPU Algorithm: Batch Desired Error: 0.000000 Max epochs: 10000 Epochs between reports: 100 Learning rate: 0.100000 Momentum: 0.000000 Shuffle: ON Error function: TANH Starting: Error on train set 0.0563584156 Error on test set 0.0563257895 Epoch 100 Error on train set 0.0158128124 Error on test set 0.0157885905 Epoch 200 Error on train set 0.0132353399 Error on test set 0.0132152764 Epoch 300 Error on train set 0.0119769620 Error on test set 0.0119561991 Epoch 400 Error on train set 0.0112424437 Error on test set 0.0112195630 Epoch 500 Error on train set 0.0107437009 Error on test set 0.0107200257 Epoch 600 Error on train set 0.0103605166 Error on test set 0.0103376042 Epoch 700 Error on train set 0.0100403074 Error on test set 0.0100191962 Epoch 800 Error on train set 0.0097571611 Error on test set 0.0097383494 Epoch 900 Error on train set 0.0094972840 Error on test set 0.0094809681 Epoch 1000 Error on train set 0.0092534767 Error on test set 0.0092397314 Epoch 1100 Error on train set 0.0090220291 Error on test set 0.0090108840 Epoch 1200 Error on train set 0.0088008856 Error on test set 0.0087923482 Epoch 1300 Error on train set 0.0085888570 Error on test set 0.0085829040 Epoch 1400 Error on train set 0.0083853239 Error on test set 0.0083819143 Epoch 1500 Error on train set 0.0081900833 Error on test set 0.0081891529 Epoch 1600 Error on train set 0.0080031529 Error on test set 0.0080046197 Epoch 1700 Error on train set 0.0078245914 Error on test set 0.0078283595 Epoch 1800 Error on train set 0.0076543707 Error on test set 0.0076603359 Epoch 1900 Error on train set 0.0074923150 Error on test set 0.0075003696 Epoch 2000 Error on train set 0.0073381159 Error on test set 0.0073481603 Epoch 2100 Error on train set 0.0071913525 Error on test set 0.0072032907 Epoch 2200 Error on train set 0.0070515368 Error on test set 0.0070652841 Epoch 2300 Error on train set 0.0069181491 Error on test set 0.0069336332 Epoch 2400 Error on train set 0.0067906706 Error on test set 0.0068078293 Epoch 2500 Error on train set 0.0066686082 Error on test set 0.0066873883 Epoch 2600 Error on train set 0.0065515204 Error on test set 0.0065718819 Epoch 2700 Error on train set 0.0064390311 Error on test set 0.0064609414 Epoch 2800 Error on train set 0.0063308338 Error on test set 0.0063542691 Epoch 2900 Error on train set 0.0062266998 Error on test set 0.0062516420 Epoch 3000 Error on train set 0.0061264569 Error on test set 0.0061528925 Epoch 3100 Error on train set 0.0060299933 Error on test set 0.0060579111 Epoch 3200 Error on train set 0.0059372275 Error on test set 0.0059666205 Epoch 3300 Error on train set 0.0058480934 Error on test set 0.0058789537 Epoch 3400 Error on train set 0.0057625249 Error on test set 0.0057948455 Epoch 3500 Error on train set 0.0056804456 Error on test set 0.0057142195 Epoch 3600 Error on train set 0.0056017553 Error on test set 0.0056369752 Epoch 3700 Error on train set 0.0055263452 Error on test set 0.0055630011 Epoch 3800 Error on train set 0.0054540779 Error on test set 0.0054921624 Epoch 3900 Error on train set 0.0053848140 Error on test set 0.0054243156 Epoch 4000 Error on train set 0.0053184042 Error on test set 0.0053593121 Epoch 4100 Error on train set 0.0052546998 Error on test set 0.0052970033 Epoch 4200 Error on train set 0.0051935427 Error on test set 0.0052372292 Epoch 4300 Error on train set 0.0051347865 Error on test set 0.0051798457 Epoch 4400 Error on train set 0.0050782855 Error on test set 0.0051247063 Epoch 4500 Error on train set 0.0050239097 Error on test set 0.0050716810 Epoch 4600 Error on train set 0.0049715303 Error on test set 0.0050206413 Epoch 4700 Error on train set 0.0049210321 Error on test set 0.0049714693 Epoch 4800 Error on train set 0.0048723132 Error on test set 0.0049240668 Epoch 4900 Error on train set 0.0048252800 Error on test set 0.0048783384 Epoch 5000 Error on train set 0.0047798492 Error on test set 0.0048341998 Epoch 5100 Error on train set 0.0047359481 Error on test set 0.0047915797 Epoch 5200 Error on train set 0.0046935058 Error on test set 0.0047504064 Epoch 5300 Error on train set 0.0046524643 Error on test set 0.0047106217 Epoch 5400 Error on train set 0.0046127713 Error on test set 0.0046721748 Epoch 5500 Error on train set 0.0045743682 Error on test set 0.0046350062 Epoch 5600 Error on train set 0.0045372057 Error on test set 0.0045990688 Epoch 5700 Error on train set 0.0045012361 Error on test set 0.0045643141 Epoch 5800 Error on train set 0.0044664103 Error on test set 0.0045306929 Epoch 5900 Error on train set 0.0044326847 Error on test set 0.0044981632 Epoch 6000 Error on train set 0.0044000074 Error on test set 0.0044666729 Epoch 6100 Error on train set 0.0043683355 Error on test set 0.0044361809 Epoch 6200 Error on train set 0.0043376205 Error on test set 0.0044066384 Epoch 6300 Error on train set 0.0043078279 Error on test set 0.0043780110 Epoch 6400 Error on train set 0.0042789057 Error on test set 0.0043502459 Epoch 6500 Error on train set 0.0042508165 Error on test set 0.0043233084 Epoch 6600 Error on train set 0.0042235283 Error on test set 0.0042971647 Epoch 6700 Error on train set 0.0041969963 Error on test set 0.0042717727 Epoch 6800 Error on train set 0.0041711926 Error on test set 0.0042471024 Epoch 6900 Error on train set 0.0041460851 Error on test set 0.0042231218 Epoch 7000 Error on train set 0.0041216379 Error on test set 0.0041997982 Epoch 7100 Error on train set 0.0040978277 Error on test set 0.0041771047 Epoch 7200 Error on train set 0.0040746210 Error on test set 0.0041550109 Epoch 7300 Error on train set 0.0040519978 Error on test set 0.0041334946 Epoch 7400 Error on train set 0.0040299348 Error on test set 0.0041125333 Epoch 7500 Error on train set 0.0040084026 Error on test set 0.0040920973 Epoch 7600 Error on train set 0.0039873808 Error on test set 0.0040721665 Epoch 7700 Error on train set 0.0039668498 Error on test set 0.0040527219 Epoch 7800 Error on train set 0.0039467923 Error on test set 0.0040337448 Epoch 7900 Error on train set 0.0039271922 Error on test set 0.0040152199 Epoch 8000 Error on train set 0.0039080293 Error on test set 0.0039971280 Epoch 8100 Error on train set 0.0038892890 Error on test set 0.0039794510 Epoch 8200 Error on train set 0.0038709559 Error on test set 0.0039621773 Epoch 8300 Error on train set 0.0038530135 Error on test set 0.0039452892 Epoch 8400 Error on train set 0.0038354506 Error on test set 0.0039287745 Epoch 8500 Error on train set 0.0038182531 Error on test set 0.0039126198 Epoch 8600 Error on train set 0.0038014064 Error on test set 0.0038968106 Epoch 8700 Error on train set 0.0037848980 Error on test set 0.0038813341 Epoch 8800 Error on train set 0.0037687204 Error on test set 0.0038661817 Epoch 8900 Error on train set 0.0037528600 Error on test set 0.0038513422 Epoch 9000 Error on train set 0.0037373099 Error on test set 0.0038368078 Epoch 9100 Error on train set 0.0037220509 Error on test set 0.0038225586 Epoch 9200 Error on train set 0.0037070781 Error on test set 0.0038085904 Epoch 9300 Error on train set 0.0036923850 Error on test set 0.0037948966 Epoch 9400 Error on train set 0.0036779642 Error on test set 0.0037814688 Epoch 9500 Error on train set 0.0036638069 Error on test set 0.0037683002 Epoch 9600 Error on train set 0.0036498983 Error on test set 0.0037553732 Epoch 9700 Error on train set 0.0036362333 Error on test set 0.0037426862 Epoch 9800 Error on train set 0.0036228085 Error on test set 0.0037302342 Epoch 9900 Error on train set 0.0036096165 Error on test set 0.0037180081 Epoch 10000 Error on train set 0.0035966476 Error on test set 0.0037060005 Training complete.
