Search

Geek Girl Joy

Artificial Intelligence, Simulations & Software

Month

July 2016

Getting started with Neural Networks using the FANN library, PHP and C9.io – Part 2

Before I get started I want to thank everyone who commented or tweeted about my last post, it really means a lot to me and definitely motivates me to want to make my posts even better!

This is the second post in this set, if you have not already read the first post in this series you can find it here:
Getting started with Neural Networks using the FANN library, PHP and C9.io

If you intend to follow along I recommend that you read that post before this one as it will walk you through setting up your test environment on c9.io to use the free FANN library which is essential if you want the code on this page to work.

Lets continue where we last left off, we had run a test to ensure that FANN was working properly. If you encountered an error, try following the steps again and if that doesn’t work leave a comment on that post or a comment below and I will try to help.

A successful test of FANN using the provided example looks something like this:

xor test (-1.000000,1.000000) -> 0.964151

I say ‘like’ because the number after -> will be different for you than it is for me or anyone else. Additionally, if we recreate the ANN by running simple_train.php again it will give us a different number than we got the first time. This is because the ‘weights’ that the neural network settled on during training are slightly different each time.

During training, the ANN does it’s best to adjust itself to match our ‘ideal’ output for a given input, which we gave it during it’s training. Every time it runs through the training it makes small adjustments to the weights used to calculate the output and it will eventually find the right set of adjustments that allow it to get close to our expectations. The ANN will continue training as long as it has not gone over the specified maximum number of training loops or if it’s output is within the specified error margin, whichever comes first. If it reaches the maximum number of training loops, the number of hidden layers or the number of hidden neurons may be improperly configured or you are using too few training loops for the complexity of the data set.

When the ANN is adjusting itself during training, rather than use ‘Integers‘ it uses what is called ‘Floating Point‘ numbers. This allows the ‘mind’ to make really small changes each time it runs through a training loop. Because the ANN uses floating point numbers to make small adjustments the response we get from the ANN will be a floating point number as well, hence 0.964151 as a response from the XOR neural network.

While not entirely analogous to a “confidence” score, it might be simpler for you to think about the number that the ANN is giving us at this time as ‘how sure’ it is that the answer it gave us was correct. At least in this case XOR is using -1 to mean 100% no and 1 to mean 100% yes. I think that is very cool! 🙂

All that sounds great but is perhaps a little confusing if you have no experience with neural networks. I assure you it won’t matter right now if you don’t fully understand all of this, simply be aware of it, in time and with some effort it will all make sense.

Here is the basic structure of an artificial neural network:
Colored neural network

An artificial neural network is made up of ‘layers’ or ‘groups’ of neurons. Broadly speaking there are three layer groups in an ANN: ‘input’, ‘output’ and sandwiched in the middle is the ‘hidden’ group. It’s important to note that the hidden group can consist of more than one ‘layer’ of neurons. Layers are like rows or columns (depending on how you are visualizing the ANN) and are sets of neurons lined up which are connected with each other. In the image above the layers are represented by columns.

Below is an example of an ANN with the layers represented as rows. The network below has one input layer consisting of eight input neurons, two hidden layers also consisting of eight neurons each (sixteen total hidden neurons) and a single output layer consisting of two output neurons. All the green and orange lines between the neurons show how each neuron is connected to the other neurons.

Two-layer feedforward artificial neural network
A Neural Network designed to control thrusters
You may be interested to know that the ANN above was designed so that if you provide it with position state, direction and other environment values it will output control values for thrusters, kind of like auto-pilot. Looks a lot simpler than having to write that using traditional procedural or object oriented methods, doesn’t it? 🙂

Here is an image I created showing the structure of the example neural network ‘XOR’ that came with FANN. It highlights the layer groups without showing the connections between the neurons as the previous images illustrated:

The structure of a artificial neural network that performs the XOR operation.
The structure of an artificial neural network that performs the XOR operation.

This neural network has two ‘input’ neurons, one ‘output’ neuron and three hidden layers with three hidden neurons on each layer for a total of nine ‘hidden’ neurons.

Don’t let the term ‘hidden’ scare you, layers and neurons are referred to as ‘hidden’ only because as programmers, developers and engineers we don’t interact with them, only the ANN does. Fundamentally, there is really no difference between the input & output neurons and the hidden neurons, they are all just boxes that contain numbers. Some of the boxes we have direct control over or access to (inputs & outputs) and the others (the hidden layers) we don’t need to concern ourselves with as they are the purview of the neural network.

Because the ANN uses what is called the ‘Sigmoid function‘ the inputs and outputs will equal or be between -1.0 and 1.0 and is demonstrated by the output we got from the XOR ANN:

xor test (-1.000000,1.000000) -> 0.964151

So the output from the ANN said that it is highly confident that based on the input it was given the answer is 0.964151, which in this case we can interpret as a 1.0, meaning that the mind is saying based on the input values provided to it, it is selecting the value of the item (whatever the item may be) in the second input and rejecting the item in the first input. If the output were -0.964151 (or something like this) it could be selecting the item in the first neuron, or in the case of the XOR operation it would be saying false. The choice is made by the ANN based off how it was trained to respond to different inputs.

XOR is a very simple neural network to implement but I hope that you are beginning to understand that ANN’s can be extremely powerful tools for solving problems that can be difficult to hand code a solution for using traditional If/Else or Case statements. In fact a common complaint by some is that sometimes it feels ‘too easy’ to solve complex problems because all they had to do was design and build a training set rather then write thousands (or potentially hundreds of thousands) of lines of complex Boolean logic to solve a problem. I am not suggesting that neural networks are a panacea to all problems, however there are some particularly difficult problems that can be solved by someone with a little training and a properly implemented neural network. They are not magic, but watching the results of so called ‘deep‘ networks can feel magical. 🙂

Artificial Neural Networks are used to do things like recognize patterns hidden in data, for example ANN’s are used by some banks & hedge funds and day traders to create investment strategies or to do OCR (Optical Character Recognition) like converting a scanned image into editable text. Making choices like which thruster or jet engine to fire, when and for how long. Robots like Baxter use neural networks for visual learning where an employee can demonstrate a task to Baxter and Baxter can watch and learn how to do it. Robotic surgeons use ANN’s to eliminate micro tremors present in the hands of surgeons to minimize the the potential for unintended incisions. ANN’s are also used in natural language processing and advanced search engines. Among many other uses!

IBM Watson uses neural networks as well as Google’s Deep Mind.

So, now that we have looked at ANN’s in general and have a rudimentary understanding of what is going on under the hood, lets start to look at how it all actually works. Before proceeding you may wish to review the XOR Operation & the XOR Gate wiki’s so this will make more sense but it shouldn’t be absolutely required strictly speaking.

FANN starts us out with the following files:

xor.data, simple_train.php & simple_test.php.

xor.data is the ‘training data’ and looks like this:

4 2 1
-1 -1
-1
-1 1
1
1 -1
1
1 1
-1

This is the data used to ‘teach’ the neural network by showing it an example problem and then following up by showing it the desired response or answer to the problem. The term ‘training’ is quite accurate because the ANN learns overtime how to answer your questions by looking at the example questions you give it, trying to answer the question, then comparing it’s output with the desired output, then adjusting itself a little to try to improve it’s answer to be closer to what you want. Essentially this means that the neural network can figure out really complex rules, just by being shown examples of the types of problems you want solved along with the answer. After sufficient training, you can ask questions that are similar to the problems the ANN was trained to answer but that it has never seen before and expect to get correct responses.

There are such things as ‘false positives’ where the neural network gets confused (poor thing) and thinks it has the right answer but actually gives the wrong answer. Proper training can reduce false positives significantly but it would be impossible to eliminate them entirely, but again they can be rendered more or less irrelevant with proper design and the understanding of why you might get false positives and developing solutions to deal with those contingencies.

For example, in OCR (Optical Character Recognition) neural networks, you won’t have to train an ANN on every single font, but the more serif and sans-serif fonts it trains with the better it will correctly identify letters in a picture. Sometimes despite your best efforts, OCR occasionally misidentifies a character, it’s going to happen. Additional hidden layers or hidden neurons may be required to combat false positives. Alternatively, you may have too many of either or both and not be able to get the ‘Sigmoid function‘ to ‘fire’ or ‘activate’ so you will get very low response (if not simply zero) from the ANN as a response regardless if you vary the input. It’s also worth noting that you can also ‘over train’ a neural network which would make it accurate to answer the questions it’s trained on but not accurate with new questions.

Having these considerations in mind there is a certain level of trial and error that goes into working out the optimal configuration for a neural network but it’s fairly forgiving and with a little practice you won’t have any trouble making your own neural networks.

So, now that you understand that basically a neural network learns by us giving it a problem and then giving it the answer and it learns to give the right answer and over time it ‘figures out’ the rules to the questions without complex programming, we can head back to the training data and break it down line by line.

The first line ( 4 2 1 ) is absolutely critical to the training file and your code will ‘crash’ if you don’t include it. The mind uses this information to understand all the data that comes after the first line.

The 4 means that there are four training sets in this file.

The 2 means that each training set consists of two inputs.

The 1 means that each training set produces 1 outputs.

I have color coded each training set so you can see what is going on a little better:

4 2 1
-1 -1
-1
-1 1
1
1 -1
1
1 1
-1

The first training set is blue. It shows the ANN that two inputs of -1 should return a -1.

The second training set is green. It shows the ANN that inputs of -1 & 1 should return 1.

The third training set is red. It shows the ANN that inputs of 1 & -1 should return 1.

The fourth (and last) training set is yellow. It shows the ANN that inputs of 1 & 1 should return -1.

In terms of Boolean logic this is a false, true, true, false proof table. In terms of electronics and hardware this set switches a circuit to low, high, high, low. If it were a thruster on a rocket headed to Mars it would be Off, On, On, Off.

These are examples of how XOR works and these same examples (without the additional information on the first line of the training file) can be seen on the XOR Gate wiki, although in that example they use zero instead of negative one.

So with these four examples we can train an artificial neural network how to do XOR and this operation (among others) can be applied to many different scenarios.

You can close the xor.data file and open simple_train.php which looks like this:

$num_input = 2;
$num_output = 1;
$num_layers = 3;
$num_neurons_hidden = 3;
$desired_error = 0.001;
$max_epochs = 500000;
$epochs_between_reports = 1000;
$ann = fann_create_standard($num_layers, $num_input, $num_neurons_hidden, $num_output);
if ($ann) {
    fann_set_activation_function_hidden($ann, FANN_SIGMOID_SYMMETRIC);
    fann_set_activation_function_output($ann, FANN_SIGMOID_SYMMETRIC);
    $filename = dirname(__FILE__) . "/xor.data";
    if (fann_train_on_file($ann, $filename, $max_epochs, $epochs_between_reports, $desired_error))
        fann_save($ann, dirname(__FILE__) . "/xor_float.net");
    fann_destroy($ann);
}

The first 7 lines declare variables that we use to configure the ANN.

On line 8 we create the Neural Network as the variable $ann:

$ann = fann_create_standard($num_layers, $num_input, $num_neurons_hidden, $num_output);

What follows is a confirmation that we were able to create the ANN object using an ‘if’ Conditional statement:

if ($ann) {

Assuming that all goes well the program sets the ‘activation’ functions for the hidden and output neurons:

fann_set_activation_function_hidden($ann, FANN_SIGMOID_SYMMETRIC);
fann_set_activation_function_output($ann, FANN_SIGMOID_SYMMETRIC);

The fann_set_activation_function_output() function is taking two variables, the first being $ann which specifies the network you are using. The second variable being the constant FANN_SIGMOID_SYMMETRIC and specifies the type of activation function to use.

Although its not important to know at this point, you may be wondering what ‘FANN_SIGMOID_SYMMETRIC’ really means. On the Datatypes Reference page for FANN or on PHP.net Predefined Constants reference, it is defined as a
‘Symmetric sigmoid activation function, aka. tanh‘ . It goes on to describe it as one of the most used activation functions and a stepwise linear approximation to symmetric sigmoid.

Put simply, this is a fast approximation of the sigmoid function that exchanges a little accuracy for speed and it is “stepwise” which simply means it is not continuously processing but rather operates in stages to to generate the sigmoid curve.

If you are really really interested in knowing what that means, here is the algorithm used when FANN_SIGMOID_SYMMETRIC is selected as your activation function:

span: -1 < y < 1
y = tanh(s*x) = 2/(1 + exp(-2*s*x)) - 1
d = s*(1-(y*y))

Don’t fret if you don’t understand it you don’t really have to in order to get results.

The next line:

$filename = dirname(__FILE__) . "/xor.data";

Sets the variable $filename to the the string value for the path pointing to the training data using the “Magic Constant” __FILE__ to the select the parent directory that this file is running from and then appends the name of the training file ‘xor.data’ using concatenation.

After that we attempt to train the ANN using the following and then check if it was successful:

if (fann_train_on_file($ann, $filename, $max_epochs, $epochs_between_reports, $desired_error))

Remember when I said the first line in the training file needs to list the number of training sets, the number of inputs and the number of outputs? Right when fann_train_on_file() is called is when the program will crash if you don’t include that.

If fann_train_on_file() returns true then the ANN was trained and it is then saved using:

fann_save($ann, dirname(__FILE__) . "/xor_float.net");

Again using the Magic Constant __FILE__ to the select the parent directory that this file is running from and then appends the name of the network file ‘xor_float.net’.

Now, all that is left to do is to destroy the ANN in memory because we don’t need it taking up space any longer:

fann_destroy($ann);

Alright, now we can close simple_train.php and open simple_test.php:

$train_file = (dirname(__FILE__) . "/xor_float.net");
if (!is_file($train_file))
	die("The file xor_float.net has not been created! Please run simple_train.php to generate it" . PHP_EOL);
$ann = fann_create_from_file($train_file);
if ($ann) {
	$input = array(-1, 1);
	$calc_out = fann_run($ann, $input);
	printf("xor test (%f,%f) -> %f\n", $input[0], $input[1], $calc_out[0]);
	fann_destroy($ann);
} else {
	die("Invalid file format" . PHP_EOL);
}

We start by selecting the neural network we want to test:

$train_file = (dirname(__FILE__) . "/xor_float.net");

Then we check the path to make sure there is a file, if not we use the PHP die() language construct to output a message advising you to run simple_train.php, in addition to the OS specific newline character with the PHP_EOL predefined constant, and then terminate the current script.

if (!is_file($train_file))
	die("The file xor_float.net has not been created! Please run simple_train.php to generate it" . PHP_EOL);

If the path is correct (which it will be if you ran simple_train.php) we create the ANN from the .net file:

$ann = fann_create_from_file($train_file);

If successful:

if ($ann) {

Create an array called $input which will store the values that we will pass into the input neurons. Since XOR takes two inputs our array will contain two values:

$input = array(-1, 1);

Next we run the inputs through the ANN and store the output as $calc_out which is an array.

$calc_out = fann_run($ann, $input);

Now all the ‘heavy lifting’ is done, we can output the results as a formatted string using printf:

printf("xor test (%f,%f) -> %f\n", $input[0], $input[1], $calc_out[0]);

For reference the %f’s are the variables (arguments) in order beginning in the second argument of printf(): $input[0] and ending with $calc_out[0]. There are 3 %f’s and 3 variables/arguments.

Note: the %f explicitly tells the computer that our variables are to be floating point numbers, for other format parameters you can review them on PHP.net.

Again, all that is left to do is to destroy the ANN in memory again because we don’t need it taking up space any longer:

fann_destroy($ann);

At the very bottom of the script we have an else conditional statement that is executed if we are unable to create the ANN using the .net file, it also uses the die() language construct to output a message and terminate the script :

else { die("Invalid file format" . PHP_EOL); }

Congratulations, you made it! 😛

My next post will go into more depth and cover how to create your own neural networks just like the XOR ANN that FANN came with.

All my posts are made possible by the generous contributions and hard work of my sponsors!

Continue to Part 3

~Joy

  
  

 

Advertisements

Getting started with Neural Networks using the FANN library, PHP and C9.io

According to Wikipedia an Artificial Neural Network (ANN) is defined as follows:

In machine learning and cognitive science, artificial neural networks (ANNs) are a family of models inspired by biological neural networks (the central nervous systems of animals, in particular the brain) which are used to estimate or approximate functions that can depend on a large number of inputs…” – Wikipedia (Artificial Neural Networks)

Sounds cool right? Who wouldn’t want their own AI, even if it’s still not exactly the simplest thing to understand? I highly recommend at least reading through the ANN Wiki so that this article makes more sense but it’s not required to be able to follow along.

There are tons of free online classes and tutorials surrounding this topic as well as paid courses online in addition to colleges and universities. Books abound on this subject as well, just checkout Amazon!

But, where to start?

Well today I am going to pull back the curtain to show you how to actually work with ANN’s using the FANN library. FANN (Fast Artificial Neural Network) Library has been around for about 13 years and is still in development by it’s creator Steffen Nissen who works in collaboration with other programmers to maintain the the library bindings for the following languages:

C++, C#, D, Java, Javascript & Node.js, PHP, Rust, Erlang, Python, MetaTrader4 (MQL4), Perl, Ruby, Harbour, Delphi, Tcl, Lua, Visual Prolog 7, SWI Prolog, Go, Soap / Web service, Matlab, R, Ada, Haskel, Grass, Octave, Squeak Smalltalk & Pure Data.

Very impressive!!

The PHP FANN bindings are currently maintained by Jakub Zelenka.

You can use FANN with your language of choice but today I am going to show you how to use the library with PHP. Why PHP? Because I like PHP! 😉

I may demonstrate it’s use with Node.js, Python & C++ or C# at some point as well but today we’ll be using PHP.

Don’t worry about trying to picking a fast compiled language, just work with one you know well. FANN is written and compiled using C++ and so will operate at the machine code level, you just issue calls to it.

Additionally, FANN is released under the LGPL license so if you follow its terms you can use it commercially as well!

You can review the FANN Library on its official site: http://leenissen.dk/fann/wp/

Or the PHP version on PHP.net: http://php.net/manual/en/book.fann.php

To get started you will need a web host to host your code. The host must offer SSH terminal access and preferably the OS should be in the Debian family of Linux (Debian, Ubuntu, Mint). You can Use other Linuxes, or even Windows or Mac OS, but the instructions are different than I outline here.

My recommendation is that you create a free account with C9.io and these instructions assume that you go that route. It’s online and you can work with some of the most popular languages used today in a very functional web based IDE.

So, without further ado…

Step 1: Create an account on C9.io

1

Step 2: Check the email account you signed up with for an email with a link that will allow you to set your account password.

2

Step 3: Once your account has been created you will land on the ‘Workspaces’ view:

3

Step 4: Now we can create a new workspace for our Neural Network project. Click the Plus (Create new workspace).

4

Enter a Workspace Name and a Description.

Note: the Workspace Name needs to be lowercase and include no special chars other than an underscore.

Next select PHP as the project “template” which basically lets C9.io know what language or platform to setup for you. Then just click the Create Workspace button at the bottom of the form.

Step 5: When everything is setup, you will see this:

5

The bash tab at the bottom lets you issue commands to the server’s terminal. By default you will have access to Git as well as compilation tools, etc. which we will be using but I will not go into details about them in this post as I will do so in later posts as well as I am providing exact code to copy and paste if you need it so don’t worry too much about what the following does in detail if you don’t understand it. I will explain generally what each of the following commands do as we go along.

First click in the bash prompt and type the following to make a copy of the latest version of FANN so you can work with it.

git clone https://github.com/libfann/fann.git

Once that finishes you will see a new folder named fann show up in the file view on the left.

7

Step 6: Go back to the bash command line and change the working folder to fann:

cd ./fann

Then you need to build the make files so you can compile the library, type the following into the bash prompt:

cmake .

This shouldn’t take long.

Step 7: Once the make files are all… made 😛 you can compile everything, type the following into the bash prompt:

sudo make install

Now sit back and wait… might take a minute.

You’re doing so well!

Step 8: Okay, so now that we have the FANN Library compiled we need to leave the fann folder and go back up to the root directory of our workspace, enter the following into the bash prompt:

cd\

Now that we are at the top level of our workspace we need to download a copy of the FANN bindings for PHP, type the following into the bash prompt:

git clone --recursive https://github.com/bukka/php-fann.git

Which won’t take long. Now we want to move into that folder, type the following into the bash prompt:

cd php-fann

Step 9: Great, now we need to to compile the extension, enter the following one at a time into the bash prompt:

phpize
./configure --with-fann
make
sudo make install

Step 10: Only one final thing to do before we are done with the configuration.

Locate the php.ini file in the root of the folder structure on the left and open it.

8

Scroll down to about line 937 under the section Dynamic Extensions. Add the following:

extension=fann.so

9

Now save and close php.ini.

Test FANN You have completed cloning the FANN library into your workspace, making and compiling it, cloning the PHP bindings into your workspace, making and compiling it and adding it to the local php.ini file.

You can now use the FANN library to make your own neural networks!

So lets test it and make sure it’s working! Click the Green Run Project button on menu on the top of the page.

10

You will get a windowed browser in the IDE, click the “Popout” button so you get a nice full-size tab to preview everything in:
11

Now just navigate to: /php-fann/examples/logic_gates by clicking on the folders.

Once there, first run the simple_train.php to train the test ANN, the page will load and say ‘xor_float.net saved’ followed by a link to ‘Test’ the Neural Network. This step uses the .data file (which is the training file for the ANN) and trains a new ANN and saves the neural network as a .net file.

click the ‘Test’ link to run simple_test.php, if you get output like this then it’s working:

xor test (-1.000000,1.000000) -> 0.964151

In my next post I will break down what each file is actually doing and then I will show you how to create your own Artificial Neural Networks using FANN.

Until then, please leave a comment below and let me know what you like or don’t like about FANN and if you had any trouble with this tutorial.

All my posts are made possible by the generous contributions and hard work of my sponsors!

Continue to Part 2

~Joy

 
 
 

 

Create a free website or blog at WordPress.com.

Up ↑