Before I get started I want to thank everyone who commented or tweeted about my last post, it really means a lot to me and definitely motivates me to want to make my posts even better!
This is the second post in this set, if you have not already read the first post in this series you can find it here:
Getting started with Neural Networks using the FANN library, PHP and C9.io
If you intend to follow along I recommend that you read that post before this one as it will walk you through setting up your test environment on c9.io to use the free FANN library which is essential if you want the code on this page to work.
Lets continue where we last left off, we had run a test to ensure that FANN was working properly. If you encountered an error, try following the steps again and if that doesn’t work leave a comment on that post or a comment below and I will try to help.
A successful test of FANN using the provided example looks something like this:
xor test (-1.000000,1.000000) -> 0.964151
I say ‘like’ because the number after -> will be different for you than it is for me or anyone else. Additionally, if we recreate the ANN by running simple_train.php again it will give us a different number than we got the first time. This is because the ‘weights’ that the neural network settled on during training are slightly different each time.
During training, the ANN does it’s best to adjust itself to match our ‘ideal’ output for a given input, which we gave it during it’s training. Every time it runs through the training it makes small adjustments to the weights used to calculate the output and it will eventually find the right set of adjustments that allow it to get close to our expectations. The ANN will continue training as long as it has not gone over the specified maximum number of training loops or if it’s output is within the specified error margin, whichever comes first. If it reaches the maximum number of training loops, the number of hidden layers or the number of hidden neurons may be improperly configured or you are using too few training loops for the complexity of the data set.
When the ANN is adjusting itself during training, rather than use ‘Integers‘ it uses what is called ‘Floating Point‘ numbers. This allows the ‘mind’ to make really small changes each time it runs through a training loop. Because the ANN uses floating point numbers to make small adjustments the response we get from the ANN will be a floating point number as well, hence 0.964151 as a response from the XOR neural network.
While not entirely analogous to a “confidence” score, it might be simpler for you to think about the number that the ANN is giving us at this time as ‘how sure’ it is that the answer it gave us was correct. At least in this case XOR is using -1 to mean 100% no and 1 to mean 100% yes. I think that is very cool! 🙂
All that sounds great but is perhaps a little confusing if you have no experience with neural networks. I assure you it won’t matter right now if you don’t fully understand all of this, simply be aware of it, in time and with some effort it will all make sense.
Here is the basic structure of an artificial neural network:
An artificial neural network is made up of ‘layers’ or ‘groups’ of neurons. Broadly speaking there are three layer groups in an ANN: ‘input’, ‘output’ and sandwiched in the middle is the ‘hidden’ group. It’s important to note that the hidden group can consist of more than one ‘layer’ of neurons. Layers are like rows or columns (depending on how you are visualizing the ANN) and are sets of neurons lined up which are connected with each other. In the image above the layers are represented by columns.
Below is an example of an ANN with the layers represented as rows. The network below has one input layer consisting of eight input neurons, two hidden layers also consisting of eight neurons each (sixteen total hidden neurons) and a single output layer consisting of two output neurons. All the green and orange lines between the neurons show how each neuron is connected to the other neurons.
Here is an image I created showing the structure of the example neural network ‘XOR’ that came with FANN. It highlights the layer groups without showing the connections between the neurons as the previous images illustrated:
This neural network has two ‘input’ neurons, one ‘output’ neuron and three hidden layers with three hidden neurons on each layer for a total of nine ‘hidden’ neurons.
Don’t let the term ‘hidden’ scare you, layers and neurons are referred to as ‘hidden’ only because as programmers, developers and engineers we don’t interact with them, only the ANN does. Fundamentally, there is really no difference between the input & output neurons and the hidden neurons, they are all just boxes that contain numbers. Some of the boxes we have direct control over or access to (inputs & outputs) and the others (the hidden layers) we don’t need to concern ourselves with as they are the purview of the neural network.
Because the ANN uses what is called the ‘Sigmoid function‘ the inputs and outputs will equal or be between -1.0 and 1.0 and is demonstrated by the output we got from the XOR ANN:
xor test (-1.000000,1.000000) -> 0.964151
So the output from the ANN said that it is highly confident that based on the input it was given the answer is 0.964151, which in this case we can interpret as a 1.0, meaning that the mind is saying based on the input values provided to it, it is selecting the value of the item (whatever the item may be) in the second input and rejecting the item in the first input. If the output were -0.964151 (or something like this) it could be selecting the item in the first neuron, or in the case of the XOR operation it would be saying false. The choice is made by the ANN based off how it was trained to respond to different inputs.
XOR is a very simple neural network to implement but I hope that you are beginning to understand that ANN’s can be extremely powerful tools for solving problems that can be difficult to hand code a solution for using traditional If/Else or Case statements. In fact a common complaint by some is that sometimes it feels ‘too easy’ to solve complex problems because all they had to do was design and build a training set rather then write thousands (or potentially hundreds of thousands) of lines of complex Boolean logic to solve a problem. I am not suggesting that neural networks are a panacea to all problems, however there are some particularly difficult problems that can be solved by someone with a little training and a properly implemented neural network. They are not magic, but watching the results of so called ‘deep‘ networks can feel magical. 🙂
Artificial Neural Networks are used to do things like recognize patterns hidden in data, for example ANN’s are used by some banks & hedge funds and day traders to create investment strategies or to do OCR (Optical Character Recognition) like converting a scanned image into editable text. Making choices like which thruster or jet engine to fire, when and for how long. Robots like Baxter use neural networks for visual learning where an employee can demonstrate a task to Baxter and Baxter can watch and learn how to do it. Robotic surgeons use ANN’s to eliminate micro tremors present in the hands of surgeons to minimize the the potential for unintended incisions. ANN’s are also used in natural language processing and advanced search engines. Among many other uses!
IBM Watson uses neural networks as well as Google’s Deep Mind.
So, now that we have looked at ANN’s in general and have a rudimentary understanding of what is going on under the hood, lets start to look at how it all actually works. Before proceeding you may wish to review the XOR Operation & the XOR Gate wiki’s so this will make more sense but it shouldn’t be absolutely required strictly speaking.
FANN starts us out with the following files:
xor.data, simple_train.php & simple_test.php.
xor.data is the ‘training data’ and looks like this:
4 2 1
-1 -1
-1
-1 1
1
1 -1
1
1 1
-1
This is the data used to ‘teach’ the neural network by showing it an example problem and then following up by showing it the desired response or answer to the problem. The term ‘training’ is quite accurate because the ANN learns overtime how to answer your questions by looking at the example questions you give it, trying to answer the question, then comparing it’s output with the desired output, then adjusting itself a little to try to improve it’s answer to be closer to what you want. Essentially this means that the neural network can figure out really complex rules, just by being shown examples of the types of problems you want solved along with the answer. After sufficient training, you can ask questions that are similar to the problems the ANN was trained to answer but that it has never seen before and expect to get correct responses.
There are such things as ‘false positives’ where the neural network gets confused (poor thing) and thinks it has the right answer but actually gives the wrong answer. Proper training can reduce false positives significantly but it would be impossible to eliminate them entirely, but again they can be rendered more or less irrelevant with proper design and the understanding of why you might get false positives and developing solutions to deal with those contingencies.
For example, in OCR (Optical Character Recognition) neural networks, you won’t have to train an ANN on every single font, but the more serif and sans-serif fonts it trains with the better it will correctly identify letters in a picture. Sometimes despite your best efforts, OCR occasionally misidentifies a character, it’s going to happen. Additional hidden layers or hidden neurons may be required to combat false positives. Alternatively, you may have too many of either or both and not be able to get the ‘Sigmoid function‘ to ‘fire’ or ‘activate’ so you will get very low response (if not simply zero) from the ANN as a response regardless if you vary the input. It’s also worth noting that you can also ‘over train’ a neural network which would make it accurate to answer the questions it’s trained on but not accurate with new questions.
Having these considerations in mind there is a certain level of trial and error that goes into working out the optimal configuration for a neural network but it’s fairly forgiving and with a little practice you won’t have any trouble making your own neural networks.
So, now that you understand that basically a neural network learns by us giving it a problem and then giving it the answer and it learns to give the right answer and over time it ‘figures out’ the rules to the questions without complex programming, we can head back to the training data and break it down line by line.
The first line ( 4 2 1 ) is absolutely critical to the training file and your code will ‘crash’ if you don’t include it. The mind uses this information to understand all the data that comes after the first line.
The 4 means that there are four training sets in this file.
The 2 means that each training set consists of two inputs.
The 1 means that each training set produces 1 outputs.
I have color coded each training set so you can see what is going on a little better:
4 2 1
-1 -1
-1
-1 1
1
1 -1
1
1 1
-1
The first training set is blue. It shows the ANN that two inputs of -1 should return a -1.
The second training set is green. It shows the ANN that inputs of -1 & 1 should return 1.
The third training set is red. It shows the ANN that inputs of 1 & -1 should return 1.
The fourth (and last) training set is yellow. It shows the ANN that inputs of 1 & 1 should return -1.
In terms of Boolean logic this is a false, true, true, false proof table. In terms of electronics and hardware this set switches a circuit to low, high, high, low. If it were a thruster on a rocket headed to Mars it would be Off, On, On, Off.
These are examples of how XOR works and these same examples (without the additional information on the first line of the training file) can be seen on the XOR Gate wiki, although in that example they use zero instead of negative one.
So with these four examples we can train an artificial neural network how to do XOR and this operation (among others) can be applied to many different scenarios.
You can close the xor.data file and open simple_train.php which looks like this:
$num_input = 2;
$num_output = 1;
$num_layers = 3;
$num_neurons_hidden = 3;
$desired_error = 0.001;
$max_epochs = 500000;
$epochs_between_reports = 1000;
$ann = fann_create_standard($num_layers, $num_input, $num_neurons_hidden, $num_output);
if ($ann) {
fann_set_activation_function_hidden($ann, FANN_SIGMOID_SYMMETRIC);
fann_set_activation_function_output($ann, FANN_SIGMOID_SYMMETRIC);
$filename = dirname(__FILE__) . "/xor.data";
if (fann_train_on_file($ann, $filename, $max_epochs, $epochs_between_reports, $desired_error))
fann_save($ann, dirname(__FILE__) . "/xor_float.net");
fann_destroy($ann);
}
The first 7 lines declare variables that we use to configure the ANN.
On line 8 we create the Neural Network as the variable $ann:
$ann = fann_create_standard($num_layers, $num_input, $num_neurons_hidden, $num_output);
What follows is a confirmation that we were able to create the ANN object using an ‘if’ Conditional statement:
if ($ann) {
Assuming that all goes well the program sets the ‘activation’ functions for the hidden and output neurons:
fann_set_activation_function_hidden($ann, FANN_SIGMOID_SYMMETRIC); fann_set_activation_function_output($ann, FANN_SIGMOID_SYMMETRIC);
The fann_set_activation_function_output() function is taking two variables, the first being $ann which specifies the network you are using. The second variable being the constant FANN_SIGMOID_SYMMETRIC and specifies the type of activation function to use.
Although its not important to know at this point, you may be wondering what ‘FANN_SIGMOID_SYMMETRIC’ really means. On the Datatypes Reference page for FANN or on PHP.net Predefined Constants reference, it is defined as a
‘Symmetric sigmoid activation function, aka. tanh‘ . It goes on to describe it as one of the most used activation functions and a stepwise linear approximation to symmetric sigmoid.
Put simply, this is a fast approximation of the sigmoid function that exchanges a little accuracy for speed and it is “stepwise” which simply means it is not continuously processing but rather operates in stages to to generate the sigmoid curve.
If you are really really interested in knowing what that means, here is the algorithm used when FANN_SIGMOID_SYMMETRIC is selected as your activation function:
span: -1 < y < 1 y = tanh(s*x) = 2/(1 + exp(-2*s*x)) - 1 d = s*(1-(y*y))
Don’t fret if you don’t understand it you don’t really have to in order to get results.
The next line:
$filename = dirname(__FILE__) . "/xor.data";
Sets the variable $filename to the the string value for the path pointing to the training data using the “Magic Constant” __FILE__ to the select the parent directory that this file is running from and then appends the name of the training file ‘xor.data’ using concatenation.
After that we attempt to train the ANN using the following and then check if it was successful:
if (fann_train_on_file($ann, $filename, $max_epochs, $epochs_between_reports, $desired_error))
Remember when I said the first line in the training file needs to list the number of training sets, the number of inputs and the number of outputs? Right when fann_train_on_file() is called is when the program will crash if you don’t include that.
If fann_train_on_file() returns true then the ANN was trained and it is then saved using:
fann_save($ann, dirname(__FILE__) . "/xor_float.net");
Again using the Magic Constant __FILE__ to the select the parent directory that this file is running from and then appends the name of the network file ‘xor_float.net’.
Now, all that is left to do is to destroy the ANN in memory because we don’t need it taking up space any longer:
fann_destroy($ann);
Alright, now we can close simple_train.php and open simple_test.php:
$train_file = (dirname(__FILE__) . "/xor_float.net");
if (!is_file($train_file))
die("The file xor_float.net has not been created! Please run simple_train.php to generate it" . PHP_EOL);
$ann = fann_create_from_file($train_file);
if ($ann) {
$input = array(-1, 1);
$calc_out = fann_run($ann, $input);
printf("xor test (%f,%f) -> %f\n", $input[0], $input[1], $calc_out[0]);
fann_destroy($ann);
} else {
die("Invalid file format" . PHP_EOL);
}
We start by selecting the neural network we want to test:
$train_file = (dirname(__FILE__) . "/xor_float.net");
Then we check the path to make sure there is a file, if not we use the PHP die() language construct to output a message advising you to run simple_train.php, in addition to the OS specific newline character with the PHP_EOL predefined constant, and then terminate the current script.
if (!is_file($train_file))
die("The file xor_float.net has not been created! Please run simple_train.php to generate it" . PHP_EOL);
If the path is correct (which it will be if you ran simple_train.php) we create the ANN from the .net file:
$ann = fann_create_from_file($train_file);
If successful:
if ($ann) {
Create an array called $input which will store the values that we will pass into the input neurons. Since XOR takes two inputs our array will contain two values:
$input = array(-1, 1);
Next we run the inputs through the ANN and store the output as $calc_out which is an array.
$calc_out = fann_run($ann, $input);
Now all the ‘heavy lifting’ is done, we can output the results as a formatted string using printf:
printf("xor test (%f,%f) -> %f\n", $input[0], $input[1], $calc_out[0]);
For reference the %f’s are the variables (arguments) in order beginning in the second argument of printf(): $input[0] and ending with $calc_out[0]. There are 3 %f’s and 3 variables/arguments.
Note: the %f explicitly tells the computer that our variables are to be floating point numbers, for other format parameters you can review them on PHP.net.
Again, all that is left to do is to destroy the ANN in memory again because we don’t need it taking up space any longer:
fann_destroy($ann);
At the very bottom of the script we have an else conditional statement that is executed if we are unable to create the ANN using the .net file, it also uses the die() language construct to output a message and terminate the script :
else { die("Invalid file format" . PHP_EOL); }
Congratulations, you made it! 😛
My next post will go into more depth and cover how to create your own neural networks just like the XOR ANN that FANN came with.
All my posts are made possible by the generous contributions and hard work of my sponsors!
~Joy
Geek Girl Joy 
8 Pingback