 |
|
|
||||||
 |
||||||||
|
||||||||
|
 |
|
|
|
Building an Artificial Neural Network There are several recognised forms or topologies for Artificial Neural Networks. Some demand greater complexity in the construction of the neurons but that only really means more data. Frank Rosenblatt of Cornell University must be credited with coming up with the notion of neural networks as a flexible computational device. He constructed his neural networks as physical components connecting them together in a deliberate attempt to emulate the internal structure and connections found in our brains. You will find that it is simpler and cheaper to follow his groundbreaking work by constructing your neural networks in software. An Artificial Neuron An artificial neuron is, at it’s simplest, a set of numerical weights to be applied to the related incoming connections and a sigmoid function to adjust the output of the neuron. All a neuron has to do is to collect the outputs of the neurons it is connected to, apply the relevant weighting to each connection and then sum the total of the weighted connections. A neuron may also add in a “bias” value and then the total of the weighted inputs and any bias is passed through a sigmoid function and the resulting value made available as the output of the neuron. The Sigmoid Function A sigmoid function is used to set the output value of any given neuron by processing the sum of the weighted input values and any bias applied. The sigmoid function outputs a value that is close to zero for a low total input value and close to one for a high input value. The slope of the function curve can be adjusted by including a Threshold value that can make the “step” between zero and one steeper or more shallow. The formula for a sigmoid function is 1/(1+e-sx) where x is the sum of the weighted connection values and any bias and s is any threshold (scale) value being applied by the neuron. In Visual Basic the function might look like this: Private Function StepOutput(InputVal As Single, ScaleFactor As Single) As Single End Function Note: Cut the comments and the declared constant and it is all just one line – easy! An Artificial Neural Network An artificial neural network is made up of a number of input neurons and a number of output neurons. There will almost certainly be one or more (so called) Hidden Layer of neurons “between” the input and output layers. The connections between the neurons are up to you although a number of well tried basic designs exist. You might elect to start with what is known as a forward only pattern with each of the input neurons connected to all of the hidden neurons and then all of the hidden neurons connected to each of your output neurons, Evolving an Artificial Neural Network You will have to decide upon the criteria you will be using to determine which neural networks you want to select and evolve to create a new generation of networks to compete against their parents. Once you have selected the neural networks that you want to evolve then the process is quite straightforward. The best strategy is to apply small random changes in most instances (just as we found when evolving potential solutions to the Traveling Salesman problem) but on occasion a larger change might be needed to take a large evolutionary step forward. What we are evolving are the weights applied to input connections, any bias values and any threshold values used in the “sigmoid” function. You can also “evolve” the number of neurons in the hidden layer (adding to or reducing their number at random) and the number and nature of the connections between the neurons. You do not normally change the number of input or output neurons as these are usually set to suite the task in hand. To apply random changes that are in the most part small but which are occasionally larger we need to be able to obtain some random values that tend towards zero – and the best distribution to use is a Gaussian distribution of random numbers. Take a look at our page on random numbers to see the Visual Basic code for the generation of Gaussian random values. It is quite easy to adapt the code to other programming languages that provide a suitable pseudo random number function such as the VB Rnd function. Building Visual Basic Artificial Neural Networks This section (and it’s linked pages) is intended to provide a recipe to help you construct your own artificial neural networks in your programming language of choice. Experiment is important and you may need to bias your designs towards the tasks you want to tackle. Thus we have not provided a pre-coded download of some generic software but if you are interested in the topic you should find enough ideas and code snippets to help you complete your task in very short order. Building artificial neural networks is easy – connecting them to the real world tasks you want to set them will probably take up most of your programming effort. You can use quite simple programming methods to create a neural network in Visual Basic. Our experimental networks were constructed using two class structures but an implementation using arrays of data would be just as effective in many instances and would probably run faster. The only slight complexity in our design came from applying “persistence” – we wanted to be able to store our neural networks (or rather their structure and data values) in a binary file format between runs within our evolutionary software environments. The Neurons Our Neuron class had the following Public properties (values): An Input value (for Input Neurons) Plus a State value (representing the output of the neuron) Some neural network structures might require some additional data elements belonging to an individual neuron but these were what was used for the noughts and crosses playing networks. The neuron class holds a private copy of each Public property – the world interacts with the public properties and public methods only. This is one of the benefits of implementing classes – you can adapt and change the internal structure and processing of the classes with no impact upon the programs interacting with those classes provided the “interface” (those public properties and the parameters passed to public methods) do not change. The neuron class also had an internal data structure to record the neuron identity and weight to be applied to each incoming connection. The Neuron class also had the following Public methods: AddConnection() to add a connection between this neuron and one in a “higher” layer as controlled by the Neural Network class as it constructs the network. NeuronEvent() – the method that tells the neuron to assess the input connection values and output a revised “state” value. Mutate() – a method that mutates the input connection weights, bias and threshold values of the neuron by adding Gaussian random values to the existing values. Note: Click on the links to see the Visual Basic code for these and other methods described on this page. There were also some Public housekeeping methods to do with saving data to a file or in retrieving data from a file. This could in fact all be managed from the neural network class if you decide on a similar structure. The Private functions of the class were the Sigmoid function described earlier and a function to supply a Gaussian distribution of random numbers to the mutate method – please see our page on random number functions. The Network itself We elected to “contain” the associated neuron classes in a set of collection classes (collection classes are a fundamental class type in Visual Basic). We could have used a single collection class or a simple object array. Of course, if your neurons aren’t classes then you might just as well implement them as arrays forming part of your network code module. Our network class had only one public property and that was an identity number (a long integer) that we used to keep track of each neural network and it’s scores in a database. The Public methods are fairly obvious: CreateNetwork() to create the network – adding the required number of each type of neuron and initialising the connections between them. MutateNeurons() used to call the Mutate() method in the neuron classes – thus mutating the whole neural network. InputValue() to accept input values to the network and assign them to the relevant input neurons. TriggerEvent() to call the NeuronEvent() method in each neuron sequenced by layer. Thus the input neurons were “triggered” first followed by any hidden layers and finally the output neurons. OutputValue() to return the “states” of the individual output neurons. Again there was also some code to do with saving and loading the network to and from a binary file which would be very dependent upon your design choices. The only significant private method was one called AddNeuron() which added a new neuron class to the relevant collection class. It also sets the neuron ID and initialised the Bias and Sigmoid function Threshold values using a random number (see below). Initialising the neuron values of a new neural network A Neural network is constructed from a number of neurons. If you are going to use evolutionary techniques to search for the best settings for the input weights, biases etc. then for each neuron you will probably start with random values. However it makes sense to limit the range of the initial values (see our section on generating random values within a specified range) as this may make it easier to obtain some reasonable output from your network from the start. For the noughts and crosses playing neural networks the initial values for connection weights and biases were in the range –0.5 to +0.5. The initial threshold values were in the range 0.5 to 2. How did we get the artificial neural networks to play noughts and crosses (tic tac toe)? Just like the evolving Travelling Salesman solutions we experimented with the noughts and crosses neural networks had no “understanding” of the game they were playing. There was no built in knowledge although we did adapt the neural network structure to allow an opportunity for the networks to evolve a response to the relative positions of the markers in the game as it progressed. For each round the Input neurons were set to +1 for each position that the neural network had a marker and –1 for each position occupied by it’s opponents marker. Empty cells were set to 0 (zero). Once the values had been propagated through the network then the output states of the Output layer neurons were read. The playing “choice” of the network was deemed to be the one with the highest value (in the range 0 to 1) associated with an empty playing position. Output values for cells already occupied in previous rounds were ignored. We later changed this process and read a single value from a single output neuron that assessed alternate boards following each potential move – selecting the move that the neural network assigned the highest output value to. The number and application of the output neurons and their associated values is thus dependent upon how you are applying your neural network to a given task. (See Asking the Right Question) Here at Adit Limited we are now working with the application of neural networks that have multiple logical outputs. Some of these outputs control a given process and some provide elements of the analysis of that process that the neural networks are assigned to. More Information on Neural Network structures http://www.geocities.com/CapeCanaveral/1624/ is a good place to start any research into different neural network topologies. The source code here is a generic form of C but the example programs are quite straightforward to follow – partly because they are short and to the point.
|
|
|
|
[ Home] [About Us] [Systems] [Software Review] |