Artificial Neural Networks - A SEMINAR
1. INTRODUCTION
1.1 What is a
Neural Network?
An Artificial Neural Network (ANN) is an information
processing paradigm that is inspired by the way biological nervous systems,
such as the brain, process information. The key element of this paradigm is the
novel structure of the information processing system. It is composed of a large
number of highly interconnected processing elements (neurones) working in
unison to solve specific problems. ANNs, like people, learn by example. An ANN
is configured for a specific application, such as pattern recognition or data
classification, through a learning process. Learning in biological systems
involves adjustments to the synaptic connections that exist between the
neurones. This is true of ANNs as well.
1.2 Historical
background
Neural network simulations appear to be a recent
development. However, this field was established before the advent of
computers, and has survived at least one major setback and several eras.
Many importand advances have been boosted by the use of
inexpensive computer emulations. Following an initial period of enthusiasm, the
field survived a period of frustration and disrepute. During this period when
funding and professional support was minimal, important advances were made by
relatively few reserchers. These pioneers were able to develop convincing
technology which surpassed the limitations identified by Minsky and Papert.
Minsky and Papert, published a book (in 1969) in which they summed up a general
feeling of frustration (against neural networks) among researchers, and was
thus accepted by most without further analysis. Currently, the neural network
field enjoys a resurgence of interest and a corresponding increase in funding.
The first artificial neuron was produced in 1943 by the
neurophysiologist Warren McCulloch and the logician Walter Pits. But the
technology available at that time did not allow them to do too much.
1.3 Why use
neural networks?
Neural networks, with their remarkable ability to derive
meaning from complicated or imprecise data, can be used to extract patterns and
detect trends that are too complex to be noticed by either humans or other
computer techniques. A trained neural network can be thought of as an
"expert" in the category of information it has been given to analyse.
This expert can then be used to provide projections given new situations of
interest and answer "what if" questions.
Other advantages
include:
- Adaptive
learning: An ability to learn how to do tasks based on the data given for
training or initial experience.
- Self-Organisation:
An ANN can create its own organisation or representation of the
information it receives during learning time.
- Real
Time Operation: ANN computations may be carried out in parallel, and
special hardware devices are being designed and manufactured which take
advantage of this capability.
- Fault
Tolerance via Redundant Information Coding: Partial destruction of a
network leads to the corresponding degradation of performance. However,
some network capabilities may be retained even with major network damage.
1.4 Neural networks versus
conventional computers
Neural networks take a different approach to problem
solving than that of conventional computers. Conventional computers use an
algorithmic approach i.e. the computer follows a set of instructions in order
to solve a problem. Unless the specific steps that the computer needs to follow
are known the computer cannot solve the problem. That restricts the problem
solving capability of conventional computers to problems that we already
understand and know how to solve. But computers would be so much more useful if
they could do things that we don't exactly know how to do.
Neural networks process information in a similar way the
human brain does. The network is composed of a large number of highly
interconnected processing elements(neurones) working in parallel to solve a
specific problem. Neural networks learn by example. They cannot be programmed
to perform a specific task. The examples must be selected carefully otherwise
useful time is wasted or even worse the network might be functioning
incorrectly. The disadvantage is that because the network finds out how to
solve the problem by itself, its operation can be unpredictable.
On the other hand, conventional computers use a cognitive
approach to problem solving; the way the problem is to solved must be known and
stated in small unambiguous instructions. These instructions are then converted
to a high level language program and then into machine code that the computer
can understand. These machines are totally predictable; if anything goes wrong
is due to a software or hardware fault.
Neural networks and conventional algorithmic computers
are not in competition but complement each other. There are tasks are more
suited to an algorithmic approach like arithmetic operations and tasks that are
more suited to neural networks. Even more, a large number of tasks, require
systems that use a combination of the two approaches (normally a conventional
computer is used to supervise the neural network) in order to perform at
maximum efficiency.
2. Human and Artificial
Neurones - investigating the similarities
2.1 How the Human Brain Learns?
Much is still unknown about how the brain trains itself
to process information, so theories abound. In the human brain, a typical
neuron collects signals from others through a host of fine structures called dendrites.
The neuron sends out spikes of electrical activity through a long, thin stand
known as an axon, which splits into thousands of branches. At the end
of each branch, a structure called a synapse converts the activity
from the axon into electrical effects that inhibit or excite activity from the
axon into electrical effects that inhibit or excite activity in the connected
neurones. When a neuron receives excitatory input that is sufficiently large
compared with its inhibitory input, it sends a spike of electrical activity
down its axon. Learning occurs by changing the effectiveness of the synapses so
that the influence of one neuron on another changes.
2.2 From Human Neurones to
Artificial Neurones
We conduct these neural networks by first trying to
deduce the essential features of neurones and their interconnections. We then
typically program a computer to simulate these features. However because our
knowledge of neurones is incomplete and our computing power is limited, our
models are necessarily gross idealisations of real networks of neurones.
3. An engineering approach
3.1 A Simple Neuron
An artificial neuron is a device with many inputs and one
output. The neuron has two modes of operation; the training mode and the using
mode. In the training mode, the neuron can be trained to fire (or not), for
particular input patterns. In the using mode, when a taught input pattern is
detected at the input, its associated output becomes the current output. If the
input pattern does not belong in the taught list of input patterns, the firing
rule is used to determine whether to fire or not.
A simple neuron
3.2 Firing Rules
The firing rule is an important concept in neural
networks and accounts for their high flexibility. A firing rule determines how
one calculates whether a neuron should fire for any input pattern. It relates
to all the input patterns, not only the ones on which the node was trained.
A simple firing rule can be implemented by using Hamming
distance technique. The rule goes as follows:
Take a collection of training patterns for a node, some
of which cause it to fire (the 1-taught set of patterns) and others which
prevent it from doing so (the 0-taught set). Then the patterns not in the
collection cause the node to fire if, on comparison , they have more input
elements in common with the 'nearest' pattern in the 1-taught set than with the
'nearest' pattern in the 0-taught set. If there is a tie, then the pattern
remains in the undefined state.
For example, a 3-input neuron is taught to output 1 when
the input (X1,X2 and X3) is 111 or 101 and to output 0 when the input is 000 or
001. Then, before applying the firing rule, the truth table is;
|
X1: |
|
0 |
0 |
0 |
0 |
1 |
1 |
1 |
1 |
|
X2: |
|
|
0 |
1 |
1 |
0 |
0 |
1 |
1 |
|
X3: |
|
0 |
1 |
0 |
1 |
0 |
1 |
0 |
1 |
|
|
|
|
|
|
|
|
|
|
|
|
OUT: |
|
0 |
0 |
0/1 |
0/1 |
0/1 |
1 |
0/1 |
1 |
As an example of the way the firing rule is applied, take
the pattern 010. It differs from 000 in 1 element, from 001 in 2 elements, from
101 in 3 elements and from 111 in 2 elements. Therefore, the 'nearest' pattern
is 000 which belongs in the 0-taught set. Thus the firing rule requires that
the neuron should not fire when the input is 001. On the other hand, 011 is
equally distant from two taught patterns that have different outputs and thus
the output stays undefined (0/1).
By applying the firing in every column the
following truth table is obtained;
|
X1: |
|
0 |
0 |
0 |
0 |
1 |
1 |
1 |
1 |
|
X2: |
|
0 |
0 |
1 |
1 |
0 |
0 |
1 |
1 |
|
X3: |
|
0 |
1 |
0 |
1 |
0 |
1 |
0 |
1 |
|
|
|
|
|
|
|
|
|
|
|
|
OUT: |
|
0 |
0 |
0 |
0/1 |
0/1 |
1 |
1 |
1 |
The difference between the two truth tables is called the
generalisation of the neuron. Therefore the firing rule gives the
neuron a sense of similarity and enables it to respond 'sensibly' to patterns
not seen during training.
3.3 Pattern Recognition - an example
An important application of neural networks is pattern
recognition. Pattern recognition can be implemented by using a feed-forward
(figure 1) neural network that has been trained accordingly. During training,
the network is trained to associate outputs with input patterns. When the
network is used, it identifies the input pattern and tries to output the
associated output pattern. The power of neural networks comes to life when a
pattern that has no output associated with it, is given as an input. In this
case, the network gives the output that corresponds to a taught input pattern
that is least different from the given pattern.
Figure 1.
For example:
The network of figure 1
is trained to recognise the patterns T and H. The associated patterns are all
black and all white respectively as shown below.
If we represent black squares with 0 and white squares with
1 then the truth tables for the 3 neurones after generalisation are;
X11: |
|
0 |
0 |
0 |
0 |
1 |
1 |
1 |
1 |
|
X12: |
|
0 |
0 |
1 |
1 |
0 |
0 |
1 |
1 |
|
X13: |
|
0 |
1 |
0 |
1 |
0 |
1 |
0 |
1 |
|
|
|
|
|
|
|
|
|
|
|
|
OUT: |
|
0 |
0 |
1 |
1 |
0 |
0 |
1 |
1 |
Top
neuron
|
X21: |
|
0 |
0 |
0 |
0 |
1 |
1 |
1 |
1 |
|
X22: |
|
0 |
0 |
1 |
1 |
0 |
0 |
1 |
1 |
|
X23: |
|
0 |
1 |
0 |
1 |
0 |
1 |
0 |
1 |
|
|
|
|
|
|
|
|
|
|
|
|
OUT: |
|
1 |
0/1 |
1 |
0/1 |
0/1 |
0 |
0/1 |
0 |
Middle
neuron
|
X21: |
|
0 |
0 |
0 |
0 |
1 |
1 |
1 |
1 |
|
X22: |
|
0 |
0 |
1 |
1 |
0 |
0 |
1 |
1 |
|
X23: |
|
0 |
1 |
0 |
1 |
0 |
1 |
0 |
1 |
|
|
|
|
|
|
|
|
|
|
|
|
OUT: |
|
1 |
0 |
1 |
1 |
0 |
0 |
1 |
0 |
Bottom
neuron
From the tables it can be seen the following
associasions can be extracted:
In this
case, it is obvious that the output should be all blacks since the input
pattern is almost the same as the 'T' pattern.
Here also,
it is obvious that the output should be all whites since the input pattern is
almost the same as the 'H' pattern.
Here, the top row is 2
errors away from the a T and 3 from an H. So the top output is black. The
middle row is 1 error away from both T and H so the output is random. The
bottom row is 1 error away from T and 2 away from H. Therefore the output is
black. The total output of the network is still in favour of the T shape.
3.4 A more complicated neuron
The previous neuron doesn't do anything that conventional
conventional computers don't do already. A more sophisticated neuron (figure 2)
is the McCulloch and Pitts model (MCP). The difference from the previous model
is that the inputs are 'weighted', the effect that each input has at decision
making is dependent on the weight of the particular input. The weight of an
input is a number which when multiplied with the input gives the weighted
input. These weighted inputs are then added together and if they exceed a
pre-set threshold value, the neuron fires. In any other case the neuron does
not fire.
Figure 2. An MCP neuron
In mathematical terms, the neuron fires if and
only if;
X1W1 + X2W2 + X3W3 + ...
> T
The addition of input
weights and of the threshold makes this neuron a very flexible and powerful
one. The MCP neuron has the ability to adapt to a particular situation by
changing its weights and/or threshold. Various algorithms exist that cause the
neuron to 'adapt'; the most used ones are the Delta rule and the back error propagation.
The former is used in feed-forward networks and the latter in feedback
networks.
3.5 Transfer Function
The behaviour of an ANN (Artificial Neural Network)
depends on both the weights and the input-output function (transfer function)
that is specified for the units. This function typically falls into one of
three categories:
·
Linear (or ramp)
·
Threshold
·
Piecewise linear
·
Sigmoid
·
Tangent hyperbolic
For linear units, the output
activity is proportional to the total weighted output.
linear
For threshold
units, the output is set at one of two levels, depending on whether
the total input is greater than or less than some threshold value.
Threshold
Compared to the threshold
activation function piecewise linear function is
more natural. Just before it reaches the threshold, it begins to rise,
so the neurons is found in other stages than just "firing" or
"not-firing" This means that the network becomes more dynamic in it's
behaviour
Piece wise linear
For sigmoid
units, the output varies continuously but not linearly as the input
changes. Sigmoid units bear a greater resemblance to real neurones than do
linear or threshold units.
The tangent hyberbolic activation-function
is a lot like the Sigmoid, except that it can give a negative output.
4. ARCHITECTURE
OF NEURAL NETWORKS
Fundamentally it is divided into three
different classes of network architecture.
·
Single layer feed forward networks
·
Multi layer feed forward networks
·
Recurrent networks
Single layer feed forward networks
A
feedforward neural network is an artificial neural network where
connections between the units do not form a directed cycle.
This is different from recurrent neural networks.
The
feedforward neural network was the first and arguably simplest type of
artificial neural network devised. In this network, the information moves in
only one direction, forward, from the input nodes, through the hidden nodes (if
any) and to the output nodes. There are no cycles or loops in the network.
Single-layer feed forward networks
The
earliest kind of neural network is a single-layer perceptron network,
which consists of a single layer of output nodes; the inputs are fed directly
to the outputs via a series of weights. In this way it can be considered the
simplest kind of feed-forward network. The sum of the products of the weights
and the inputs is calculated in each node, and if the value is above some
threshold (typically 0) the neuron fires and takes the activated value
(typically 1); otherwise it takes the deactivated value (typically -1). Neurons
with this kind of activation function are also called McCulloch-Pitts
neurons or threshold neurons. In the literature the term perceptron
often refers to networks consisting of just one of these units. They were
described by Warren McCulloch and Walter Pitts
in the 1940s.
The
Multilayer feed forward Networks
The following diagram illustrates a perceptron
network with three layers:
This network has an input layer (on the left) with
three neurons, one hidden layer (in the middle) with three neurons and
an output layer (on the right) with three neurons.
There is one neuron in the input layer for each predictor
variable. In the case of categorical variables, N-1 neurons are used to represent
the N categories of the variable.
Input Layer: A vector of predictor variable values (x1...xp)
is presented to the input layer. The input layer (or processing before the
input layer) standardizes these values by subtracting the median and dividing
by the interquartile range and distributes the values to each of the neurons in
the hidden layer. In addition to the predictor variables, there is a constant
input of 1.0, called the bias that is fed to each of the hidden layers;
the bias is multiplied by a weight and added to the sum going into the neuron.
Hidden Layer: Arriving at a
neuron in the hidden layer, the value from each input neuron is multiplied by a
weight (wji), and the resulting weighted values are added
together producing a combined value uj. The weighted sum (uj)
is fed into a transfer function, σ, which outputs a value hj.
The outputs from the hidden layer are distributed to the output layer.
Output Layer :Arriving at a neuron in the output layer, the value from
each hidden layer neuron is multiplied by a weight (wkj), and
the resulting weighted values are added together producing a combined value vj.
The weighted sum (vj) is fed into a transfer function, σ,
which outputs a value yk. The y values are the outputs
of the network.
If a regression analysis is being performed with a
continuous target variable, then there is a single neuron in the output layer,
and it generates a single y value. For classification problems with categorical
target variables, there are N neurons in the output layer producing N
values, one for each of the N categories of the target variable.
Recurrent network
Contrary
to feedforward networks, recurrent neural networks (RNs) are models with
bi-directional data flow. While a feedforward network propagates data linearly
from input to output, RNs also propagate data from later processing stages to
earlier stages.
Recurrent
Networks
A
simple recurrent network is a
variation on the Multi-Layer Perceptron, sometimes called an "Elman
network" due to its invention by Jeff Elman. A three-layer network is used, with
the addition of a set of "context units" in the input layer. There
are connections from the middle (hidden) layer to these context units fixed
with a weight of one. At each time step, the input is propagated in a standard
feed-forward fashion, and then a learning rule (usually back-propagation) is
applied. The fixed back connections result in the context units always
maintaining a copy of the previous values of the hidden units (since they
propagate over the connections before the learning rule is applied). Thus the
network can maintain a sort of state, allowing it to perform such tasks as
sequence-prediction that are beyond the power of a standard Multi-Layer
Perceptron.
In
a fully recurrent network, every neuron receives inputs from every other
neuron in the network. These networks are not arranged in layers. Usually only
a subset of the neurons receive external inputs in addition to the inputs from
all the other neurons, and another disjunct subset of neurons report their
output externally as well as sending it to all the neurons. These distinctive
inputs and outputs perform the function of the input and output layers of a
feed-forward or simple recurrent network, and also join all the other neurons
in the recurrent processing.
Perceptron
This is a very simple model and consists of a single
`trainable' neuron. Trainable means that its threshold and input weights are
modifiable. Inputs are presented to the neuron and each input has a desired
output (determined by us). If the neuron doesn't give the desired output, then
it has made a mistake. To rectify this, its threshold and/or input weights must
be changed. How this change is to be calculated is determined by the learning
algorithm.
The output of the perceptron is constrained to boolean
values - (true,false), (1,0), (1,-1) or whatever. This is not a limitation
because if the output of the perceptron were to be the input for something
else, then the output edge could be made to have a weight. Then the output
would be dependant on this weight.
The perceptron looks like -
- x1,
x2, ..., xn are inputs. These could be real numbers or
boolean values depending on the problem.
- y
is the output and is boolean.
- w1,
w2, ..., wn are weights of the edges and are real
valued.
- T
is the threshold and is real valued.
The output y is 1 if
the net input which is
w1 x1 + w2 x2
+ ... + wn xn
is greater than the threshold T. Otherwise the
output is zero.
The idea is that we should be able to train this
perceptron to respond to certain inputs with certain desired outputs. After the
training period, it should be able to give reasonable outputs for any kind of
input. If it wasn't trained for that input, then it should try to find the best
possible output depending on how it was trained.
So during the training period we will present the
perceptron with inputs one at a time and see what output it gives. If the
output is wrong, we will tell it that it has made a mistake. It should then
change its weights and/or threshold properly to avoid making the same mistake
later.
Note that the model of the perceptron normally given is
slightly different from the one pictured here.Usually, the inputs are not
directly fed to the trainable neuron but are modified by some
"preprocessing units". These units could be arbitrarily complex,
meaning that they could modify the inputs in any way. These units have been deliberately
eliminated from our picture, because it would be helpful to know what can be
achieved by just a single trainable neuron, without all its "powerful
friends".
To understand the kinds of things that can be done using
a perceptron, we shall see a rather simple example of its use - Compute the
logical operations "and", "or", "not" of some
given boolean variables.
Computing
"and": There are n inputs, each
either a 0 or 1. To compute the logical "and" of these n inputs, the
output should be 1 if and only if all the inputs are 1. This can easily be
achieved by setting the threshold of the perceptron to n. The weights of all
edges are 1. The net input can be n only if all the inputs are active.
Input | Desired Output --------|---------------- 1 1 | 1 1 0 | 0 0 1 | 0 0 0 | 0
Computing
"or": It is also simple to see
that if the threshold is set to 1, then the output will be 1 if atleast one
input is active. The perceptron in this case acts as the logical
"or".
Input | Desired Output --------|---------------- 1 1 | 1 1 0 | 1 0 1 | 1 0 0 | 0
Computing
"not": The logical
"not" is a little tricky, but can be done. In this case, there is
only one boolean input. Let the weight of the edge be -1, so that the input
which is either 0 or 1 becomes 0 or -1. Set the threshold to 0. If the input is
0, the threshold is reached and the output is 1. If the input is -1, the
threshold is not reached and the output is 0.
The
XOR Problem
There are problems which cannot be solved by any perceptron.
Infact there are more such problems than problems which can be solved using
perceptrons. The most often quoted example is the XOR problem - build a
perceptron which takes 2 boolean inputs and outputs the XOR of them. What we
want is a perceptron which will output 1 if the two inputs are different and 0
otherwise.
Input | Desired Output --------|---------------- 0 0 | 0 0 1 | 1 1 0 | 1 1 1 | 0
Consider the following perceptron
as an attempt to solve the problem -
If the inputs are both
0, then net input is 0 which is less than the threshold (0.5). So the output is
0 - desired output.
If one of the inputs is
0 and the other is 1, then the net input is 1. This is above threshold, and so
the output 1 is obtained.
But the given perceptron fails for the last case.
5. The Learning Process
The memorisation of patterns and the subsequent response
of the network can be categorised into two general paradigms:
·
Associative
mapping in which the network learns
to produce a particular pattern on the set of input units whenever another
particular pattern is applied on the set of input units. The associtive mapping
can generally be broken down into two mechanisms:
o
Auto-association: an
input pattern is associated with itself and the states of input and output
units coincide. This is used to provide pattern completition, ie to produce a
pattern whenever a portion of it or a distorted pattern is presented. In the
second case, the network actually stores pairs of patterns building an
association between two sets of patterns.
o
Hetero-association: is
related to two recall mechanisms:
§
nearest-neighbour
recall, where the output pattern produced corresponds to the input pattern
stored, which is closest to the pattern presented, and
§
interpolative recall,
where the output pattern is a similarity dependent interpolation of the
patterns stored corresponding to the pattern presented. Yet another paradigm,
which is a variant associative mapping is classification, ie when there is a
fixed set of categories into which the input patterns are to be classified.
·
Regularity detection in which units learn to respond to particular properties
of the input patterns. Whereas in asssociative mapping the network stores the
relationships among patterns, in regularity detection the response of each unit
has a particular 'meaning'. This type of learning mechanism is essential for
feature discovery and knowledge representation.
Every neural network posseses knowledge which is
contained in the values of the connections weights. Modifying the knowledge
stored in the network as a function of experience implies a learning rule for
changing the values of the weights.
Information is stored in the weight
matrix W of a neural network. Learning is the determination of the weights.
Following the way learning is performed, we can distinguish two major
categories of neural networks:
·
Fixed networks in which the weights cannot be changed, ie dW/dt=0. In
such networks, the weights are fixed a priori according to the problem to
solve.
·
Adaptive
networks which are able to change
their weights, ie dW/dt not= 0.
All learning methods used for adaptive neural networks
can be classified into two major categories:
·
Supervised Learning
·
Unsupervised learning
Supervised
learning
In supervised training, both the
inputs and the outputs are provided. The network then processes the inputs and
compares its resulting outputs against the desired outputs. Errors are then
propagated back through the system, causing the system to adjust the weights
which control the network. This process occurs over and over as the weights are
continually tweaked. The set of data which enables the training is called the
"training set." During the training of a network the same set of data
is processed many times as the connection weights are ever refined.
The current commercial network development packages
provide tools to monitor how well an artificial neural network is converging on
the ability to predict the right answer. These tools allow the training process
to go on for days, stopping only when the system reaches some statistically
desired point, or accuracy. However, some networks never learn. This could be
because the input data does not contain the specific information from which the
desired output is derived. Networks also don't converge if there is not enough data
to enable complete learning. Ideally, there should be enough data so that part
of the data can be held back as a test. Many layered networks with multiple
nodes are capable of memorizing data. To monitor the network to determine if
the system is simply memorizing its data in some nonsignificant way, supervised
training needs to hold back a set of data to be used to test the system after
it has undergone its training. (Note: memorization is avoided by not having too
many processing elements.)
If a network simply can't solve the problem, the designer
then has to review the input and outputs, the number of layers, the number of
elements per layer, the connections between the layers, the summation,
transfer, and training functions, and even the initial weights themselves.
Those changes required to create a successful network constitute a process
wherein the "art" of neural networking occurs.
Another part of the designer's creativity governs the
rules of training. There are many laws (algorithms) used to implement the
adaptive feedback required to adjust the weights during training. The most
common technique is backward-error propagation, more commonly known as
back-propagation. These various learning techniques are explored in greater
depth later in this report.
Yet, training is not just a technique. It involves a
"feel," and conscious analysis, to insure that the network is not
overtrained. Initially, an artificial neural network configures itself with the
general statistical trends of the data. Later, it continues to
"learn" about other aspects of the data which may be spurious from a
general viewpoint.
When finally the system has been correctly trained, and
no further learning is needed, the weights can, if desired, be
"frozen." In some systems this finalized network is then turned into
hardware so that it can be fast. Other systems don't lock themselves in but
continue to learn while in production use.
Unsupervised, or Adaptive Learning.
The other type of training is called
unsupervised training. In unsupervised training, the network is provided with
inputs but not with desired outputs. The system itself must then decide what
features it will use to group the input data. This is often referred to as
self-organization or adaption.
At the present time, unsupervised learning is not well
understood. This adaption to the environment is the promise which would enable
science fiction types of robots to continually learn on their own as they
encounter new situations and new environments. Life is filled with situations
where exact training sets do not exist. Some of these situations involve
military action where new combat techniques and new weapons might be
encountered. Because of this unexpected aspect to life and the human desire to
be prepared, there continues to be research into, and hope for, this field.
Yet, at the present time, the vast bulk of neural network work is in systems
with supervised learning. Supervised learning is achieving results.
One of the leading researchers into unsupervised learning
is Tuevo Kohonen, an electrical engineer at the Helsinki University of
Technology. He has developed a self-organizing network, sometimes called an
auto-associator, that learns without the benefit of knowing the right answer.
It is an unusual looking network in that it contains one single layer with many
connections. The weights for those connections have to be initialized and the
inputs have to be normalized. The neurons are set up to compete in a
winner-take-all fashion.
Kohonen continues his research into networks that are
structured differently than standard, feedforward, back-propagation approaches.
Kohonen's work deals with the grouping of neurons into fields. Neurons within a
field are "topologically ordered." Topology is a branch of mathematics
that studies how to map from one space to another without changing the
geometric configuration. The three-dimensional groupings often found in
mammalian brains are an example of topological ordering.
Kohonen has pointed out that the lack of topology in neural
network models make today's neural networks just simple abstractions of the
real neural networks within the brain. As this research continues, more
powerful self learning networks may become possible. But currently, this field
remains one that is still in the laboratory.
5.1 Learning Rates.
The rate at which ANNs learn depends
upon several controllable factors. In selecting the approach there are many
trade-offs to consider. Obviously, a slower rate means a lot more time is spent
in accomplishing the off-line learning to produce an adequately trained system.
With the faster learning rates, however, the network may not be able to make
the fine discriminations possible with a system that learns more slowly.
Researchers are working on producing the best of both worlds.
Generally, several factors besides time have to be
considered when discussing the off-line training task, which is often described
as "tiresome." Network
complexity, size, paradigm selection, architecture, type of learning rule or
rules employed, and desired accuracy must all be considered. These factors
play a significant role in determining how long it will take to train a
network. Changing any one of these factors may either extend the training time
to an unreasonable length or even result in an unacceptable accuracy.
Most learning functions have some provision for a
learning rate, or learning constant. Usually this term is positive and between
zero and one. If the learning rate is greater than one, it is easy for the
learning algorithm to overshoot in correcting the weights, and the network will
oscillate. Small values of the learning rate will not correct the current error
as quickly, but if small steps are taken in correcting errors, there is a good
chance of arriving at the best minimum convergence.
5.2 Learning Laws.
Many learning laws are in common
use. Most of these laws are some sort of variation of the best known and oldest
learning law, Hebb's Rule. Research into different learning functions continues
as new ideas routinely show up in trade publications. Some researchers have the
modeling of biological learning as their main objective. Others are
experimenting with adaptations of their perceptions of how nature handles
learning. Either way, man's understanding of how neural processing actually
works is very limited. Learning is certainly more complex than the
simplifications represented by the learning laws currently developed. A few of
the major laws are presented as examples.
Hebb's Rule: The first, and undoubtedly the best known, learning rule
was introduced by Donald Hebb. The description appeared in his book The
Organization of Behavior in 1949. His basic rule is: If a neuron receives
an input from another neuron, and if both are highly active (mathematically
have the same sign), the weight between the neurons should be strengthened.
Hopfield Law: It is similar to Hebb's rule with the exception that it
specifies the magnitude of the strengthening or weakening. It states, "if
the desired output and the input are both active or both inactive, increment
the connection weight by the learning rate, otherwise decrement the weight by
the learning rate."
The Delta Rule: This rule is a further variation of Hebb's Rule. It is one
of the most commonly used. This rule is based on the simple idea of
continuously modifying the strengths of the input connections to reduce the
difference (the delta) between the desired output value and the actual output
of a processing element. This rule changes the synaptic weights in the way that
minimizes the mean squared error of the network. This rule is also referred to
as the Widrow-Hoff Learning Rule and the Least Mean Square (LMS) Learning Rule.
The way that the Delta Rule works is that the delta error
in the output layer is transformed by the derivative of the transfer function
and is then used in the previous neural layer to adjust input connection
weights. In other words, this error is back-propagated into previous layers one
layer at a time. The process of back-propagating the network errors continues
until the first layer is reached. The network type called Feedforward,
Back-propagation derives its name from this method of computing the error term.
When using the delta rule, it is important to ensure that
the input data set is well randomized. Well ordered or structured presentation
of the training set can lead to a network which can not converge to the desired
accuracy. If that happens, then the network is incapable of learning the
problem.
The Gradient Descent
Rule: This rule is similar to the Delta
Rule in that the derivative of the transfer function is still used to modify
the delta error before it is applied to the connection weights. Here, however,
an additional proportional constant tied to the learning rate is appended to
the final modifying factor acting upon the weight. This rule is commonly used,
even though it converges to a point of stability very slowly.
It has been shown that different learning rates for
different layers of a network help the learning process converge faster. In
these tests, the learning rates for those layers close to the output were set
lower than those layers near the input. This is especially important for
applications where the input data is not derived from a strong underlying
model.
Kohonen's Learning
Law: This procedure, developed by Teuvo
Kohonen, was inspired by learning in biological systems. In this procedure, the
processing elements compete for the opportunity to learn, or update their
weights. The processing element with the largest output is declared the winner
and has the capability of inhibiting its competitors as well as exciting its
neighbors. Only the winner is permitted an output, and only the winner plus its
neighbors are allowed to adjust their connection weights.
Further, the size of the neighborhood can vary during the
training period. The usual paradigm is to start with a larger definition of the
neighborhood, and narrow in as the training process proceeds. Because the
winning element is defined as the one that has the closest match to the input
pattern, Kohonen networks model the distribution of the inputs. This is good
for statistical or topological modeling of the data and is sometimes referred
to as self-organizing maps or self-organizing topologies.
6. Applications of neural networks
6.1 Neural Networks in Practice
Given this description of neural networks and how they
work, what real world applications are they suited for? Neural networks have
broad applicability to real world business problems. In fact, they have already
been successfully applied in many industries.
Since neural networks are best at identifying patterns or
trends in data, they are well suited for prediction or forecasting needs
including:
·
Sales forecasting
·
Industrial process control
·
Customer research
·
Data validation
·
Risk management
·
Target marketing
But to give you some more specific examples; ANN are also
used in the following specific paradigms: recognition of speakers in
communications; diagnosis of hepatitis; recovery of telecommunications from
faulty software; interpretation of multimeaning Chinese words; undersea mine
detection; texture analysis; three-dimensional object recognition; hand-written
word recognition; and facial recognition.
6.2 Neural
networks in medicine
Artificial Neural Networks (ANN) are currently a 'hot'
research area in medicine and it is believed that they will receive extensive
application to biomedical systems in the next few years. At the moment, the
research is mostly on modelling parts of the human body and recognising
diseases from various scans (e.g. cardiograms, CAT scans, ultrasonic scans,
etc.).
Neural networks are ideal in recognising diseases using
scans since there is no need to provide a specific algorithm on how to identify
the disease. Neural networks learn by example so the details of how to
recognise the disease are not needed. What is needed is a set of examples that
are representative of all the variations of the disease. The quantity of
examples is not as important as the 'quantity'. The examples need to be
selected very carefully if the system is to perform reliably and efficiently.
6.2.1 Modelling and Diagnosing the
Cardiovascular System
Neural Networks are used experimentally to model the
human cardiovascular system. Diagnosis can be achieved by building a model of
the cardiovascular system of an individual and comparing it with the real time
physiological measurements taken from the patient. If this routine is carried
out regularly, potential harmful medical conditions can be detected at an early
stage and thus make the process of combating the disease much easier.
6.2.2 Electronic noses
ANNs are used experimentally to implement electronic
noses. Electronic noses have several potential applications in telemedicine.
Telemedicine is the practice of medicine over long distances via a
communication link.
6.2.3 Instant
Physician
An application developed in the mid-1980s called the
"instant physician" trained an autoassociative memory neural network
to store a large number of medical records, each of which includes information
on symptoms, diagnosis, and treatment for a particular case. After training,
the net can be presented with input consisting of a set of symptoms; it will
then find the full stored pattern that represents the "best"
diagnosis and treatment.
6.3 Neural
Networks in business
Business is a diverted field with several general areas
of specialisation such as accounting or financial analysis. Almost any neural
network application would fit into one business area or financial analysis.
There is some potential for using neural networks for
business purposes, including resource allocation and scheduling. There is also
a strong potential for using neural networks for database mining, that is,
searching for patterns implicit within the explicitly stored information in
databases.
6.3.1 Marketing
There is a marketing application which has been
integrated with a neural network system. The Airline Marketing Tactician (a
trademark abbreviated as AMT) is a computer system made of various intelligent
technologies including expert systems. A feedforward neural network is
integrated with the AMT and was trained using back-propagation to assist the
marketing control of airline seat allocations. The system is used to monitor and
recommend booking advice for each departure. Such information has a direct
impact on the profitability of an airline and can provide a technological
advantage for users of the system.
6.3.2 Credit
Evaluation
The HNC company, founded by Robert Hecht-Nielsen, has
developed several neural network applications. One of them is the Credit
Scoring system which increase the profitability of the existing model up to
27%.
7. Conclusion
The computing world has a lot to gain fron neural
networks. Their ability to learn by example makes them very flexible and
powerful. Furthermore there is no need to devise an algorithm in order to
perform a specific task; i.e. there is no need to understand the internal
mechanisms of that task. They are also very well suited for real time systems
because of their fast responseand computational times which are due to their
parallel architecture.
Neural networks also contribute to other areas of
research such as neurology and psychology. They are regularly used to model
parts of living organisms and to investigate the internal mechanisms of the
brain.
Perhaps the most exciting aspect of neural networks is
the possibility that some day 'consious' networks might be produced. There is a
number of scientists arguing that conciousness is a 'mechanical' property and
that 'consious' neural networks are a realistic possibility.
Finally, I would like to state that even though neural
networks have a huge potential we will only get the best of them when they are
intergrated with computing, AI, fuzzy logic and related subjects.
BIBLIOGRAPHY
- Simon Haykin, Neural Networks,
- Robert Schalkoff, Artificial Neural Networks,
- http://en.wikipedia.org/wiki/Neural_network
- http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html
- http://www.cs.stir.ac.uk/~lss/NNIntro/InvSlides.html
- http://www.neoxi.com/NNR/index.html
- http://www.mathworks.com/ & http://www.mathtools.net/.
Comments
Post a Comment