Single Chip-VLSI-realization of a neural net for image recognition
Frank Stüpmann Steffen Rode Gundolf Geske
Neurosystems GmbH Neurosystems GmbH Institute GS, Dep. Electrical Engineering
Joachim-Jungius-Straße 9 Joachim-Jungius-Straße 9 University of Rostock
18059 Rostock, Germany 18059 Rostock, Germany Einsteinstraße 2., 18059 Rostock
[e-mail] [e-mail] [e-mail]
Abstract  It will be shown the newest results of a hardware
known with all its advantages and disadvantages. A lot of
realization of a neural net for fast decision making functions in
users will apply this chip and more than fifty percent of
real time. There is a digital micro core with several functions 
current applications of neural networks use the BPA. In the
proceeding of the learning and testing of the net, supervising of
final version it will have 100 input neurons, 60 hidden
training process and computation of some calculations in pre-
neurons and 10 output neurons. The activation of the neurons
and post-processing. The patterns are automatically presented
lies in the range of [0,1]. A sigmoid function where the final
to the network. The heart of the classifier is a trainable
value is reached asymptotically very fast is used.
integrated analog neural network structure. Because of its speed
1
the hardware realization is able to solve real time image
f (x) =
recognition problems. Simulations with a standard particle
1+ e- x
image have shown good results not only for the 2D case, but also
with ² > 0.
for the 3D case. Particle positions are extracted from image
In the backpropagation algorithm the changing of the weights
series first. In a second step, noisy particles were identified from
W of the multi-layer perceptron is realized after propagation
images.
of the input pattern i(p) (p"L) by:
I. INTRODUCTION
? W(u,v) = ?d(p)a(p) with u " Ui-1 , v " Ui
p v u
The chip is meant to be used for making decision functions in
(2 d" i d" n) and · > 0, whereas
real time. [3], [9] deal with the examination of existing
(p) (p) (p)
Å„Å‚
f`(net )(t - a ) if v " Un
v v v
ôÅ‚
hardware realization of neural nets. There are some analog
d(p) =
òÅ‚f`(net (p) ) ~ (p) Å" W(v, ~) if v " U j ; 2 d" j d" n -1
v
v
"d
v v
neural net chips [4], [5], [6], [7], [8]. In [3] it was stated that
ôÅ‚ ~
ół v
previous solutions contain some disadvantages. Thus the
is.
number of the integrated neurons is small and often on-chip
There is au(p) the activation of the unit u after propagation of
learning is not possible. The low complexity, not sufficient
the input patterns i(p) and tv(p), v"Un is the default output
for many problems, only permits a restricted number of
prescribed from the output pattern t(p) of an output unit un.
applications. Therefore the aim of the work was deduced to
The realized net has no bias values, like the implementation
contribute to the development of an fast, complex neural
of the function classification.
integrated circuit capable of learning.
The classifier consists of the units switch, classification and
III. CONTROL UNIT
control. The switch unit carries out the switching between
learning vectors and input vectors requested from the unit
The control unit controls the chip. This unit is subdivided in
classification in correspondence to the learning process or
control functions which have the following tasks during the
the working process respectively.
separate oparating phases:
The net s topology integrated in the chip in analog circuitry
is the multi-layer perceptron. The learning algorithm used is
control function phase, in which
the backpropagation algorithm (BPA). The chip uses a
functions the unit has a
SIMD-architecture. The time the data takes from input to
meaning
output in the working process is 2µs. The internal resolution
pattern- presentation of the input- learning phase
is 6 bit and the resolution of input/output is 10 bit. The chip
control and output-patterns
has analog input/output buffers. The technology used is the
weight- supervision and control learning, test and
0.6 µm CMOS-technology CUP from Austria Microsystems
control of initialization, update operating phase
(AMS). The IP-core integrated on the chip is a 32 bit micro and refresh
error-control supervision of the error learning and test
controller. The task the control function has in the chip will
in the learning- and test phase
be shown here.
phase
random-unit random numbers for learning and test
II. MULTILAYER PERCEPTRON WITH LEARNING
initialization and pattern phase
AUTOMATION
presentation
TABLE I
The net s topology integrated in the chip is the multi-layer
Tasks of control function
perceptron. The learning algorithm used is the
backpropagation algorithm (BPA), this algorithm is well
PATTERN CONTROL in every cycle in another order. This characteristic of pattern
The pattern control function controls the reading in of learn presentation is a basic condition for the success of learning.
patterns into the pattern memory. Afterwards this function The controller generates the random numbers with method of
reads the patterns from the pattern memory and presents additive congruence. The numbers are serially selected after
them to input and output of the function classification. On that [12]. Beside the simple implementation of this algorithm
this occasion every pattern is presented separately for a short the advantages are the occurrence of any possible
time. If all patterns from the pattern memory are presented combination in each cycle and the fast process because of the
and if the learning was successful, the test patterns are absence of complicated and time consuming multiplication
presented to the classificator. The output of the pattern and division instructions.
elements and also of the variable learnrate is realized as 10
bit value. IV. CONSIDERATIONS ON REALIZED ACCURACY
WEIGHT-CONTROL In different papers [13], [14] complex problem solving were
simulated with the aid of multi-layer perceptrons.
The internal representation of knowledge has to be adapted to Accordingly resolutions of 9 to 13 bit are required for
the function which has to be solved. The algorithms successful instruction. For finding the global minimum the
necessary for this purpose are realized in the function steps of updating have to be as small as possible. The
classification. The system stays in the state of learning until backpropagation algorithm requires an accuracy of weights
the error in all patterns becomes small enough and the of 12 bit. If the resolution of weight presentation is less than
adaptation of the classificator is successful. 12 bit the learning is cancelled because the changes of weight
Weight control controls the initialization of the weights, the are lower than the lowest significant bit of the weight. The
state of learning, the learn rate and the outputs for the change would be ignored. The increase of the learn rate
learning patterns. counteracts this and effects a faster converging of the net.
Before the learning weight-control forwards the initialization The magnitude of the learn rate is limited by the width of
values of the weights to the synapses of the function dales of the decreasing gradient. The 12 bit resolution is
classification. Furthermore the central refresh of weights and beyond the technological limit of the analog CMOS circuitry.
the saving to the pattern/weight memory are realized by Among other things this is constituted by the thermal noise.
weight-control. Undersized modifications at the input of the multiplier can
not effect modifications at the output. The best resolution of
ERROR CONTROL analog circuitry is reached at 8 bit [15]. According to this the
changes of weight have to be great enough to reach
During the learning process the square error is calculated measurable changes at the output and low enough to ensure a
from the digitized value handed over by the output neurons save gradient decreasing. If the weight values have two
and the expected outputs. If the calculated error is greater digits, for realization of a decreasing gradient an accuracy of
than the maximum permissible net error the instruction of the 12 bit is necessary. The limiting of the range of weight value
net has to be continued. If the error is less than the desired and the update value is necessary. According to experience
minimal error the learning is successful and the net has for the instruction of a multi-layer perceptron weight ranges
converged. If the converging fails the net has to be of Ä…24.48 are sufficient [13]. At a resolution of 12 bit the
reinitialized and to be reinstructed with a smaller learn rate. lowest significant bit (LSB) accords to a value of 0,01. But
To control the learning only the learn rate and the number of the greatest change of a weight per learn step is never higher
learning cycles are available. After the learning the test than 0.32. According to this the change of update values
patterns are presented to the classificator. The observed error within a range of Ä…0.32 can always be ensured. Assuming
is summed to a summation error. This function is also solved that the accuracy is 6 bit, the lowest change of weight is
by the error control. If greater errors appear, the net has also
to be reinstructed. It is switched over to learning mode again. ?wLSB = 0.64/(26) = 0.01.
RANDOM UNIT As written above this corresponds to the LSB at weights with
a resolution of 12 bit. On reproduction process a lower
The random function is one of the most important functions accuracy is sufficient. According to this it is reasonable to
of the controller, because it is both necessary for differ between resolution of weight memory at the learning
initialization and for pattern presentation. At the beginning of process and the reproduction process. Furthermore the global
learning the weights are initialized with low random values error calculation and the local update calculation have to be
so that the values of weights are approximately zero. The separated. The error calculation is shifted to the controller.
derivation of the logistic activation function has a maximum For calculating the error the actual values of the output
in that region. It has to be guaranteed, that the values of neurons are used. These values are read from classification in
weight are different from zero. During the learning control either case. The requirements to the resolution of
takes care of presenting each pattern to the net only once and implementation of neural networks have been analysed
among other things by Graf [9] and Spaanenburg [10]. Thus
the nets allow a resolution of learn parameters and weight SIMULATION MODEL
changes of 6 bit. This accords to our implementation.
A net with 9 outputs is chosen. 3 neurons a time comply with
V. SIMULATION OF THE USED BACKPROPAGATION one direction in the range (x, y, z). All output areas of the
NET IN THE IMAGE DEPTH RECOGNITION neurons for the x, y and z-direction are within the real
interval [0, 1]. Any real value of the output corresponds with
Due to the usage of standard backpropagation algorithm the the position of the object balance point in the particular
chip can be used in many different fields of application. direction. According to this a net output from around zero for
Owing to its operational speed the algorithm is suitable in any levels (x, y, z) signifies a location of the balance point of
real-time applications. Most of real-time recognition the object in the point of origin. All of the 3 neurons of one
problems exist in the field of image recognition. direction are trained to the same point. The redundancy is
In a MATLAB simulation the usage of the topology of the used for generating an average value from these terms.
chip for image recognition was analysed. This application Example:
deals with the recognition of an object in range on different
conditions. The purpose is not only the recognition of an Direction Output Position of the
object with a certain form but also the determination of its value balance point
of the object
position in the same procedure.
Avarage value X 0,3 3
The range of features is three-dimensional. It consists of an
neurons 1-3 (3/10)
x-y-array, four image depth levels and a background.
Avarage value Y 0,7 7
The classificator is the multi-layer perceptron described
neurons 4-6 (7/10)
before which contains one input layer, one hidden layer and
Avarage value Z 0,6 6
one output layer. The input layer is an array of 10*10 pixels.
neurons 7-9 (3/5)
The z-information is coded within this array. The output
layer contains 9 neurons. The layers are fully connected with
Fig.2 LOCATION OF THE BALANCE POINT OF THE OBJECT
one another.
Test patterns are used which consist of one background and
The outputs of the 3 neurons of one direction ought to be as
one object standing  in front . As objects rectangles are used.
close to each other as possible. For the output the dispersion
A two-dimensional picture results which corresponds to the
of the single outputs is a confident measure: If the result is
mapping of all possible positions of the object in a three-
lower than a stipulated dispersion limit it is usable. If it is
dimensional room. Only integers are admitted as pixels of an
greater it is not utilizable and a second measuring has to be
object. A finite number of patterns results which are
conducted. Because the chip has a very high throughput of
presented to the net for training.
only 2 µs it is even possible to measure a second time if the
post process of the micro controller is fast enough. The
CODING OF DEPTH INFORMATION BY GREY SCALES
confidence measure for the appraisal of the net outputs is
also capable of being implemented with small expense.
The image room contains pixels in x and y direction. The
a) b) c)
conversion of the array in a one-dimensional input vector
takes place by a successive lining up of array lines. This
vector is presented to the classificator (perceptron) parallel to
its 100 inputs. The coding of the third dimension takes place
according to the following image:
FIG2. DIFFERENT DISPERSIONS OF THE REDUNDANT
0
CLASSIFICATION RESULTS GIVE INFORMATION ABOUT THE
1
RELIABILITY OF THE RESULTS:
2
3 A) SMALL DISPERSION  POSITION IS SAFELY RECOGNIZED;
4
B) MIDDLE DISPERSION  POSITION IS RIGHTLY RECOGNIZED
WITH HIGH PROBABILITY;
Y C) GREAT DISPERSION  POSITION CAN NOT BE SAFELY
RECOGNIZED.
Z
VI. RESULTS
X
The average error in x and y-direction at undisturbed patterns
is between 1.7 % and 3.0 %. At images with objects
FIG1. THREE-DIMENSIONAL ROOM WITH OBJECTS IN THE 4 LEVELS OF disturbed with 20 % there was a average error of 3.3 % to
THE Z-DIRECTION
9.0 %.
It was proven that by the used standard algorithm width; Conf. Proc. Autonomous Minirobots for
- already learned undisturbed images can be Research and Edutainment; pages 171-180;
recognized, Paderborn, Germany; 2001
- the position of these objects can be assessed, [11] Kwon, O.-J.; Bang, S.-Y.: Comments on  The effects
- at the presentation of disturbed images the objects and of quantization on multilayer neural networks . IEEE
their positions can be recognized further. Transactions on Neural Networks; vol. 9; pages 718-
719; 1998
VII. CONCLUSIONS [12] Pfenniger, E.: Erzeugen von Zufallszahlen mittels
Schieberegister, www.ing.pfenniger.ch/zufall.html,
It is shown that the coding of depth information with the aid 1998.
of the extent of the object (distance dependent representation [13] Crains, G. and Tarassenko, L.: Precision issues for
of the extent) yields better results than intensity coding learning with analog VLSI multilayer perceptrons,
(coding by colour). Variations of the extent of the objects are IEEE Micro, vol 5, pages 54-56, 1995
no further problem for the recognizing system. [14] Hollis, P. and Harper, J. and Paulos, J.: The effects of
The task of the integrated controller core is the controlling of precision constraints in a backpropagation learning
the training patterns and the providing of the parameters for network, Neural Computation, vol 2, pages 363-373,
the weight initialization and weight change. 1990
During the working phase the controller is not needed for net
controlling. It is applicable for classical pre- and post
processes. Thus the presented chip is a self-sufficient system
consisting of simple components. Among other things
complex tasks like depth image recognition can be solved by
the chip.
VIII. REFERENCES
[1] Stüpmann, F.: Self learning neural structure  a
contribution for analog hardware realization of neural
nets, PhD thesis , University of Rostock FB ETIT,
Germany, 2001
[2] Rode, S.: Realization of a multi-layer perceptron in
analog VLSI, Master s thesis, University of Rostock
FB ETIT, Germany, 2001
[3] Lindsey, C.S.: Neural networks in hardware:
architectures, products and applications,
www.practicle.kth.se/lindsey, 1998. Lecture at Royal
Institute of Techn. Stockholm, Sweden.
[4] INTEL: 80170NW electrically trainable analog neural
network. INTEL Information Sheet E358, INTEL
Corporation, 2200 Mission College Boulevard, Santa
Clara, USA, 1990
[5] INTEL: 80170NX Neural network technology and
applications. Technical report INTEL Corporation,
2200 Mission College Boulevard, Santa Clara, USA,
1992
[6] Ramacher, U. and Rückert, U.: VLSI design of neural
networks; Kluwer, Boston, USA, 1991
[7] Masa, P. and Hoen, K. and Wallinga, H.: A high-
speed analog neural processor; IEEE Micro-Journal;
vol. 14, pages 40-50; 1994
[8] Hammerstrom, D.: A VLSI architecture for high
performance, low-cost, on-chip-learning; IEEE-
Journal; vol. II, pages 537-544; 1990;
[9] Graf, H.P. and Sackinger, E. and Jackel, L.D.: Recent
developments of electronic neural nets in north
America; Journal of VLSI Signal Processing; vol. 5;
pages 19-31; 1993
[10] Spaanenburg, L.: Training neural nets for small word