Any online info: GA & NN topology?
Kimmo K K Fredriksson
construct optimal NN topology? (Couldn't find in the FAQ anyway...)
Thanks!
Hans Andersen
>Is there any information available via ftp or www about using GA to
>construct optimal NN topology? (Couldn't find in the FAQ anyway...)
There is a nice list of papers on this topic in comp.ai.genetic at
the moment. We have also written a paper on the topic which I would
be happy to send to you (or anyone else who might be interested).
Our work involves using a niche formation technique to evolve
different "species" of neurons in parallel.
Regards
Hans Andersen,
Department of Electrical and Computer Engineering,
University of Queensland,
Brisbane, Australia.
Kimmo K K Fredriksson
: Is there any information available via ftp or www about using GA to
: construct optimal NN topology? (Couldn't find in the FAQ anyway...)
Thanks to all who replied. Here are the answers (edited, to make them
shorter) I got, if someone else is interested.
1. Belew, R.K., McInerney, J., and Schraudolph, N.N.:
Evolving Network: using the genetic algorithm with
connectionist learning, In C.G. Langton, C. Taylor, J.D.
Farmer and S. Rasmussen (Eds.) Artificial Life II, Redwood
City, CA:Addison-Wesley, 1991.
2. Bellgard, M. I., Tsang, C. P.: Some Experiments on
the Use of Genetic Algorithms in a Boltzmann Machine.
International Joint Conference on Neural Networks, pp.
2645-2652, Singapore, 1991.
3. Chalmers, D.J. (1990) The Evolution of Learning: An
experiment in Genetic Connectionism, In. D.S. Touretsky,
J.L. Elman, T.J. Sejnowski, and G.E. Hinton (Eds.)
Proceedings of the 1990 Connectionists Models Summer School.
4. Fogel, D.B., Fogel, L.J., and Porto, V.W.: Evolving
Neural Networks, Biological Cybernetics, 63, 487-493, 1990.
5. Gruau, F. (1993) Genetic Synthesis of Modular Neural
Networks, In S. Forrest (Ed.) Genetic Algorithms:
Proceedings of the 5th International Conference, Morgan
Kaufman.
6. Homaifar, A. Guan, S. (1990) Training weights of
neural networks by genetic algorithms, Proceedings of the
2nd IASTED, Anaheim, CA, USA.
5. Koza, J.R. and Rice, J.P.: Genetic Generation of both
the Weights and Architecture for a Neural Network, IJCNN-91,
Seattle, WA, USA, 1991.
6. Marti, L.: Genetically Generated Neural Networks II:
Search for an Optimal Representation, IJCNN'92, Vol. II,
Baltimore, June, 1992.
7. Miller, G.F., Todd, P.M., and Hedge, S.U.: Designing
neural networks using genetic algorithms, In J.D. Schaffer
(Ed.) 3rd ICGA, San Mateo, CA:Morgan Kaufmann, 1989.
8. Srinivas, M., Patnaik, L.M. (1991) Learning Neural
Network Weights using Genetic Algorithms - Improving
Perfromance by Search-Space Reduction, International Joint
Conference on Neural Networks, (II-187-II-192), Seattle, WA,
USA.
9. Romaniuk, S.G.: Evolutionary Growth Perceptrons, In
S. Forrest (Ed.) Genetic Algorithms: Proceedings of the 5th
International Conference, Morgan Kaufmann, 1993.
10. Romaniuk, S.G.: Trans-Dimensional Learning,
International Journal of Neural Systems, Vol. 4, No. 2
(June), 171-185, 1993.
11. Romaniuk, S.G.: Learning to Learn: Automatic
Adaptation of Learning Bias, AAAI-94, Seattle, WA, USA,
1994.
12. Romaniuk, S.G.: Application of Learning to Learn to
Real-World Pattern Recognition, To appear in: International
Conference on Artificial Neural Networks and Genetic
Algorithms, (ICANNGA-95), France, 1995.
13. Whitley, D. (1989) The GENITOR Algorithm and
Selective Presssure: Why Rank-based Allocation of
Reproductive Trials is Best, ICGA-3, Morgan Kaufmann, San
Well it's german ...
You'll find a text describing my master thesis and instructions
how to obtain it via aftp. Since the whole thesis is written german,
I'll continue in german.
Diplomarbeit erhaeltlich: "Topologieveraenderndes Lernen in Neuronalen
Netzwerken mittels Genetischer Algorithmen".
In dieser Arbeit beschreibe ich ein Verfahren, wie mittels eines
Genetischen Algorithmus sowohl die Topologie eines Netzwerkes als auch die
Gewichte der synaptischen Verbindungen veraendert werden koennen. Das
Verfahren zeigt sich als einem herkoemmlichen BackProp ueberlegen.
Aus dem Inhalt:
1 Einleitung
1.1 Der Weg zum Neuronalen Netzwerk
1.2 Wo liegt das Problem?
1.3 Was sich in dieser Arbeit findet
1.4 Was sich nicht in dieser Arbeit findet
2 Genetische Algorithmen
2.1 Allgemeine Verfahrensweise
2.2 Rekombinationsmethoden
2.3 Mutationsmethoden
2.4 Andere Ansaetze
2.4.1 Warum eigentlich binaer?
2.4.2 Die definierende Laenge
2.4.3 Der GA von Holland --- genetic plan
2.4.4 Der GA von Goldberg
2.4.5 Die GA von Whitley
2.4.6 Der GA von Eshelman --- CHC
2.5 Anwendung von Genetischen Algorithmen
3 Genetische Algorithmen und Neuronale Netzwerke
3.1 Binaere Kodierung
3.2 Das XOR-Experiment
3.2.1 Eine Abschaetzung fuer die Kontrollparameter
3.2.2 Ergebnisse und Auswertung
3.3 GA und herkoemmliche Lernverfahren
3.3.1 Bewertung des Verfahrens
3.3.2 Leistungsvergleich BP und GA
3.4 Zusammenfassung der Resultate
3.5 Anwendungsbeispiele
4 Topologie-veraenderndes Lernen
4.1 Das Design Neuronaler Netzwerke
4.2 Topologieaenderung mit Genetischen Algorithmen
4.2.1 Binaere Kodierung der Topologie
4.3 Das VARTOP-Experiment
4.3.1 Warum gerade diese Werte?
4.3.2 Ergebnisse
4.4 Die Meinung der Anderen
5 Anhang
5.1 Die genetic-Bibliothek
5.1.1 Ein- und mehrstufige GA's
5.1.2 Genome und Ranglisten
5.1.3 Selektion
5.1.4 Zufallszahlen
5.1.5 Debug-Moeglichkeiten
5.2 Programmcode XOR
5.3 Programmcode VARTOP
5.4 Programmcode fuer die Bibliotheksfunktionen
5.4.1 g0-Routinen
5.4.2 Routinen fuer 1-stufigen GA
5.4.3 Routinen fuer 2-stufigen GA
5.4.4 Routinen fuer Ranglistenverwaltung
5.5 Weitere Supportmodule
5.5.1 Behandlung von Kommandozeilenargumenten
5.5.2 Debugroutinen
5.5.3 Zufallszahlengenerator
5.5.4 Diverse Hilfsfunktionen
5.5.5 Makefile zur Erzeugung der Bibliothek
Die gesamte Arbeit umfasst circa 146 Seiten.
Auf dem Rechner:
ftp.neuro.informatik.uni-kassel.de (141.51.188.3)
im Verzeichnis:
/pub/NeuralNets/We_and_our_work/papers
finden sich folgende Dateien:
ga_and_nn.diplom.ps.gz Der Text der Diplomarbeit
dj550c-pictures.tar.gz Verschiedene bunte Grafiken
excel-sheets.tar.gz Die Grafiken als MS Excel 4.0
*.xlc Dateien
current.bib.gz Das Literaturverzeichnis als
BIBTEX
und der in der Arbeit abgedruckte Programmtext:
im Verzeichnis:
/pub/NeuralNets/GA-and-NN
als Datei:
GENlib.tar.gz
Beachten Sie bitte hierbei die Lizenzbestimmungen (in der Datei LICENSE)
In dem Postscript Text finden sich an Stelle der bunten Grafiken nur leere
Seiten, nur mit Kopf- und Fusszeilen versehen. Auf diese Seiten ist dann
der entsprechende File aus dem dj-Archiv zu drucken. Diese Files wurden fuer
einen HP DJ550C erzeugt (vom Excel unter Windows), Sie werden vermutlich mit
einem 500C nicht zu drucken sein! Im excel-Archiv finden sich die erzeugenden
Dateien, wer keinen Farbdrucker hat kann sich die Dateien auf dem Bildschirm
ansehen oder in BW ausdrucken.
--
* FG Neuronale Netzwerke / Uni Kassel *
* Jochen Ruhland *
* Heinrich-Plett-Str. 40 *
* D-34132 Kassel *
* joc...@neuro.informatik.uni-kassel.de *
* Tel: +49-561-804-4376 FAX: -4244 *
_________
Dear netter,
You can get info about a neuro-genetic hybrid software from Accel Infotech
at ac...@solomon.technet.sg. It is a commercial software.
Tommie Lin
_________
Try Article "Using GA to optimize Neural Nets" by Roeland Lengers (its a bibliography)
in: comp.ai.genetic
--
Martin "Lolly" Lorenz
_______
Newsgroups: comp.ai.genetic
From: len...@latcs1.lat.oz.au (Roeland Lengers)
Subject: Using GA to optimize Neural Nets
Hello everyone,
I noticed that there where a few questions posted about using Genetic
Algorithms to optimize Neural Networks (weights as well as topology).
I am cuurently doing research on doing just this, although I haven't
figured out yet what technique to use exactly. However, I gathered
some references that might be usefull to others as well.
Here goes:
[Yao, 1993] Xin Yao: A Review of Evolutionary Artificial Neural
Networks. In: International Journal of Intteligent Systems, Vol. 8,
pages 539-567, John Wiley & Sons, Inc., 1993.
This is a very useful article, it contains lots of references to GA/NN
combinations. It describes three uses of the combination: using GA's
to evolve the weights, the topology and the learning algorithm. Very
useful.
A list on articles in the 1993 International Conference on Neural
Networks. Just to give you an idea where you could get your material.
[Adler, 1993] Dan Adler: Genetic Algorithms and Simulated Annealing:
A marriage Proposal. In: 1993 IEEE International Conference on Neural
Networks, pages 1104-1109.
[Peck, 1993] Charles C. Peck and Atam P. Dhawan: Genetic Algorithm
based Input Selection for a Neural Network Function Approximator with
Application to SSME Health Monitoring. In: 1993 IEEE International
Conference on Neural Networks, pages 1115-1122.
[Oliker, 1993] S. Oliker, M. Furst and O. Maimon: Design
Architectures and Training of Neural Networks with a Distributed
Genetic Algorithm. In: 1993 IEEE International Conference on Neural
Networks, pages 119-202.
[Koza, 1993] John R. Koza, Martin A . Keane and James P. Rice:
Performance Improvement of Machine Learning via Automatic Discovery of
Facilitating Functions as Applied to a problem of Symbolic System
Identification. In: 1993 IEEE International Conference on Neural
Networks, pages 191-198.
[McInerney, 1993] Michael McInerney and Atam P. Dhawan: Use of
Genetic Algorithms with Back Propagation in Training of Feed-Forward
Neural Networks. In: 1993 IEEE International Conference on Neural
Networks, pages 203-208.
[Ichikawa, 1993] Yoshiaki Ichikawa and Yoshikazu Ishii: Retaining
Diversity of Genetic Algorithms for Multivariable Optimization and
Neural Network Learning. In: 1993 IEEE International Conference on
Neural Networks, pages 1110-1114.
[Sin, 1993] Sam-Kit Sin and Rui J.P. deFigueoredo: A Method for the
Design of Evolutionary Multilayer Neural Networks. In: 1993 IEEE
International Conference on Neural Networks, pages 869-870.
[McDonnel, 1993] John R. McDonnel & Don Waagen: Evolving Neural
Network Connectivity. In: 1993 IEEE International Conference on Neural
Networks, pages 863-868.
Then a few articles that can be found in the neuroprose directory on
archive.cis.ohio-state.edu. Some very useful articles, just get the
INDEX file from the subdir, and check out the keywords.
[Schiffman, 1992] W. Schiffman, M. Joost and R. Werner: Synthesis and
Performance of Multilayer Neural Network Architectures. Technical
Report 16/1992. University of Koblenz, Institute fur Physics,
Koblenz, Germany, 1992. FTP: archive.cis.ohio-state.edu in
/pub/neuroprose/schiff.gann.ps.Z.
[Boers, 1992] Egbert J.W. Boers and Herman Kuiper: Biological
Metaphors and the Design of Modular Artificial Neural Networks.
Master's Thesis for Departements of Computer Science and Experimental
and Theoretical Psychology at Leiden University, the Netherlands. FTP:
archive.cis.ohio-state.edu in
/pub/neuroprose/boers.biological-metaphors.ps.Z.
This article descibes evolving a topology, not the weights.
[Perrone, 1992] Michael P. Perrone and Leon N. Cooper: When Networks
Disagree: Ensemble Methods for Hybrid Neural Networks. In: "Neural
Networks for Speech and Image Processing" R.J. Mammone, ed.
Chapman-Hall, 1993. FTP: archive.cis.ohio-state.edu in
/pub/neuroprose/perrone.MSE-averaging.ps.Z.
And for a change: a book.
[Michalewicz, 1992] Zbigniew Michalewicz: Genetic Algorithms + Data
Structures = Evolution Programs. Springer-Verlag, Berlin, 1992.
I put this in because it gives you some idea why you shouldn't use a
binary representation for your net. Altough I still haven't figured
out how I'm going to represent my NN. Most likely I'm going to use
Evolutionary Programming instead of the "Pure GA". Other articles in
this list refer to the same problem.
And for a bit more of the biological side of things:
[Nolfi, 1992] Stefano Nolfi and Domenico Parisi: Growing Neural
Networks. Technical Report PCIA-91-15, Institute of Psychology,
National Research Council, Rome. FTP: kant.irmkant.rm.cnr.it (or
150.146.7.5) /pub/econets/nolfi.growing.ps.Z.
[Nolfi, 1994] Stefano Nolfi and Domenico Parisi: `Genotypes' for
Neural Networks. In M.A. Arbib, ed.: The Handbook of Brain Theory and
Neural Networks. Bradford Books, MIT Press. The draft is in: Technical
Report 94-06. Institute of Psychology, National Research Council,
Rome. FTP: kant.irmkant.rm.cnr.it (or 150.146.7.5)
/pub/econets/nolfi.genes.ps.Z.
Another stochastical optimization technique to use for evolving the
neural net is simulated annealing. Look at this article to get an
idea of its usefulness.
[Ingber, 1992] Lester Ingber and Bruce Rosen: Genetic Algorithms and
Very Fast Simulated Reannealing: A Comparison. Mathematical and
Computer Modelling, 16(11), pages 87-100, 1992. FTP:
alumni.caltech.edu /pub/ingber/asa92_saga.ps.Z.
And the last reference is about a masters thesis by James Spofford. It
describes a combined weights/topology evolution of a NN. It has some
drawbacks, that will hopefully be solved in version 2 of GANNET. This
version will be announced soon by Dr. Kenneth Hintz (look in the file
/gannet/thesis/readme (if I remember correctly).
[Spofford, 1990] Jason Joseph Spofford: Evolving Neural Networks with
a Genetic Algorithm. Masters Thesis, George Mason University, Fairfax,
Virginia, 1990. FTP: fame.gmu.edu /gannet/thesis/thesis1..5,a,b.ps.
Hope someone out there in cyberspace finds this useful. If so, or if
someone else has got some more references that might be useful, please
mail them to me.
Thanks.
Accel Infotech
: kfre...@cc.Helsinki.FI (Kimmo K K Fredriksson) writes:
: >Is there any information available via ftp or www about using GA to
: >construct optimal NN topology? (Couldn't find in the FAQ anyway...)
: There is a nice list of papers on this topic in comp.ai.genetic at
: the moment. We have also written a paper on the topic which I would
: be happy to send to you (or anyone else who might be interested).
: Our work involves using a niche formation technique to evolve
: different "species" of neurons in parallel.
: Regards
: Hans Andersen,
: Department of Electrical and Computer Engineering,
: University of Queensland,
: Brisbane, Australia.
try wang.nnga.ps.Z from neuroprose.
try http://www.singapore.com/products/nfga too.
--
------------------------------------------------------------
Gerald Goh, M.Sc.,MSCS Phone: +65-3366997
Accel Infotech (S) Pte Ltd Fax: +65-3362833
------------------------------------------------------------