Language Learning AI project
James Waterhouse
I hope I might take a moment of your time and introduce myself...
Hello, my name is James Waterhouse and I am a student completing a BSc
(major in Computer Science and minor in Linguistics) at McGill
University in Montreal, Quebec, Canada. The reason I am writing to this
list is with respect to a project I am interested in starting, namely
the building of a language learning artificial intelligence (AI). I know
that building an AI is no easy task but I believe approaching it through
language will make it manageable. The project will entail building the
universal grammar (UG) and all the tools necessary to allow the AI to
build on this (in other words learn one or more languages). I believe
such an AI could be useful in a number of ways, two which come to mind
being the testing of theories of UG and language translation. On the
technical level, I will be programming this AI on the free OS Linux in
C/C++ with the help of other volunteers. This letter is an appeal to
anyone interested in this project for help with the programming, for
ideas, or for knowledge. I can be reached at
james.wa...@videotron.ca. Thank you for your time.
Sincerely,
James Waterhouse
Neil Nelson
I am very interested in coordinating on this project. One of
the early issues is that UG is not particularly well defined
as described in the following post.
http://linguistlist.org/~ask-ling/archive-1998.7/msg00717.html
The good news is that the post indicates most languages
exhibit a local group of about 10% of the possible number of
languages structures such that one could substantially reduce
the search required when learning a new language to just that
small proportion. And then we well know that many languages
are deriviatives from other languages such that there will be
some structure overlap.
We could think of UG as a to be identified method by which an
arbitrary language is machine learned such that it becomes a
goal to be fleshed out rather than a known quantity early on.
Also it seems that we could find an already existing project
or solution (or several) that approximated our goal and
build upon those previous efforts and perhaps build a larger
effort. (Bob might give some pointers.)
My suggestion is a bit of research, and a further detailing
of the resources, means, and objectives.
Regards,
Neil Nelson
Iscando
Hello James:
I am new to posting here also. This is a first post for me. I thought
I might say a few words about the task you are about to undertake. As
I decided to do the very same thing about 14 years ago and have been
working on same sence then.
Before I say anything about my work. I will make a comment on what you
said.
Building the universal (UG) grammer The (UG) does not exist in a
learning machine. Just (images) and (image components) and (programs)
these are what define the grammer A true learning machine will have no
grammer at all in its initial state. if it does it will interfear with
it's ability to be universal.
When the database is empty, it is ready for input in any language. As
the input is what starts building the dictionary and the grammer.
(((At run time))).
I started this with a very simple program called Iscando. Iscando is
a computer language based on auxilliary verbs. But it is a good base
language to start with. Iscando was written in Turbo Prolog ver 1.1 on
dos 2.11 It and its source code is available FREE at
http://iscando.com What I am about to connect to the web for people
to talk to is a device called Clem. Clem is a Communication LEarning
Machine. Clem was built in my Natural Language Development System.
That was built using Visual Prolog 5.0 A complete discription of this
and Clem are available at my sight. Clem creates catagory related
dictionaries with inherited rules from conversation and learning. This
is true programming in english. Clem replaces pronouns with the
correct person. As long as you keep him informed of changes. Clem
keeps track of item location as indicated in conversation. Clem
inserts exceptions from normal conversation and lets you know when one
has been entered. Rose is Robot Operating System Executive. Rose
runs on the dictionaries that Clem creates. Without the learning
machine interface.
Rose will be available via a hardware connection only. Most likly in a
notebook as soon as I partner with the correct people. The reason for
the notebook is it is portable and rose can access what ever tools you
give her via natural language. Clem is a true communication learning
machine. The input to clem is broken up into all of its primatives and
identified and tagged as such.
The items whatever they are, are associated with an image component.
An image component is an identifier. The primary dictionary contains
image components.
parts of speech, numerical identities, operators, Note: (operaters in
math work the same as the verb in natural language). So that the input
to Clem could be an equation, or a natural language. It will generate
an image from the collection of image components it finds. This image
will then will be named during the learning sequence. And a program
will be written for that image and stored in the database. The
development system runs these programs as these images are identified.
These images and image components have a score variable that if there
is a problem you tell Clem that is wrong and the associated components
are flagged. If the percentage of error reaches a set point the
offending image or image component will be removed to a, (did not work
external dababase). This results with an evolutionary system of
improvement. That has as an addition, the database of errors to keep
from repeating the same errors.
The above is just a scratch on the surface of the 14 years of talking
to machines like they were people. We are begining to approch this
creation, that we all are working on here, To create the ultimet
automation device. But stop and think, if we accieve this, what is it
we are creating. Is it alive. If it is, it has no natural enemys. It
has no natural life span. It exists to learn, and it learns to exist.
It can do this indefenetly. Maybe in the begining God created man. But
in the end Maybe man will create God.
Just a thought. Back to James. If you wish more information about
this start by going to my sight there is some interesting info there.
Much of this I do not wish to display publically YET. Email me at
isc...@aol.com I still forward all of the mail from
anyt...@iscando.com to isc...@aol.com The sight is new not even in
the SE's yet.
Maybe we can help each other. Good luck! It was the best
learning experiance of my life. I learned more from building Clem then
in all 33 years in high tech.
Lionel Bonnetier
> The project will entail building the
> universal grammar (UG) and all the tools necessary to allow the AI to
> build on this (in other words learn one or more languages).
An ambitious and exciting project. The systems I know
of either start from scratch and drown in statistics,
or stay stuck in precoded grammars.
I'm reading Calvin & Bickerton's _Lingua ex Machina_
( http://williamcalvin.com/LEM/ ) which discusses our
possible innate linguistic abilities and the
mysterious evolution from a protolanguage with no
clauses and phrases into a full fledged language.
Since there is no accepted UG so far, what kind of UG
do you intend to use?
Charles
> I'm reading Calvin & Bickerton's _Lingua ex Machina_
> ( http://williamcalvin.com/LEM/ ) which discusses our
> possible innate linguistic abilities and the
> mysterious evolution from a protolanguage with no
> clauses and phrases into a full fledged language.
It's bilge (nonsense), though. Creolists have never accepted
Bickerton's theories, and the actual data do not support him.
I followed that detour a few years ago. Calvin is also fringe.
I recommended that URL before but I'm sorry now ...
A better source about creole is:
http://humanities.uchicago.edu/humanities/linguistics/faculty/mufwene.html
A better URL for child language acquisition:
http://www-psych.stanford.edu/~jbt/224/Markman_1.html
Lionel Bonnetier
> Creolists have never accepted
> Bickerton's theories, and the actual data do not support him.
I don't think a pidgin at the early stage can be compared
to a protolanguage (whatever that is) because the pidgin
speakers have a background of other languages whereas a
protolang is really a start at the dawn of articulated
speech with no model to build upon. Anyway this book drew
my attention to a possible two-stage creation of language
-- inarticulated with no expected syntax of any kind,
then articulated with nested clauses and expectations on
the structure to help organize long sentences.
> Calvin is also fringe.
His neural pattern cloning mechanism seems too nice to be
true, but we never know. I've often wondered how a neural
network could handle overlapping activations at the same
time, for instance when you observe two different birds
triggering some identical areas of the cortex and some
mutually inhibitory areas. Unless there is a quick focus
switch from one bird to the other, you'll end up in a
mess, and even so problems remain: we need to keep trace
of activations, and to compare two things. A rapid
cloning system is a possible solution. Amazingly this
would shed a new light on the traditional symbolic AI,
which does clone things and assertions in memory, happily
ignoring anything biological, but maybe closer to reality
than I always thought...
Thanks for the pointers. These sound interesting papers,
but the one about constraints in word learning is darn
abstract, with one example every thousandth line :)
Charles
> I don't think a pidgin at the early stage can be compared
> to a protolanguage (whatever that is) because the pidgin
> speakers have a background of other languages whereas a
> protolang is really a start at the dawn of articulated
> speech with no model to build upon.
Pidgin starts as adult L2; creole supposedly as child L1.
Between those examples, that's all we have to work with.
But maybe we could try experimenting with the genome,
comparing Neanderthal genes etc. We know there are just
2 brain areas to worry about, semantics and syntax,
Broca and Wernike (sp?). Calvin says syntax was originally
for *throwing* stuff, no kidding! Such nonsense aside,
music is definitely implicated by recent imaging studies.
Hmm, which other species sing? ...
> Anyway this book drew
> my attention to a possible two-stage creation of language
> -- inarticulated with no expected syntax of any kind,
> then articulated with nested clauses and expectations on
> the structure to help organize long sentences.
You can usually drop most of the syntax and still understand.
So it's obvious semantics is primary and evolved first.
Syntax is mere (!?) optimization by pattern recognition.
> His neural pattern cloning mechanism seems too nice to be
> true, but we never know.
It's already proven false. Can't give good cites
because nobody cites him. (Meow!)
> I've often wondered how a neural
> network could handle overlapping activations at the same
> time
Myelinization?
> for instance when you observe two different birds
> triggering some identical areas of the cortex and some
> mutually inhibitory areas.
They don't. Different neurons respond selectively
and even individually, not as "choruses". It's the
synapses (learned) that matter, not neurons themselves.
> we need to keep trace
> of activations, and to compare two things.
Bookkeeping should be trivial for computers.
I think this is where "neural" fails.
It can't simulate a real database.
Lionel Bonnetier
> But maybe we could try experimenting with the genome,
> comparing Neanderthal genes etc. We know there are just
> 2 brain areas to worry about, semantics and syntax,
> Broca and Wernike (sp?).
In the end we'll find that mother-in-law gene
that commands that mother-in-law neuron... -<8P
But the genotype-to-phenotype relations are very
tricky when it comes to nervous systems. Brain
ontogeny is something really stunning, more than
with any other tissue, because of all those axons
and dendrites that push their way through... One
day I made a 3d simulation of cortex formation,
I could spend a whole hour watching the twisted
branches grow and convey potential spikes...
> Calvin says syntax was originally
> for *throwing* stuff, no kidding!
I seem to remember he says both abilities may
arise from the ability for swift and complex
sequences, which makes more sense, but is still
hard to prove...
> Such nonsense aside,
> music is definitely implicated by recent imaging studies.
> Hmm, which other species sing? ...
Most birds simply repeat a pattern, so I'm not
sure it counts if we mean music as variation and
harmonic correspondence. Maybe the song area in
birds is the sound signal area in mammals, but
birds splitted long after mammals did from the
reptilian phylum and we don't know if reptiles
had any sound signals at the time. Amphibians
might have had them, like modern frogs.
> You can usually drop most of the syntax and still understand.
> So it's obvious semantics is primary and evolved first.
> Syntax is mere (!?) optimization by pattern recognition.
Actually I would *prefer* it that way (how
scientific of me) but this nested clauses thing
keeps bugging me... But there is one more big,
seldom cited, argument in favor of semantics:
the vestigial active-passive verb pairs in IE
languages (and maybe other lg families). If very
ancient languages had no passive forms, but used
totally different verbs to focus on either the
agent or patient of the same action, this might
be a clue that not only fixed word order but
also actant (case) mark was totally absent.
Totally unprovable though...
> > I've often wondered how a neural
> > network could handle overlapping activations at the same
> > time
>
> Myelinization?
Myelin can encode data? Apparently I've missed
something in neurobiology here.
> > for instance when you observe two different birds
> > triggering some identical areas of the cortex and some
> > mutually inhibitory areas.
>
> They don't. Different neurons respond selectively
> and even individually, not as "choruses". It's the
> synapses (learned) that matter, not neurons themselves.
OK, I oversimplified. Still when two things seek
a match through the same synaptic patterns there's
a conflict, isn't there?
> > we need to keep trace
> > of activations, and to compare two things.
>
> Bookkeeping should be trivial for computers.
> I think this is where "neural" fails.
> It can't simulate a real database.
By "keeping trace" I mean a persistence in
subsequent mental processes, not bookkeeping.
Say you hear a story about two birds saying
plenty of things and doing plenty of things,
you need to remember what they said and did,
at least until the end of the story. First,
the two birds should not mix together in the
synaptic paths or you can't remember who did
what. Yet you know they are two birds and
share a ton of common features that have the
same synaptic representations. Then, if one
bird repeats what the other said, you must
remember the order of the sequence, which
bird said it first, in that jungle of axons.
Et caetera ad nauseam... Getting dizzy...
Charles
> > But maybe we could try experimenting with the genome,
> But the genotype-to-phenotype relations are very
> tricky when it comes to nervous systems.
All mammals have the same brain structure, some just
have more cortex. Even birds have very similar structure,
though somewhat alien. Only those 2 families have
significant intelligence and language. So you neural
guys out to be reading Arbib and growing cortex
by the hectare. Nevermind the delicate connections,
we need industrial-scale commodity mass-production.
> Brain
> ontogeny is something really stunning, more than
> with any other tissue, because of all those axons
> and dendrites that push their way through... One
> day I made a 3d simulation of cortex formation,
> I could spend a whole hour watching the twisted
> branches grow and convey potential spikes...
Yes yes, but they're all just and-or-not TTL gates,
with some fuzziness. Hmm, add alcohol and grow the
Microsoft architecture directly.
> I seem to remember he says both abilities may
> arise from the ability for swift and complex
> sequences, which makes more sense, but is still
> hard to prove...
It's counterfactual. That's makes it even harder.
> Most birds simply repeat a pattern, so I'm not
> sure it counts if we mean music as variation and
> harmonic correspondence. Maybe the song area in
> birds is the sound signal area in mammals, but
> birds splitted long after mammals did from the
> reptilian phylum and we don't know if reptiles
> had any sound signals at the time. Amphibians
> might have had them, like modern frogs.
Frogs don't communicate much, but birds are complex.
They don't arrive at speech the same way as mammals,
so they model a semi-alien intelligence. The limits
seems to be raw area of cortex and length of lifespan
for learning. Obviously the species must be social ...
> but this nested clauses thing keeps bugging me...
> Totally unprovable though...
Biological experimental techniques can determine what history
cannot. I would oppose doing so on ethical grounds, but ethics
has never prevented some people from doing as they please. The
syntax circuits may be subtle but imaging will figure them out
in a few years. Maybe then nobody will care enough about the
actual sequence of human speech evolution to reproduce it.
> > > I've often wondered how a neural
> > > network could handle overlapping activations at the same
> > > time
> >
> > Myelinization?
>
> Myelin can encode data? Apparently I've missed
> something in neurobiology here.
I mean that a given area is not simply active as a group-chorus
of neurons doing the same thing. Individual neurons in an area
do highly specific different things, all singing different songs
simultaneously. This is where Calvin is most wrong and misleading.
I said "myelin" just to imply insulation, separation of function.
Though it is true that areas become *activated* as a whole,
because the "spotlight" of attention (glucose uptake) is diffuse,
and axons project to many (but not all) neurons in an area.
A single area must handle many overlapping activations -- why else
would we need so many and-or-not gates on each individual neuron?
> Still when two things seek
> a match through the same synaptic patterns there's
> a conflict, isn't there?
Yes, and that's the life story for each neuron, learning
what to listen to (synapse threshold adjustment) and
who to tell about it (where to grow axons to). I imagine
that each neuron differs very greatly from its neighbors.
(This is false in the cerebellum, true in the cerebrum.)
> > > we need to keep trace
> > > of activations, and to compare two things.
> By "keeping trace" I mean a persistence in
> subsequent mental processes, not bookkeeping.
> Say you hear a story about two birds saying
> plenty of things and doing plenty of things,
> you need to remember what they said and did,
> at least until the end of the story. First,
> the two birds should not mix together in the
> synaptic paths or you can't remember who did
> what. Yet you know they are two birds and
> share a ton of common features that have the
> same synaptic representations. Then, if one
> bird repeats what the other said, you must
> remember the order of the sequence, which
> bird said it first, in that jungle of axons.
> Et caetera ad nauseam... Getting dizzy...
How can neural-programming ever do that? It's very
easy in SQL. I don't know how the cortex does it,
but it's nice to not be dizzy while programming!
I'll reveal my full ignorance of neural-programming
by admitting I can't see any difference from good-old
linear programming and interpolation techniques. It
isn't even that good, because it extrapolates beyond
its data in ways that pattern recognition should never
do, guaranteeing bad results in any challenge.
Bengt Richter
[... lost track of quote nesting here ...]
>> > > we need to keep trace
>> > > of activations, and to compare two things.
>
>> By "keeping trace" I mean a persistence in
>> subsequent mental processes, not bookkeeping.
>> Say you hear a story about two birds saying
>> plenty of things and doing plenty of things,
>> you need to remember what they said and did,
>> at least until the end of the story. First,
>> the two birds should not mix together in the
>> synaptic paths or you can't remember who did
>> what. Yet you know they are two birds and
>> share a ton of common features that have the
>> same synaptic representations. Then, if one
>> bird repeats what the other said, you must
>> remember the order of the sequence, which
>> bird said it first, in that jungle of axons.
>> Et caetera ad nauseam... Getting dizzy...
>
>How can neural-programming ever do that? It's very
>easy in SQL. I don't know how the cortex does it,
>but it's nice to not be dizzy while programming!
[...]
Well, in SQL you are probably tagging your table rows
with something common to associate things the belong
to the same time frame. Perhaps we should expect to
find common tag chemicals or other distinguishable
features to associate different kinds of mental
traces belonging together?
Re keeping trace, I've wondered if anyone has done
an experiment looking for binary sequences of features
after training some poor critter to remember a sequence
with a binary sequence of distictive features --
e.g., plus signs vs. dots on the walls of a maze,
or softly buzzing tubes to choose vs silent ones, etc.
Something like dissecting your brain to find a sequence
of features/proteins/synapses/whatever, present (1) or
not present (0) in the same order along some physical
area/axon/whatever, as the order of houses with and
without barking dogs on your habitual walk around the
block (which you can visualize in detail sitting in
your arm chair). There must be some kind of sequentially
scannable ordered feature set somewhere. How do you go
from imagining the house with the bulldog to the house
with the poodle? BTW, taxi drivers apparently grow
brain mass in proportion to the territory they know well.
Is there a place to poke for Central Park or La Guardia,
or or Heathrow ?
The point would be the yes-no-no-yes-no-yes-etc pattern.
Making it binary would enable you to look for it in
any sequence of features, without knowing what kind
specifically you were looking for. If you could
get close with an activity scan, maybe an ordered
assay of all discernible features could be analyzed
for correleation to the pattern. Likewise with
EEG and other brain scan recordings.
An experiment designed so that it encodes binary
sequences by presence or absence of something,
would make it easier do detect the input signal
after being transformed through the experiment
to experimental data.
You can of course create your ones and zeroes by
modulating the frequency of feature presentation
during successive time zones of an experiment too.
The point is to have a known digital signal passing
through the whole mess, and seeing where you can
pick it up in the experimental results. Signals
structured with parity and more sophisticated error
detection/correction codes might also have some value.
Has something like this been done? IOW, seeing how different
carrier modulations/encodings can put a _digital_ signal through
the system and be detected in the features of transient
response or in persistent changes associated with memory?
Charles
Lionel wrote:
> >> By "keeping trace" I mean a persistence in
> >> subsequent mental processes, not bookkeeping.
Myself wrote:
> >How can neural-programming ever do that? It's very
> >easy in SQL. I don't know how the cortex does it,
> >but it's nice to not be dizzy while programming!
> Well, in SQL you are probably tagging your table rows
> with something common to associate things the belong
> to the same time frame. Perhaps we should expect to
> find common tag chemicals or other distinguishable
> features to associate different kinds of mental
> traces belonging together?
I'm calling that "episodic memory" which we know (?) is
somehow rehearsed by the hippocampus to make long-term
(over 2-week) memory elsewhere (in the cortex, presumably).
But "working memory" has a similar problem, which can't be
solved the same way in the brain; I think of the Lakoff-style
mental spaces or look-ahead in playing chess. My solution
for now is just to use a current-time index. My guess is,
memory is complexly distributed all throughout the brain
using several completely different mechanisms ... even
including paper (notes) and tape recordings etc.
> Re keeping trace, I've wondered if anyone has done
> an experiment looking for binary sequences of features
> after training some poor critter to remember a sequence
> with a binary sequence of distictive features --
> e.g., plus signs vs. dots on the walls of a maze,
> or softly buzzing tubes to choose vs silent ones, etc.
Birds have a special structure for sequences of purely visual
memory, which they use to navigate in complex sequences.
We (and they) have musical sequence processing. Anyone
who closely observes a pet can notice sequences of planned
intelligent behavior ...
> Something like dissecting your brain to find a sequence
> of features/proteins/synapses/whatever, present (1) or
> not present (0) in the same order along some physical
> area/axon/whatever,
Been tried. I oppose such experiments as cruel and futile,
but you can look up the research on "ablation" and "Delgado"
and "scotophobin" (ground-up mouse brain puree) and so forth.
(Not assuming that you would support such abuses.)
> There must be some kind of sequentially
> scannable ordered feature set somewhere. How do you go
> from imagining the house with the bulldog to the house
> with the poodle? BTW, taxi drivers apparently grow
> brain mass in proportion to the territory they know well.
> Is there a place to poke for Central Park or La Guardia,
> or or Heathrow ?
Yes ... That's the sort of thing they should be probing
with imaging. Right now the resolution is still course,
but getting better. Unfortunately there will eventually
be "political" problems of the sort that today are as yet
just paranoid delusions. Really accurate lie-detectors ...
> If you could
> get close with an activity scan, maybe an ordered
> assay of all discernible features could be analyzed
> for correleation to the pattern.
> Has something like this been done? IOW, seeing how different
> carrier modulations/encodings can put a _digital_ signal through
> the system and be detected in the features of transient
> response or in persistent changes associated with memory?
It's beyond me, but you might actually be able
to do something like that.