draft for comments
Pei Wang
I put a draft at
http://groups.google.com/group/open-nars/web/Evaluation.pdf for
comments and English corrections.
TITLE: The Evaluation of AGI Systems
ABSTRACT: The paper surveys the evaluation approaches used in AGI
research, and argues that the proper way of evaluation is to combine
empirical comparison with human intelligence and theoretical analysis
of the assumptions and implications of the AGI models.
Thanks in advance!
Pei
Jeff Thompson
> I put a draft at
> http://groups.google.com/group/open-nars/web/Evaluation.pdf for
> comments and English corrections.
>
"Though the evaluation of research result"
should be
"Though the evaluation of research results"
"Very common, one technique performs better"
should be
"Very commonly, one technique performs better"
"One solution proposed in the mainstream AI"
should be
"One solution proposed in mainstream AI"
"and few technique has all the desired properties"
should be
"and few techniques have all the desired properties"
"but avoid it to a large extent"
should be
"but avoids it to a large extent"
"it seems clear that the mainstream AI"
should be
"it seems clear that mainstream AI"
"AI has been driving by"
should be
"AI has been driven by"
"it seems obvious that the ultimate criteria"
should be
"it seems obvious that the ultimate criterion"
(only a single criterion is mentioned)
"there is still the question on which data should be collected"
should be
"there is still the question of which data should be collected"
"various variations of Turing Test have been proposed"
should be
"various variations of the Turing Test have been proposed"
"or to see how many human jobs it can be employed to replace human"
should be
"or to see how many human jobs it can be employed in to replace a human"
"human intelligence is a complicated phenomena"
should be
"human intelligence is a complicated phenomenon"
(only a single phenomenon is mentioned)
"may remind us the well-know story"
should be
"may remind us of the well-know story"
"Why not to take all types of human data"
should be
"Why not take all types of human data"
"different from the human brain in hardware level"
should be
"different from the human brain at the hardware level"
"by comparing they with human intelligence"
should be
"by comparing them with human intelligence"
"not the ends by itself, but a means to an ends"
should be
"not the end by itself, but a means to an end"
(refers to a single "end")
"correct and optimal in certain sense"
should be
"correct and optimal in a certain sense"
"attempt to make computers to do so"
should be
"attempt to make computers do so"
"With such a understanding of intelligence"
should be
"With such an understanding of intelligence"
"the correct or optimal solution to problems, and how to find or build them"
should be
"the correct or optimal solution to problems, and how to find or build it"
("it" refers to a single "solution")
"one of the traditional theory"
should be
"one of the traditional theories"
"to what extend a theory"
should be
"to what extent a theory"
"referred to as a revision of probabilistic model"
should be
"referred to as a revision of a probabilistic model"
"When an axiom of probability theory has be violated"
should be
"When an axiom of probability theory has to be violated"
"basic assumptions of NLP"
should be
"basic assumptions of PLN"
"Every ... and their applications into concrete domains are almost always"
should be
"Every ... and its applications into concrete domains is almost always"
("its" and "is" refer to the singular "Every")
"Such a case can be find"
should be
"Such a case can be found"
"to what extent these systems can be properly evaluated"
should be
"to what extent can these systems be properly evaluated"
"no matter how excellent the theory is in its own"
should be
"no matter how excellent the theory is on its own"
"compared with what required"
should be
"compared with what is required"
"there is no "natural" or "self-evidence" way"
should be
"there is no "natural" or "self-evident" way"
"justifiable to evaluate AGI system"
should be
"justifiable to evaluate an AGI system"
"standard for AGI, which, even possible to be achieved"
should be
"standard for AGI, which, even if possible to achieve"
"in unnecessary ways resulting "Artificial Human Intelligence""
should be
"in unnecessary ways resulting in "Artificial Human Intelligence""
"basics of human intelligence and to express it"
should be
"basics of human intelligence and to express them"
("basics" is plural)
"standard for AGI, which, even possible to be achieved"
should be
"standard for AGI, which, even if possible to achieve"
"will lack key characters of intelligence"
should be
"will lack key characteristics of intelligence"
"of intelligence as we commonly know"
should be
"of intelligence as we commonly know them"
"and lead the research in deviant paths"
should be
"and lead the research on deviant paths"
"deviant paths resulting "Artificially Designated Intelligence""
should be
"deviant paths resulting in "Artificially Designated Intelligence""
"empirical approach and theoretical approach of evaluation actually depends"
should be
"empirical approach and theoretical approach of evaluation actually depend"
"and still use them for subproblems here or there"
should be
"and still uses them for subproblems here or there"
"depend on certain learning mechanism"
should be
"depend on certain learning mechanisms"
"if NARS does something fundamental different"
should be
"if NARS does something fundamentally different"
Jeff Thompson
> I put a draft at
> http://groups.google.com/group/open-nars/web/Evaluation.pdf for
> comments and English corrections.
Comments....
You say "NARS is based on the belief that "intelligence" is the
capability of adaptation with insufficient knowledge and resources."
But in other threads, you have given answers to some questions of mine
such as:
"Beliefs like <canary --> ExistingThing> won't be confirmed too often in
experience (though it has no negative evidence), so its confidence won't
be too high, and its priority will be very low. I doubt it will cause
trouble in a realistic setting."
This sounds like you're saying that NARS won't give a wrong result
because it won't have sufficient resources to find that troublesome
result, even though it is allowed by the logic. In other words, if NARS
is given sufficient resources to explore more of the logical space, it
is more likely, not less, to come up with a wrong answer. This suggests
that NARS not only *can* operate with insufficient resources, but that
it *must*. The need to limit resources seems to be a built-in
assumption. Is that so? I wonder if you should say this in the paper.
Pei Wang
Pei
Pei Wang
Thanks for raising this interesting topic.
For a given problem, if NARS spends more time on it, usually the
solution will have a higher quality, but sometimes the solution will
indeed get worse. I believe it is also true for human beings ---
sometimes it is even better to leave a topic than to keep thinking
about it. The hard problem is to decide when this is the case.
There are people who claim that "limitations of knowledge and
computational capability need not be a disadvantage" --- see the works
by Gerd Gigerenzer. I won't go that far. To me, "limitations of
knowledge and computational capability" is the reality to be faced, so
whether it is desired or not doesn't matter too much.
However, since NARS assumes AIKR, it is not the preferred model when
the assumption is not true. For example, if a problem can be solved by
exhaustive search, then no intelligence is needed --- furthermore,
"intelligence" may make things worse here!
Since the current paper only uses NARS as an example, I won't add this
discussion into it. However, I'll probably address this issue in
another paper under preparation, which will compare models of
rationality.
Pei