> We don't have any papers scheduled, let me know if you're interested
> in sharing any.
I would like to share:
Measuring Cognitive Abilities of Machines, Humans and Non-Human
Animals in a Unified Way: towards Universal Psychometrics
José Hernández-Orallo, David L. Dowe, M.Victoria Hernández-Lloreda
We present and develop the notion of ‘universal psychometrics’ as
a subject of study, and eventually a discipline, that focusses on the
measurement of cognitive abilities for the machine kingdom, which
comprises any (cognitive) system, individual or collective, either
artificial, biological or hybrid. Universal psychometrics can be
built, of course, upon the experience, techniques and methodologies
from (human) psychometrics, comparative cognition and related areas.
Conversely, the perspective and techniques which are being developed
in the area of machine intelligence measurement using (algorithmic)
information theory can be of much broader applicability and
implication outside artificial intelligence. This general approach to
universal psychometrics spurs the re-understanding of most (if not
all) of the big issues about the measurement of cognitive abilities,
and creates a new foundation for (re)defining and mathematically
formalising the concept of cognitive task, evaluable subject,
interface, task choice, difficulty, agent response curves, etc. We
introduce the notion of a universal cognitive test and discuss whether
(and when) it may be necessary for exploring the machine kingdom. On
the issue of intelligence and very general abilities, we also get some
results and connections with the related notions of no-free-lunch
theorems and universal priors.
http://users.dsic.upv.es/~flip/papers/TR-upsycho2012.pdf