Dissertation Announcement: Learning from Human-Generated Reward
31 views
Skip to first unread message
W Bradley Knox
May 9, 2013, 1:14:29 PM5/9/13
to rl-...@googlegroups.com, hri-anno...@cs.byu.edu, ml-...@googlegroups.com, robotics-...@duerer.usc.edu
=========
Greetings,
I'm proud to announce (somewhat belatedly) my PhD dissertation, defended in August 2012 at the University of Texas at Austin. Entitled
"Learning from Human-Generated Reward"
and written with Peter Stone's supervision, the dissertation has been named runner-up for the IFAAMAS Victor Lesser Distinguished Dissertation Award, which will be given out at the AAMAS banquet tonight. The dissertation contains research that received the AAMAS 2010 Pragnesh Jay Modi Best Student Paper Award and also was a finalist for the Ro-Man 2012 CoTeSys Cognitive Robotics Best Paper Award. (Additionally, for those at AAMAS, Guangliang Li will present a paper Friday morning in Session E5 that builds upon the user interface side of this dissertation work, a paper written in collaboration with Shimon Whiteson, Hayley Hung, and myself.)
The dissertation focuses roughly on the question: If reward in a reinforcement learning framework is given by a live human trainer as he or she observes the agent's behavior—rather than from the usual pre-coded reward function—how should an agent use these feedback signals to best learn the behavior that the human intends to teach?
The dissertation formalizes this learning problem, which we call interactive shaping (in Ch. 2); contains a thorough review of past work on this topic, noting the interesting pattern that all such work used relatively myopic temporal discounting (Ch. 2); proposes the TAMER framework as an approach to the problem and describes the application of TAMER to robot learning (Ch. 3); through "TAMER+RL", uses human feedback to improve upon reinforcement learning when pre-coded reward is also received by the agent (Ch. 4); examines the effects on the trainer of various manipulations, including making the agent misbehave whenever the trainer reduces his or her frequency of feedback (Ch. 5); and an exploration of the effect on task performance of employing various temporal discounting rates, leading to the first non-myopic algorithm to learn from human reward (Ch. 6). For those of you who are already familiar with TAMER, I recommend Chapter 6, which I believe contributes significant insight to the problem of learning from human reward.
A video of my defense and accompanying slides can be found at my dissertation page:
http://web.media.mit.edu/~bradknox/Dissertation.html
The dissertation document itself can be found here:
http://www.cs.utexas.edu/~bradknox/papers/thesis-knox.pdf
Additionally, I'm slowly but surely preparing the TAMER codebase for release and hope to announce soon. If you're interested in receiving the code now, email me.
I sincerely hope that this dissertation will be helpful to the researchers of this community. If you have any questions or comments, please do not hesitate to contact me.
Cheers,
Brad
===========
"Learning from Human-Generated Reward"
Robots and other computational agents are increasingly becoming part of our daily lives. They will need to be able to learn to perform new tasks, adapt to novel situations, and understand what is wanted by their human users, most of whom will not have programming skills. To achieve these ends, agents must learn from humans using methods of communication that are naturally accessible to everyone. This thesis presents and formalizes interactive shaping, one such teaching method, where agents learn from real-valued reward signals that are generated by a human trainer. In interactive shaping, a human trainer observes an agent behaving in a task environment and delivers feedback signals. These signals are mapped to numeric values, which are used by the agent to specify correct behavior. A solution to the problem of interactive shaping maps human reward to some objective such that maximizing that objective generally leads to the behavior that the trainer desires.
Interactive shaping addresses the aforementioned needs of real-world agents. This teaching method allows human users to quickly teach agents the specific behaviors that they desire. Further, humans can shape agents without needing programming skills or even detailed knowledge of how to perform the task themselves. In contrast, algorithms that learn autonomously from only a pre-programmed evaluative signal often learn slowly, which is unacceptable for some real-world tasks with real-world costs. These autonomous algorithms additionally have an inflexibly defined set of optimal behaviors, changeable only through additional programming. Through interactive shaping, human users can (1) specify and teach desired behavior and (2) share task knowledge when correct behavior is already indirectly specified by an objective function. Additionally, computational agents that can be taught interactively by humans provide a unique opportunity to study how humans teach in a highly controlled setting, in which the computer agent’s behavior is parametrized.
This thesis answers the following question. How and to what extent can agents harness the information contained in human-generated signals of reward to learn sequential decision-making tasks? The contributions of this thesis begin with an operational definition of the problem of interactive shaping. Next, I introduce the tamer framework, one solution to the problem of interactive shaping, and describe and analyze algorithmic implementations of the framework within multiple domains. This thesis also proposes and empirically examines algorithms for learning from both human reward and a pre-programmed reward function within an MDP, demonstrating two techniques that consistently outperform learning from either feedback signal alone. Subsequently, the thesis shifts its focus from the agent to the trainer, describing two psychological studies in which the trainer is manipulated by either changing their perceived role or by having the agent intentionally misbehave at specific times; we examine the effect of these manipulations on trainer behavior and the agent’s learned task performance. Lastly, I return to the problem of interactive shaping, for which we examine a space of mappings from human reward to objective functions, where mappings differ by how much the agent discounts reward it expects to receive in the future. Through this investigation, a deep relationship is identified between discounting, the level of positivity in human reward, and training success. Specific constraints of human reward are identified (i.e., the “positive circuits” problem), as are strategies for overcoming these constraints, pointing towards interactive shaping methods that are more effective than the already successful tamer framework.
Greetings,
I'm proud to announce (somewhat belatedly) my PhD dissertation, defended in August 2012 at the University of Texas at Austin. Entitled
"Learning from Human-Generated Reward"
and written with Peter Stone's supervision, the dissertation has been named runner-up for the IFAAMAS Victor Lesser Distinguished Dissertation Award, which will be given out at the AAMAS banquet tonight. The dissertation contains research that received the AAMAS 2010 Pragnesh Jay Modi Best Student Paper Award and also was a finalist for the Ro-Man 2012 CoTeSys Cognitive Robotics Best Paper Award. (Additionally, for those at AAMAS, Guangliang Li will present a paper Friday morning in Session E5 that builds upon the user interface side of this dissertation work, a paper written in collaboration with Shimon Whiteson, Hayley Hung, and myself.)
The dissertation focuses roughly on the question: If reward in a reinforcement learning framework is given by a live human trainer as he or she observes the agent's behavior—rather than from the usual pre-coded reward function—how should an agent use these feedback signals to best learn the behavior that the human intends to teach?
The dissertation formalizes this learning problem, which we call interactive shaping (in Ch. 2); contains a thorough review of past work on this topic, noting the interesting pattern that all such work used relatively myopic temporal discounting (Ch. 2); proposes the TAMER framework as an approach to the problem and describes the application of TAMER to robot learning (Ch. 3); through "TAMER+RL", uses human feedback to improve upon reinforcement learning when pre-coded reward is also received by the agent (Ch. 4); examines the effects on the trainer of various manipulations, including making the agent misbehave whenever the trainer reduces his or her frequency of feedback (Ch. 5); and an exploration of the effect on task performance of employing various temporal discounting rates, leading to the first non-myopic algorithm to learn from human reward (Ch. 6). For those of you who are already familiar with TAMER, I recommend Chapter 6, which I believe contributes significant insight to the problem of learning from human reward.
A video of my defense and accompanying slides can be found at my dissertation page:
http://web.media.mit.edu/~bradknox/Dissertation.html
The dissertation document itself can be found here:
http://www.cs.utexas.edu/~bradknox/papers/thesis-knox.pdf
Additionally, I'm slowly but surely preparing the TAMER codebase for release and hope to announce soon. If you're interested in receiving the code now, email me.
I sincerely hope that this dissertation will be helpful to the researchers of this community. If you have any questions or comments, please do not hesitate to contact me.
Cheers,
Brad
===========
"Learning from Human-Generated Reward"
Robots and other computational agents are increasingly becoming part of our daily lives. They will need to be able to learn to perform new tasks, adapt to novel situations, and understand what is wanted by their human users, most of whom will not have programming skills. To achieve these ends, agents must learn from humans using methods of communication that are naturally accessible to everyone. This thesis presents and formalizes interactive shaping, one such teaching method, where agents learn from real-valued reward signals that are generated by a human trainer. In interactive shaping, a human trainer observes an agent behaving in a task environment and delivers feedback signals. These signals are mapped to numeric values, which are used by the agent to specify correct behavior. A solution to the problem of interactive shaping maps human reward to some objective such that maximizing that objective generally leads to the behavior that the trainer desires.
Interactive shaping addresses the aforementioned needs of real-world agents. This teaching method allows human users to quickly teach agents the specific behaviors that they desire. Further, humans can shape agents without needing programming skills or even detailed knowledge of how to perform the task themselves. In contrast, algorithms that learn autonomously from only a pre-programmed evaluative signal often learn slowly, which is unacceptable for some real-world tasks with real-world costs. These autonomous algorithms additionally have an inflexibly defined set of optimal behaviors, changeable only through additional programming. Through interactive shaping, human users can (1) specify and teach desired behavior and (2) share task knowledge when correct behavior is already indirectly specified by an objective function. Additionally, computational agents that can be taught interactively by humans provide a unique opportunity to study how humans teach in a highly controlled setting, in which the computer agent’s behavior is parametrized.
This thesis answers the following question. How and to what extent can agents harness the information contained in human-generated signals of reward to learn sequential decision-making tasks? The contributions of this thesis begin with an operational definition of the problem of interactive shaping. Next, I introduce the tamer framework, one solution to the problem of interactive shaping, and describe and analyze algorithmic implementations of the framework within multiple domains. This thesis also proposes and empirically examines algorithms for learning from both human reward and a pre-programmed reward function within an MDP, demonstrating two techniques that consistently outperform learning from either feedback signal alone. Subsequently, the thesis shifts its focus from the agent to the trainer, describing two psychological studies in which the trainer is manipulated by either changing their perceived role or by having the agent intentionally misbehave at specific times; we examine the effect of these manipulations on trainer behavior and the agent’s learned task performance. Lastly, I return to the problem of interactive shaping, for which we examine a space of mappings from human reward to objective functions, where mappings differ by how much the agent discounts reward it expects to receive in the future. Through this investigation, a deep relationship is identified between discounting, the level of positivity in human reward, and training success. Specific constraints of human reward are identified (i.e., the “positive circuits” problem), as are strategies for overcoming these constraints, pointing towards interactive shaping methods that are more effective than the already successful tamer framework.
_________________________
W. Bradley Knox, PhD
Postdoctoral Associate
Personal Robots
MIT Media Laboratory
brad...@mit.edu
http://media.mit.edu/~bradknox
–––––––––––––––––––––––––
W. Bradley Knox, PhD
Postdoctoral Associate
Personal Robots
MIT Media Laboratory
brad...@mit.edu
http://media.mit.edu/~bradknox
–––––––––––––––––––––––––
Reply all
Reply to author
Forward
0 new messages