RALLI, Robotic Action-Language
Learning through Interaction
funded by WWTF.

Future social robots will need the ability to acquire new tasks and behaviors on the job both through observation and through natural language instruction, for robot designers cannot build in all environmental and task contingencies in typical application domains such as health care settings or people’s homes. In this project, we tackle the critical subproblem of learning new actions and their corresponding words by the artificial system observing how those actions are performed and expressed by humans. As a result, robots will not only understand action concepts, but will also be able to learn and execute actions, and talk about them. We propose to develop psychologically plausible observation and experimentation-based algorithms for action verb learning that are deeply integrated with natural language understanding and generation. The algorithms will allow the robot to observe and interpret human actions, and distill from them constitutive aspects such as the motion trajectories and involved objects. At the same time, the robot will also be able to parse a concomitant utterance describing the action and thus relate syntactic elements of the verbal description to semantic elements in the action representation. By aligning syntactic roles and semantic types, the robot then can build an association between verb phrase structure and action argument types. The outcome of the project will thus contribute an important, currently missing part for future social robots.

---

Project

Demos and software release

Publications

Empirical data

Consortium

Media