Robotic Grounding and LLMs: Advancements and Challenges
Casey Kennington<https://www.caseyreddkennington.com/>
Computer Science, Boise State
09-Nov
jeudi/thursday 10h30
https://uqam.zoom.us/j/83002459798
ABSTRACT: Large Language Models (LLMs) are primarily trained using large amounts of text, but there have also been noteworthy advancements in incorporating vision and other sensory information into LLMs. Does that mean LLMs are ready for embodied agents such as robots? While there have been important advancements, technical and theoretical challenges remain including use of closed language models like ChatGPT, model size requirements, data size requirements, speed requirements, representing the physical world, and updating the model with information about the world in real time. In this talk, I explain recent advance on incorporating LLMs into robot platforms, challenges, and opportunities for future work.
Casey Kennington is associate professor in the Department of Computer Science at Boise State University where he does research on spoken dialogue systems on embodied platforms. His long-term research goal is to understand what it means for humans to understand, represent, and produce language. His National Science Foundation CAREER award focuses on enriching small language models with multimodal information such as vision and emotion for interactive learning on robotic platforms. Kennington obtained his PhD in Linguistics from Bielefeld University, Germany.
Josue Torres-Foncesca, Catherine Henry, Casey Kennington. Symbol and Communicative Grounding through Object Permanence with a Mobile Robot<https://aclanthology.org/2022.sigdial-1.14/>. In Proceedings of SigDial, 2022.
Clayton Fields and Casey Kennington. Vision Language Transformers: A Survey<https://arxiv.org/abs/2307.03254>. arXiv, 2023.
Casey Kennington. Enriching Language Models with Visually-grounded Word Vectors and the Lancaster Sensorimotor Norms<https://aclanthology.org/2021.conll-1.11/>. In Proceedings of CoNLL, 2021
Casey Kennington. On the Computational Modeling of Meaning: Embodied Cognition Intertwined with Emotion<https://arxiv.org/abs/2307.04518>. arXiv, 2023.
14-Sep
Benjamin Bergen
UCSD
LLMs are Impressive But We Still Need Grounding
21-Sep
Dimitri C Mollo
Umea
Grounding in LLMs: Functional AI Ontologies
28-Sep
Dave Chalmers
NYU
Does Thinking Require Grounding?
05-Oct
Ellie Pavlick
Brown
Symbols and Grounding in LLMs
12-Oct
Paul Rosenbloom
USC
Rethinking the Physical Symbol Systems Hypothesis
19-Oct
Melanie Mitchell
Santa Fe Ins
Language and Grounding
26-Oct
Dor Abrahamson
Berkeley
Enactive Symbol Grounding in Mathematics Education
02-Nov
09-Nov
Eric Schulz
Casey Kennington
Tuebingen
Boise State
Machine Psychology
Robotic grounding and LLMs
16-Nov
Usef Faghihi
UQTR
« Algorithmes de Deep Learning flous causaux »
23-Nov
Anders Søgaard
Copenhagen
LLMs: Indication or Representation?
30-Nov
Christoph Durt
Freiburg IAS
LLMs, Patterns, and Understanding
07-Dec
Jake Hanson
ASU
Falsifying the Integrated Information Theory of Consciousness
14-Dec
Frédéric Alexandre
Bordeaux
« Apprentissage continu et contrôlé cognitif »
cog Intuitive Physical Reasoning and Mental Simulation
Todd Gureckis
Psychology, NYU
UQÀM ISC DIC CRIA
Séminaire en informatique cognitive/Cognitive Informatics Seminar
Thursday, 10:30 am
December 15, 2022
Zoom: https://uqam.zoom.us/j/88481835073
( videos of past seminars: https://youtu.be/XePaBMc_HFg )
Abstract: The ability to reason about the physics of our world (e.g., what arrangements of objects are stable, how things will fall or move under a force) is central to human intelligence. One influential hypothesis is that this capacity stems from the ability to perform “mental simulations” of physical events (in effect, playing a mental “movie” of the future evolution of a scene according to the laws of physics). In this talk, I’ll try to pin down several core commitments of the mental simulation approach that must be present for the general theory to be viable. I then will describe experiments we conducted recently trying to test these commitments. Along the way, we stumbled into several curious and novel errors and biases in human physical reasoning ability that we believe represent limits to the universality of contemporary simulation theories. If there is time, I will discuss a related project considering how efficient or optimal people are when they “experiment” in the physical world in order to learn the covert properties of objects such as mass or attractive/repulsive forces like magnetism.
Todd M. Gureckis, Professor of Psychology, New York University, studies how people actively explore their world in order to learn, including everyday reasoning capacities for the physical and social world. His research combines methods of computational modeling, developmental psychology, cognitive neuroscience, and online data collection. He is the founder and a lead developer of the psiTurk<https://psiturk.org/> package, a tool for facilitating online experiments used in hundreds of research labs. His work has been recognized by the NSF CAREER award, the Presidential Early Career Award (PECASE) from the Office of Science and Technology Policy at the White House, the James S. McDonnell Foundation Scholar award, and several paper and conferences awards with his students including the Marr Prize from the Cognitive Science Society, the Clifford T. Morgan Prize from the Psychonomic Society. He has variously served an Associate Editor for Cognitive Science, Topics in Cognitive Science, and Computational Brain and Behavior.
References
https://gureckislab.org/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…> :
https://gureckislab.org/papers/#/ref/ludwin2021limits<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…>
https://gureckislab.org/papers/#/ref/ludwinpeery2020broken<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…>
https://gureckislab.org/papers/#/ref/bramley2018intuitive<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…>
Hi there,
The Peyrache Lab and the Graduate Students' Association for Neuroscience is hosting a workshop on our software, Pynapple, which is a software for analyzing neurophysiological data.
"The Graduate Students' Association for Neuroscience is hosting an Introductory Workshop on Pynapple, a lightweight Python library for analyzing your neurophysiological data. The workshop runs from Oct. 25-27th, and coffee & lunch will be provided! Registration is now open and space is limited, so we urge you to complete our registration form before the deadline on October 23!
Link to registration form: https://forms.office.com/r/NZJHnXJ8G9"
Please spread the word in your circles.
Thank you so much!
Dhruv
Sent from Outlook for Android<https://aka.ms/AAb9ysg>
Rethinking the Physical Symbol Systems Hypothesis
Paul Rosenbloom<https://viterbi.usc.edu/directory/faculty/Rosenbloom/Paul>
Computer Science, USC
Thursday 10:30 am (EDT)
October 12
https://uqam.zoom.us/j/83002459798
ABSTRACT: It is now more than a half-century since the Physical Symbol Systems Hypothesis (PSSH) was first articulated as an empirical hypothesis. More recent evidence from work with neural networks and cognitive architectures has weakened it, but it has not yet been replaced in any satisfactory manner. Based on a rethinking of the nature of computational symbols – as atoms or placeholders – and thus also of the systems in which they participate, a hybrid approach is introduced that responds to these challenges while also helping to bridge the gap between symbolic and neural approaches, resulting in two new hypotheses, one – the Hybrid Symbol Systems Hypothesis (HSSH) – that is to replace the PSSH and the other focused more directly on cognitive architectures. This overall approach has been inspired by how hybrid symbol systems are central in the Common Model of Cognition and the Sigma cognitive architectures, both of which will be introduced – along with the general notion of a cognitive architecture – via “flashbacks” during the presentation.
Paul S. Rosenbloom is Professor Emeritus of Computer Science in the Viterbi School of Engineering at the University of Southern California (USC). His research has focused on cognitive architectures (models of the fixed structures and processes that together yield a mind), such as Soar and Sigma; the Common Model of Cognition (a partial consensus about the structure of a human-like mind); dichotomic maps (structuring the space of technologies underlying AI and cognitive science); “essential” definitions of key concepts in AI and cognitive science (such as intelligence, theories, symbols, and architectures); and the relational model of computing as a great scientific domain (akin to the physical, life and social sciences).
Rosenbloom, P. S. (2023). Rethinking the Physical Symbol Systems Hypothesis<https://www.dropbox.com/s/l9v7mjddktlokgo/Rosenbloom-PSSH-HSSH%20Final%20D.…>. In Proceedings of the 16th International Conference on Artificial General Intelligence (pp. 207-216). Cham, Switzerland: Springer.
Laird, J. E., Lebiere, C. & Rosenbloom, P. S. (2017). A Standard Model of the Mind: Toward a Common Computational Framework across Artificial Intelligence<https://www.dropbox.com/s/z50a70vl8sn3all/LLR-SMM-AI%20Magazine-Published-P…>, Cognitive Science, Neuroscience, and Robotics. AI Magazine, 38, 13-26.
Rosenbloom, P. S., Demski, A. & Ustun, V. (2016). The Sigma cognitive architecture and system: Towards functionally elegant grand unification<https://www.dropbox.com/s/hwv6eok7uhcps91/jagi-2016-0001.pdf?dl=0>. Journal of Artificial General Intelligence, 7, 1-103.
Rosenbloom, P. S., Demski, A. & Ustun, V. (2016). Rethinking Sigma’s graphical architecture: An extension to neural networks<https://www.dropbox.com/s/3q0mhigs9gv7mid/RSGA%20AGI%202016%20Final%20D.pdf…>. Proceedings of the 9th Conference on Artificial General Intelligence (pp. 84-94).
Upcoming Seminars:
14-Sep
Benjamin Bergen
UCSD
LLMs are Impressive But We Still Need Grounding
21-Sep
Dimitri C Mollo
Umea
Grounding in LLMs: Functional AI Ontologies
28-Sep
Dave Chalmers
NYU
Does Thinking Require Grounding?
05-Oct
Ellie Pavlick
Brown
Symbols and Grounding in LLMs
12-Oct
Paul Rosenbloom
USC
Rethinking the Physical Symbol Systems Hypothesis
19-Oct
Melanie Mitchell
Santa Fe Ins
Language and Grounding
26-Oct
Dor Abrahamson
Berkeley
Enactive Symbol Grounding in Mathematics Education
02-Nov
09-Nov
Eric Schulz
Casey Kennington
Tuebingen
Boise State
Machine Psychology
Robotic grounding and LLMs
16-Nov
Usef Faghihi
UQTR
« Algorithmes de Deep Learning flous causaux »
23-Nov
Anders Søgaard
Copenhagen
LLMs: Indication or Representation?
30-Nov
Christoph Durt
Freiburg IAS
LLMs, Patterns, and Understanding
07-Dec
Jake Hanson
ASU
Falsifying the Integrated Information Theory of Consciousness
14-Dec
Frédéric Alexandre
Bordeaux
« Apprentissage continu et contrôlé cognitif »
https://uqam.zoom.us/j/83002459798
Active inference and artificial curiosity
Karl J. Friston
Director, Wellcome Centre for Human Neuroimaging
Institute of Neurology, UCL London
10:30 am
Thursday, December 8
Zoom: https://uqam.zoom.us/j/88481835073
Cognitive Informatics Seminar
Séminaire en informatique cognitive
UQÀM ISC DIC CRIA
Abstract: This talk offers a formal account of insight and learning in terms of active (Bayesian) inference. It deals with the dual problem of inferring states of the world and learning its statistical structure. In contrast to current trends in machine learning (e.g., deep learning), we focus on how agents learn from a small number of ambiguous outcomes to form insight. I will use simulations of abstract rule-learning and approximate Bayesian inference to show that minimising (expected) free energy leads to active sampling of novel contingencies. This epistemic, curiosity-directed behaviour closes `explanatory gaps' in knowledge about the causal structure of the world, thereby reducing ignorance, in addition to resolving uncertainty about states of the known world. We then move from inference to model selection or structure learning to show how abductive processes emerge when agents test plausible hypotheses about symmetries in their generative models of the world. The ensuing Bayesian model reduction evokes mechanisms associated with sleep and has all the hallmarks of aha moments.
[A person with his hand on his chin Description automatically generated with medium confidence]
Karl Friston, theoretical neuroscientist and authority on brain imaging, invented statistical parametric mapping (SPM), voxel-based morphometry (VBM) and dynamic causal modelling (DCM). Mathematical contributions include variational Laplacian procedures and generalized filtering for hierarchical Bayesian model inversion. Friston currently works on models of functional integration in the human brain and the principles that underlie neuronal interactions. His main contribution to theoretical neurobiology is a free-energy principle for action and perception (active inference).
A repository of active inference papers: GitHub - BerenMillidge/FEP_Active_Inference_Papers: A repository for major/influential FEP and active inference papers.<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co…> Theoretical lecture on the physics behind active inference: I am therefore I think by Karl Friston - YouTube<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtu…>
Intuitive Physical Reasoning and Mental Simulation
Todd Gureckis
Psychology, NYU
UQÀM ISC DIC CRIA
Séminaire en informatique cognitive/Cognitive Informatics Seminar
Thursday, 10:30 am
December 15, 2022
Zoom: https://uqam.zoom.us/j/88481835073
( videos of past seminars: https://youtu.be/XePaBMc_HFg )
Abstract: The ability to reason about the physics of our world (e.g., what arrangements of objects are stable, how things will fall or move under a force) is central to human intelligence. One influential hypothesis is that this capacity stems from the ability to perform “mental simulations” of physical events (in effect, playing a mental “movie” of the future evolution of a scene according to the laws of physics). In this talk, I’ll try to pin down several core commitments of the mental simulation approach that must be present for the general theory to be viable. I then will describe experiments we conducted recently trying to test these commitments. Along the way, we stumbled into several curious and novel errors and biases in human physical reasoning ability that we believe represent limits to the universality of contemporary simulation theories. If there is time, I will discuss a related project considering how efficient or optimal people are when they “experiment” in the physical world in order to learn the covert properties of objects such as mass or attractive/repulsive forces like magnetism.
Todd M. Gureckis, Professor of Psychology, New York University, studies how people actively explore their world in order to learn, including everyday reasoning capacities for the physical and social world. His research combines methods of computational modeling, developmental psychology, cognitive neuroscience, and online data collection. He is the founder and a lead developer of the psiTurk<https://psiturk.org/> package, a tool for facilitating online experiments used in hundreds of research labs. His work has been recognized by the NSF CAREER award, the Presidential Early Career Award (PECASE) from the Office of Science and Technology Policy at the White House, the James S. McDonnell Foundation Scholar award, and several paper and conferences awards with his students including the Marr Prize from the Cognitive Science Society, the Clifford T. Morgan Prize from the Psychonomic Society. He has variously served an Associate Editor for Cognitive Science, Topics in Cognitive Science, and Computational Brain and Behavior.
References
https://gureckislab.org/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…> :
https://gureckislab.org/papers/#/ref/ludwin2021limits<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…>
https://gureckislab.org/papers/#/ref/ludwinpeery2020broken<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…>
https://gureckislab.org/papers/#/ref/bramley2018intuitive<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…>
Intuitive Physical Reasoning and Mental Simulation
Todd Gureckis
Psychology, NYU
UQÀM ISC DIC CRIA
Séminaire en informatique cognitive/Cognitive Informatics Seminar
Thursday, 10:30 am
December 15, 2022
Zoom: https://uqam.zoom.us/j/88481835073
Abstract: The ability to reason about the physics of our world (e.g., what arrangements of objects are stable, how things will fall or move under a force) is central to human intelligence. One influential hypothesis is that this capacity stems from the ability to perform “mental simulations” of physical events (in effect, playing a mental “movie” of the future evolution of a scene according to the laws of physics). In this talk, I’ll try to pin down several core commitments of the mental simulation approach that must be present for the general theory to be viable. I then will describe experiments we conducted recently trying to test these commitments. Along the way, we stumbled into several curious and novel errors and biases in human physical reasoning ability that we believe represent limits to the universality of contemporary simulation theories. If there is time, I will discuss a related project considering how efficient or optimal people are when they “experiment” in the physical world in order to learn the covert properties of objects such as mass or attractive/repulsive forces like magnetism.
Todd M. Gureckis, Professor of Psychology, New York University, studies how people actively explore their world in order to learn, including everyday reasoning capacities for the physical and social world. His research combines methods of computational modeling, developmental psychology, cognitive neuroscience, and online data collection. He is the founder and a lead developer of the psiTurk<https://psiturk.org/> package, a tool for facilitating online experiments used in hundreds of research labs. His work has been recognized by the NSF CAREER award, the Presidential Early Career Award (PECASE) from the Office of Science and Technology Policy at the White House, the James S. McDonnell Foundation Scholar award, and several paper and conferences awards with his students including the Marr Prize from the Cognitive Science Society, the Clifford T. Morgan Prize from the Psychonomic Society. He has variously served an Associate Editor for Cognitive Science, Topics in Cognitive Science, and Computational Brain and Behavior.
References
https://gureckislab.org/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…> :
https://gureckislab.org/papers/#/ref/ludwin2021limits<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…>
https://gureckislab.org/papers/#/ref/ludwinpeery2020broken<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…>
https://gureckislab.org/papers/#/ref/bramley2018intuitive<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgureckisl…>
Atlas of Forecasts: Modeling and Mapping Desirable Futures
Katy Börner
Victor H. Yngve Distinguished Professor of Engineering and Information Science
Luddy School of Informatics, Computing, and Engineering, Indiana University, USA
UQÀM ISC DIC CRIA
Séminaire en informatique cognitive/Cognitive Informatics Seminar
Thursday, 10:30 am
December 1, 2022
Zoom: https://uqam.zoom.us/j/88481835073
Abstract: Envisioning and implementing desirable futures requires a deep understanding of developments in science and technology as well as the ability to both simulate and communicate the likely impact of alternative actions. At a time when our relationship to a vulnerable planet Earth is especially important, such a profound awareness of complex, interlinked systems is needed more than ever. Atlas of Forecasts uses advanced data visualizations to introduce different types of computational models and demonstrates how model results can be used to inform effective decision-making. The models aim to capture the structure and dynamics of developments in education and the job market, progress in science and technology, and the impact of government policies—all from the micro to the macro levels. Model results can help us decide which human skills are needed in an artificial intelligence–empowered economy; which courses and degrees are most effective in upskilling and reskilling the current and future workforce; what progress in science and technology is likely to happen; and how policymakers can future-proof regions or nations.
Katy Börner’s research focuses on the development of data analysis and visualization techniques for information access, understanding, and management. She is particularly interested in the formalization, measurement, and systematic improvement of people’s data visualization literacy; the study of the structure and evolution of scientific disciplines; the construction and usage of a Human Reference Atlas; and the development of cyberinfrastructures for large-scale scientific collaboration and computation.
References
Börner, Katy. 2021. Atlas of Forecasts: Modeling and Mapping Desirable Futures<https://www.amazon.com/Atlas-Forecasts-Modeling-Mapping-Desirable/dp/026204…>. Cambridge, MA: The MIT Press.
Börner, Katy, Andreas Bueckle, and Michael Ginda. 2019. Data visualization literacy: Definitions, conceptual frameworks, exercises, and assessments.<https://www.pnas.org/content/116/6/1857> PNAS, 116 (6) 1857-1864.
Börner, Katy. 2015. Atlas of Knowledge: Anyone Can Map<http://scimaps.org/atlas2>. Cambridge, MA: The MIT Press.
Börner, Katy. 2010. Atlas of Science: Visualizing What We Know<http://scimaps.org/atlas/>. Cambridge, MA: The MIT Press.
The observer’s grounding problem in human-robot interaction
Tom Ziemke
Computer and Information Department, Linköping University, Sweden
UQÀM ISC DIC CRIA
Cognitive Informatics Seminar /Séminaire en informatique cognitive
Thursday, November 17
10:30 am
ZOOM: https://uqam.zoom.us/j/88481835073
Abstract: People commonly attribute intentional mental states, such as beliefs and goals, to robots (Thellman et al., 2022; Ziemke, 2020). In a recent paper we formulated the perceptual belief attribution problem (Thellman & Ziemke, 2021): How can people interacting with robots understand what they know about the shared physical environment without knowing much about those robots’ sensors, perception, memory, etc.? In this talk I’ll focus on the observer’s grounding problem, which is the other side of the same coin, i.e., the fact that in interaction with a robot people tend to make anthropomorphic, folk-psychological attributions, based on their own grounding rather than the robot’s
[Tom ZIEMKE | Professor | PhD | Linköping University, Linköping | LiU | Department of Computer and Information Science (IDA)]Bio: Tom Ziemke is Professor of Cognitive Systems at Linkoping University, Sweden. His main research interests are in situated/embodied cognition and social interaction, with a current focus on people’s interaction with different types of autonomous technologies, ranging from social robots to automated vehicles. A long-standing research interest is the relation between cognition and computation – and the resulting (mis-) conceptions of AI among both researchers and the general public
References:
Understanding robots https://www.science.org/doi/10.1126/scirobotics.abe2987
Explainability in Social Robotics https://doi.org/10.1145/3461781
Mental State Attribution to Robots https://doi.org/10.1145/3526112
AI/robotics and active visual and tactile perception
Lorenzo Natale
Institute of Technology, Genoa
10:30 am
Thursday, November 10
Zoom: https://uqam.zoom.us/j/88481835073
Cognitive Informatics Seminar
Séminaire en informatique cognitive
UQÀM ISC DIC CRIA
Abstract: Modern AI algorithms provide exceptional performance but require long training time and large datasets that are expensive to annotate. On the other hand, robots can actively interact with the environment and humans using their sensory system to learn on-line how to perceive and interact with objects. To extract structured information, however, the robot needs to be endowed with appropriate sensors, fast learning algorithms, and exploratory behavior that guide the interaction with the world.
In this talk I will introduce the sensory system we developed for the iCub humanoid robot, and in particular the tactile sensing technology. I will then review work in which we studied how to use visual and tactile feedback to explore unknown objects and to control the interaction between the hand and the objects for shape modelling, object discrimination and tracking. Finally, I will present recent work in which we developed fast learning algorithms for object segmentation that leverage on the interaction with a teacher and active learning for adaptation to new contexts.
Lorenzo Natale, Senior Researcher at the Italian Institute of Technology and coordinator of the Center for Robotics and Intelligent Systems, was one of the main contributors to the design and development of the iCub humanoid robot. His research interests span artificial vision, tactile perception and software architectures for robotics.
References:
Ceola, F., Maiettini, E., Pasquale, G., Meanti, G., Rosasco, L., and Natale, L., Learn Fast, Segment Well: Fast Object Segmentation Learning on the iCub Robot, IEEE Transactions on robotics, 2022.
Maiettini, E., Tikhanoff, V., and Natale, L., Weakly-Supervised Object Detection Learning through Human-Robot Interaction, in Proc. International Conference on Humanoid Robotics, Munich, Germany, 2021
Vezzani, G., Pattacini, U., Battistelli, G., Chisci, L., and Natale, L., Memory Unscented Particle Filter for 6-DOF Tactile Localization, in IEEE Transactions on Robotics, vol. 33, no. 5, pp. 1139-1155, 2017