Embodied Knowledge representation & Reasoning
Prior multimodal knowledge requires representation in a structured form, so that generalisation and reasoning is accommodated. The need for structured knowledge in intelligent systems has led to the development of knowledge bases, ontologies and semantic graphs. While Neuroscience findings point to a multisensory, multimodal and distributed semantic memory in the human brain, these computational resources remain static storage spaces of mostly -verbal concepts- with ad hoc defined associations/relations. From a computational perspective, generalisation and common sense reasoning in intelligent systems still stumbles upon the gap between high-level, symbolic representations and low-level sensorymotor experiences.
Our ultimate objective in this research line is the development of a computational semantic memory structured in a way that it will enable generalisation and will serve multimodality. A basic step towards this direction is PRAXICON, a dynamic, recursive and referential semantic network with biological basis that aims to address the representation and integration challenges of embodied and enactive cognition.
Concepts in the PRAXICON may have a concrete (physical) or abstract reference pertaining to entities, movements and features with no domain constraints. They have multiple representations (symbolic, perceptual, motoric). All of them are important, but some of them are more important than others for generalisation and reasoning. Associations between concepts in the PRAXICON are simple or recursive in nature; they pertain to a finite set of pragmatic relation types that provide constraints on the production of new associations, without restricting such production though. The PRAXICON draws a clear line between language and semantic memory; it considers language an additional modality that contributes (significantly and uniquely) to the acquisition and use of generalised knowledge through dynamic interaction with perception and action, rather than an efficient representation means of such knowledge. We argue that the PRAXICON is necessary for intelligent systems, so that they move beyond one shot learning to large scale generalisation and common sense reasoning. In Human-Computer/Robot interaction in particular, such skill (prior knowledge genaralisation) is key for the machine to deal with the unexpected and reach towards creativity.
PRAXICON incorporates our research on goal-directed action, and in particular the principles and structures defined in the Minimalist Grammar of Action. A preliminary version of the PRAXICON is available open to the public. It was developed with the POETICON and POETICON++ projects, i.e., within human-robot interaction applications (following verbal instructions from humans for performing everyday tasks, and verbalization of what a human does in a visual scene). The embodied concepts and relations in the PLT Embodied Concepts Dataset (doi: 10.5281/zenodo.11624849) follow the PRAXICON structures; when their translation into English and French will conclude, a new version of PRAXICON populated with this data will be released (expected: December 2024).