Definitions of key terms and concepts in our research
Semantic representations specifying the content of actions to be executed, generated by the decision-making service before being rendered into executable commands.
The strategic and tactical layers are connected through a bidirectional interface that enables efficient inter-layer communication and data transfer. High-level action commands sent by the OntoAgent through the Interface APIs are unpacked and used to update corresponding variables in the blackboard.
Physical and communicative outputs generated by the system. Represents the system's ability to interact with and affect the environment.
Specific rules and algorithms for controlling physical outputs. Implements safety constraints and control policies for physical actions.
Focuses cognitive resources on relevant information for strategic decision-making and directs sensory focus for immediate task relevance. Manages information filtering and prioritization at both cognitive and tactical levels.
The process of determining whether an agent has sufficient understanding of a situation to proceed with action despite potential incomplete information.
Discourse-level action-response patterns governing turn-taking behaviors in dialog (e.g., request-compliance, question-answer, proposal-acceptance sequences).
Hierarchical structures in the tactical layer that execute operations with priority ordering, enabling reactive responses while maintaining planned behavior.
The communication channel between strategic and tactical layers enabling data transfer and command execution across the dual-control architecture.
A meta-script that helps agents operating in teams organize themselves to accomplish shared goals, with different versions for team leaders and subordinates.
Shared understanding between team members about goals, plans, and situational awareness, essential for effective collaboration.
The intended function or purpose of an utterance (broader than speech acts as it includes non-linguistic communication), such as requests, assertions, or questions.
Semantic relationships between propositions in dialog turns that may be explicit or inferred, important for assessing actionability.
Long-term storage of remembered instances of world objects, events, and past processing experiences, enabling agents to leverage past experiences for future decision-making.
The system's ability to provide transparent, human-understandable explanations of its reasoning, decisions, and actions through traces of cognitive processing.
Intermediate semantic specifications for language generation, encoding communicative content grounded in ontological concepts prior to surface realization through the natural language generator.
The process of connecting symbolic representations to physical world entities, perceptual data, or episodic memory instances (indicated by # indices in representations).
A dual-control cognitive-robotic architecture that integrates strategic (cognitive) level decision-making with tactical (robotic) level control, enabling robots to function as trusted teammates in human-robot teams through transparent reasoning and natural language communication.
The cognitive architecture incorporated in the strategic layer (aka OntoAgent). Neurosymbolic, multimodal cognitive-robotic systems implemented in the HARMONIC architecture that can interpret experiences, reason, and learn using ontologically-grounded knowledge. This is used interchageably with OntoAgent.
Self-monitoring capabilities enabling introspection of internal states, team member modeling (mindreading), and dynamic strategy adjustment based on situational assessment.
The metacognitive capability of modeling teammates' mental states, beliefs, capabilities, and intentions to enable effective collaboration.
The cognitive architecture incorporated in the strategic layer (aka LEIA), responsible for semantic interpretation, attention management, goal-setting, sophisticated planning, and addressing unexpected challenges in interpretable ways. This is used interchageably with LEIA.
A knowledge base API providing unified format for representing and accessing knowledge across the system, supporting inheritance, flexible organization into "spaces," and efficient querying.
A hierarchical knowledge repository containing formalized representations of entities (concepts), relationships, properties, and procedural schemas (scripts) that serve as the semantic foundation for agent reasoning.
The process of converting multimodal sensory inputs (speech, vision, haptic) into ontologically-grounded meaning representations for unified reasoning.
The tactical layer provides the strategic layer with preprocessed multimodal (speech, vision, etc.) perception data and relays robot state information, employing a suite of perception models within the perception services.
Various sensory data streams feeding into the system. Provides the raw information needed for environmental understanding and decision-making.
Handles raw sensory input and provides real-time environmental awareness. Processes sensor data for immediate action and safety monitoring. Across the two systems, we use three different robots: a UGV and a Drone in simulation environments, and a tabletop serial manipulator. Each robot operates its own instance of OntoAgent at the strategic level and customized low-level planners at the tactical level.
Low-level processing for environmental sensing and map creation. Handles SLAM, object detection, and spatial understanding.
Instances of scripts with parameter values set for specific situations, executed through the goal and plan agenda.
Requirements that must be satisfied before a plan can be executed, checked and resolved during plan instantiation.
True resolution of referring expressions to specific instances in episodic memory, distinguished from textual coreference resolution.
Complex events or procedural knowledge recorded as sequences of events with coreferenced participants and props, representing how typical actions unfold.
Tactical plans for immediate future actions, often derived from strategic goals. Bridges the gap between high-level planning and immediate execution.
Generates high-level plans and makes decisions based on long-term goals and knowledge. Handles complex reasoning about goals, constraints, and trade-offs.
Long-term storage of generic knowledge about types of events and objects, distinct from specific instances stored in episodic memory.
Working memory containing currently active concept instances and representations of entities and events that are part of the current task context.
The cognitive component responsible for high-level decision-making, planning, perception interpretation, attention management, and goal selection. Implements System 2 (slow, deliberative reasoning) processing.
The robotic control component responsible for low-level robot control, execution of motor actions, reflexive attention, sensor processing, and reactive behaviors. Implements System 1 (fast, reflexive) processing.
In the action translation pipeline of the tactical layer, specialized robotic planners, controllers, algorithms, and action sequence models are employed as part of the Reasoning and Decision Making modules and Effector Services to translate abstract action commands from OntoAgent into precise, executable robot control operations.
The robot designated to manage interactions with human operators, select domain scripts, resolve preconditions, and coordinate with subordinate robots.
Ontologically-grounded semantic structures encoding interpreted linguistic input, preserving speaker intent, propositional content, and pragmatic markers.
Ontologically-grounded interpretations of visual perceptual data, unifying multimodal sensory inputs into symbolic knowledge structures similar in form to TMRs.