Teaching through Instruction

Learning tasks from natural language instructions

SCENARIO

The autonomous UGV robot receives a text describing a task from the human operator Danny and learns a sequence of operations, a script, from these instructions. Then, when asked to carry out the task in a particular situation, the robot creates a situation-specific plan based on the script it learned and then executes the plan.

Step 0 of 12
Jump to Step:

0 LEIA's Functioning Flow

A teammate teaches LEIA, a HARMONIC-based robot, in English, how to perform a novel procedure.

LEIA learns and demonstrates that it did learn by performing the procedure when asked to do so.

First, we will allow you to step through the process, and then we will show the demo without comment, in real time.

There are two phases in this process:

  • Learning Phase - LEIA receives and interprets instructions to learn a new script
  • Execution Phase - LEIA applies the learned script to perform the task

Ready to explore? Click NEXT or any step number above to begin!

Watch a video demo with explanation below.

Simulation of shipboard environment showing a UGV

Simulation of shipboard environment showing a UGV (Click to zoom)

1 Learning Phase

LEARNING PHASE

LEIA processes textual instructions to acquire a new script:

1 Receive textual input — Danny provides instructions in natural language
2 Interpret the input semantically — Convert into a Text Meaning Representation (TMR)
3 Recognize the learning need — Identify that the input implies a need to learn a script
4 Learn the script — Store the procedure for future execution
Learning Phase Processing

2 Human Input & Text Interpretation

DANNY: "Here is how you replace a shipboard power panel's fuse. First, verify that the system is turned off. Then open the panel and pull out the fuse. Read its amperage rating. Put it in the discard bin and go to the stores. Retrieve a new fuse with the same amperage rating. Return to the panel and insert the fuse into the holder. Then, close the panel. Turn the system back on and you're done."

What's happening:

The human teammate describes a procedure in English. LEIA receives the text and begins TEXT-INTERPRETATION to convert natural language into a semantic meaning representation.

Current Agenda:

@AGENT-FUNCTIONING-FLOW
  PERCEPTION-RECOGNITION
  TEXT-INTERPRETATION
    SYNTAX-INTERPRETATION        ← Current
    SYNMAP-INTERPRETATION
    BASIC-SEMANTICS-INTERPRETATION
    PROCEDURAL-SEMANTICS-INTERPRETATION
    EXTENDED-SEMANTICS-INTERPRETATION
    SITUATIONAL-SEMANTICS-INTERPRETATION
  REASONING-ABOUT-ACTION
    REMEMBER-FROM-XMR
    LEARN-FROM-XMR
      DETECT-START-LEARNING-TRIGGERS
      EXTRACT-INFORMATION-FROM-XMR
      DETECT-END-LEARNING-TRIGGERS
      WRAP-LEARNING-FROM-XMR
    MODIFY-AGENDA-FROM-XMR
TEXT-INTERPRETATION in progress

3 TMR Generation & Learning Recognition

DANNY: "Here is how you replace a shipboard power panel's fuse. First, verify that the system is turned off. Then open the panel and pull out the fuse. Read its amperage rating. Put it in the discard bin and go to the stores. Retrieve a new fuse with the same amperage rating. Return to the panel and insert the fuse into the holder. Then, close the panel. Turn the system back on and you're done."

What's happening:

TEXT-INTERPRETATION generates a Text Meaning Representation (TMR) that captures the full semantic content of Danny's instructions. LEIA recognizes that this is a SCRIPT-LEARNING-EVENT, indicating the human is teaching a new procedure.

Full TMR:

#REQUEST-ACTION.1
    $THEME  #SCRIPT-LEARNING-EVENT.1
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
#SCRIPT-LEARNING-EVENT.1
    $THEME  #REPLACE-PART.1
    $AGENT  ->LEIA.1
#REPLACE-PART.1
    $AGENT  #LEIA.1
    $THEME  #FUSE.1
    $TIME  *FIND-ANCHOR-TIME
#REQUEST-ACTION.2
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
    $THEME  #CHECK-POWER-STATE-OF-SHIP-CONTROL-SYSTEM.1
    $TIME  *FIND-ANCHOR-TIME(>)
#POWERED-ON.1
    $DOMAIN  #SHIP-CONTROL-SYSTEM.1
    $RANGE  False
#CHECK-POWER-STATE-OF-SHIP-CONTROL-SYSTEM.1
    $THEME  #POWERED-ON.1
    $TIME  *FIND-ANCHOR-TIME(>)
#REQUEST-ACTION.3
    $DISCOURSE-RELATION  #REQUEST-ACTION.4
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
    $THEME  #OPEN-DOOR.1
    $TIME  *FIND-ANCHOR-TIME(>)
#REQUEST-ACTION.4
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
    $THEME  #REMOVE-FUSE.1
    $TIME  *FIND-ANCHOR-TIME(>)
#OPEN-DOOR.1
    $THEME  #DOOR.1
    $TIME  *FIND-ANCHOR-TIME(>)
#REMOVE-FUSE.1
    $THEME  #FUSE.1
    $INVOLVES-OBJECT  #FUSE-HOLDER.1
    $TIME  *FIND-ANCHOR-TIME(>)
#REQUEST-ACTION.5
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
    $THEME  #READ-FUSE-LABEL.1
    $TIME  *FIND-ANCHOR-TIME(>)
#READ-FUSE-LABEL.1
    $THEME  #LABEL.1
    $TIME  *FIND-ANCHOR-TIME(>)
#REQUEST-ACTION.6
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
    $THEME  #DISPOSE-OF-FUSE.1
    $TIME  *FIND-ANCHOR-TIME(>)
    $DISCOURSE-RELATION  #REQUEST-ACTION.7
#DISPOSE-OF-FUSE.1
    $THEME  #FUSE.1
    $DESTINATION  #TRASH-CAN.1
    $TIME  *FIND-ANCHOR-TIME(>)
#REQUEST-ACTION.7
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
    $THEME  #AGENT-WAYPOINT-EVENT.1
    $TIME  *FIND-ANCHOR-TIME(>)
#AGENT-WAYPOINT-EVENT.1
    $DESTINATION  #PART-STORAGE-LOCATION.1
    $TIME  *FIND-ANCHOR-TIME(>)
#PART-STORAGE-LOCATION.1
    $CARDINALITY  ('>', 1)
#REQUEST-ACTION.8
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
    $THEME  #ACQUIRE-FUSE.1
    $TIME  *FIND-ANCHOR-TIME(>)
#ACQUIRE-FUSE.1
    $THEME  #FUSE.2
    $TIME  *FIND-ANCHOR-TIME(>)
#REQUEST-ACTION.9
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
    $THEME  #AGENT-WAYPOINT-EVENT.2
    $TIME  *FIND-ANCHOR-TIME(>)
    $DISCOURSE-RELATION  #REQUEST-ACTION.10
#AGENT-WAYPOINT-EVENT.2
    $DESTINATION  #POWER-PANEL.1
    $TIME  *FIND-ANCHOR-TIME(>)
#REQUEST-ACTION.10
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
    $THEME  #INSERT-FUSE.1
    $TIME  *FIND-ANCHOR-TIME(>)
#INSERT-FUSE.1
    $THEME  #FUSE.1
    $DESTINATION  #FUSE-HOLDER.1
    $TIME  *FIND-ANCHOR-TIME(>)
#REQUEST-ACTION.11
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
    $THEME  #CLOSE-DOOR.1
    $TIME  *FIND-ANCHOR-TIME(>)
#CLOSE-DOOR.1
    $THEME  #DOOR.1
    $TIME  *FIND-ANCHOR-TIME(>)
#REQUEST-ACTION.12
    $AGENT  ->HUMAN.1
    $BENEFICIARY  ->LEIA.1
    $THEME  #TURN-SHIP-CONTROL-SYSTEM-SWITCH-ON.1
    $TIME  *FIND-ANCHOR-TIME(>)
#TURN-SHIP-CONTROL-SYSTEM-SWITCH-ON.1
    $THEME  #SWITCH.1
    $INVOLVES-OBJECT  #SHIP-CONTROL-SYSTEM.1
    $TIME  *FIND-ANCHOR-TIME(>)
#END-LEARNING-EVENT.1
    $TIME  *FIND-ANCHOR-TIME

// Object definitions
#LABEL.1
    $HAS-TEXT  #AMPERAGE-RATING.1
#POWER-PANEL.1
    $HAS-OBJECT-AS-PART  #FUSE.1
    $HAS-OBJECT-AS-PART  #DOOR.1
#FUSE.1
    $HAS-OBJECT-AS-PART  #LABEL.1
#AMPERAGE-RATING.1
#SWITCH.1
#SHIP-CONTROL-SYSTEM.1
    $HAS-OBJECT-AS-PART  #SWITCH.1
#FUSE-HOLDER.1
    $CONTAINS-OBJECT  #FUSE.1
#DOOR.1
#FUSE.2
    $AMPERAGE-RATING  #AMPERAGE-RATING.2
    $AGE  [0.0001, '<>', 0.1]
#AMPERAGE-RATING.2
#TRASH-CAN.1
#LEIA.1
    $COREFER  ->LEIA.1
TMR Generated & Learning Need Recognized

4 Learn the Script

What's happening:

REASONING-ABOUT-ACTION interprets REQUEST-ACTION and triggers LEARN-FROM-XMR, a procedure that learns from meaning representations of input from any perceptual modality, not just text. LEIA extracts the sequence of actions and stores them as a reusable script.

Updated Agenda:

@AGENT-FUNCTIONING-FLOW
  PERCEPTION-RECOGNITION
  TEXT-INTERPRETATION
    ...
  REASONING-ABOUT-ACTION         ← Current
    REMEMBER-FROM-XMR
    LEARN-FROM-XMR               ← Triggered
      DETECT-START-LEARNING-TRIGGERS
      EXTRACT-INFORMATION-FROM-XMR
      DETECT-END-LEARNING-TRIGGERS
      WRAP-LEARNING-FROM-XMR
    MODIFY-AGENDA-FROM-XMR

A script has been learned. This is a concise rendition of the script that was just learned, presented to underscore LEIA's transparency.

LEIA's Thoughts - Summary of Learned Script:

Here's a summary of the script I learned:

REPLACE-FUSE
    OPEN-DOOR-1
    REMOVE-FUSE-1
    DISPOSE-OF-FUSE-1
    AGENT-WAYPOINT-EVENT-1
    ACQUIRE-FUSE-1
    AGENT-WAYPOINT-EVENT-2
    INSERT-FUSE-1
    CLOSE-DOOR-1
    TURN-SHIP-CONTROL-SYSTEM-SWITCH-ON-1

Details of the script learning process will be described in a planned publication.

Learning Phase Complete

5 Execution Phase

What's happening:

The Learning Phase is complete. LEIA now enters the Execution Phase where it will demonstrate that it learned the procedure by executing it when asked.

EXECUTION PHASE

LEIA applies the learned script to perform the task:

1 Receive textual input — Danny requests LEIA to perform the learned task
2 Interpret the input semantically — Convert into a TMR
3 Recognize the skill execution need — Identify that the input implies executing a known skill
4 Create and execute a plan — Instantiate a situation-specific plan from the script
Execution Phase Starting

6 Human Request & TMR Generation

DANNY: "Replace the panel's fuse."

What's happening:

LEIA's human teammate asks LEIA to demonstrate what it just learned. LEIA interprets the execution request semantically into a TMR, identifying the specific task and linking to episodic memory of the learned script.

Generated TMR:

FUSE.1
    COREFER         EPISODIC-MEMORY.1/FUSE.1
POWER-PANEL.1
    HAS-OBJECT-AS-PART  FUSE.1
    COREFER         EPISODIC-MEMORY.1/POWER-PANEL.1
REQUEST-ACTION-REPLACE-FUSE.1
    AGENT           EPISODIC-MEMORY.1/HUMAN.1
    BENEFICIARY     EPISODIC-MEMORY.1/LEIA.1
    THEME           FUSE.1
    TIME            *FIND-ANCHOR-TIME(>)
TMR Generated

7 Recognize Skill & Create Plan

What's happening:

LEIA recognizes that the TMR contains a REQUEST-ACTION that matches the REPLACE-FUSE script it learned earlier. This triggers the skill execution pathway. LEIA instantiates the learned script as a situation-specific plan.

REQUEST-ACTION Match:

REQUEST-ACTION-REPLACE-FUSE.1
    AGENT           EPISODIC-MEMORY.1/HUMAN.1
    BENEFICIARY     EPISODIC-MEMORY.1/LEIA.1
    THEME           FUSE.1
    TIME            *FIND-ANCHOR-TIME(>)

The plan is instantiated:

@REPLACE-FUSE
    OPEN-DOOR
    REMOVE-FUSE
    DISPOSE-OF-FUSE
    AGENT-WAYPOINT-EVENT
    ACQUIRE-FUSE
    AGENT-WAYPOINT-EVENT
    INSERT-FUSE
    CLOSE-DOOR
    TURN-SHIP-CONTROL-SYSTEM-SWITCH-ON
Plan Instantiated

8 Precondition Check

What's happening:

LEIA attempts to execute the first step of the plan but discovers a precondition failure. The system must be turned off before the panel door can be opened safely. LEIA's knowledge base includes safety constraints that prevent unsafe actions.

Plan Status:

@REPLACE-FUSE
    OPEN-DOOR              ← Blocked!
    REMOVE-FUSE
    DISPOSE-OF-FUSE
    AGENT-WAYPOINT-EVENT
    ACQUIRE-FUSE
    AGENT-WAYPOINT-EVENT
    INSERT-FUSE
    CLOSE-DOOR
    TURN-SHIP-CONTROL-SYSTEM-SWITCH-ON

The precondition requires the system to be turned off before opening the door.

Precondition Failed

9 Subgoal Generation

What's happening:

LEIA autonomously generates a subgoal to satisfy the failed precondition. Rather than failing or asking for help, LEIA reasons about what action will make the precondition true and creates a sub-plan to turn off the system first.

Updated Plan:

@REPLACE-FUSE
    OPEN-DOOR
    REMOVE-FUSE
    DISPOSE-OF-FUSE
    AGENT-WAYPOINT-EVENT
    ACQUIRE-FUSE
    AGENT-WAYPOINT-EVENT
    INSERT-FUSE
    CLOSE-DOOR
    TURN-SHIP-CONTROL-SYSTEM-SWITCH-ON
@TURN-SHIP-CONTROL-SYSTEM-SWITCH-OFF  ← Subgoal added

Processes:

#TURN-SHIP-CONTROL-SYSTEM-SWITCH-OFF.1  [Callback]
Subgoal Plan Created

10 Success - Executing Plan

What's happening:

After successfully turning off the system, LEIA continues with the main plan. Each step is executed in sequence, with completed steps shown in green. LEIA is currently inserting the new fuse.

Plan Progress:

@REPLACE-FUSE
    OPEN-DOOR              ✓
    REMOVE-FUSE            ✓
    DISPOSE-OF-FUSE        ✓
    AGENT-WAYPOINT-EVENT   ✓
    ACQUIRE-FUSE           ✓
    AGENT-WAYPOINT-EVENT   ✓
    INSERT-FUSE            ← Current
    CLOSE-DOOR
    TURN-SHIP-CONTROL-SYSTEM-SWITCH-ON

Processes:

#OPEN-DOOR.1  [Callback]
Executing Plan Steps

11 Plan Completion

What's happening:

LEIA has successfully completed all steps of the plan, including turning the system back on. This demonstrates that LEIA learned the fuse replacement procedure from Danny's natural language instructions and can execute it in context.

Final Plan Status:

@REPLACE-FUSE
    OPEN-DOOR                              ✓
    REMOVE-FUSE                            ✓
    DISPOSE-OF-FUSE                        ✓
    AGENT-WAYPOINT-EVENT                   ✓
    ACQUIRE-FUSE                           ✓
    AGENT-WAYPOINT-EVENT                   ✓
    INSERT-FUSE                            ✓
    CLOSE-DOOR                             ✓
    TURN-SHIP-CONTROL-SYSTEM-SWITCH-ON     ✓ Complete!

Processes:

#TURN-SHIP-CONTROL-SYSTEM-SWITCH-ON.1  [Callback]

LEIA has successfully demonstrated that it learned the fuse replacement procedure and can execute it in context!

Task Complete - Demo Finished

12 Watch Full Demo - Simulation

Complete Teaching through Instruction Demonstration

Watch the full demonstration showing HARMONIC's ability to learn a new procedure from natural language instruction and execute it when requested.

Complete HARMONIC Teaching through Instruction Demo

(Click video to enlarge and hear the audio explanation)

Download video

Demo on Teaching Through Instruction

Unmute for audio explanation

Download video

Demo on a Real Robot

This video has no audio

Download video