Multi-Robot Search & Retrieve

Heterogeneous robot team coordination with UGV and drone

SCENARIO

In an apartment environment, a heterogeneous multi-robot team consisting of a UGV and a drone assists a human, Danny, in locating a lost set of keys.

Step 0 of 10
Jump to Step:

0 Understanding HARMONIC's Distributed Planning Process

Welcome to the HARMONIC Multi-Robot Demo!

This demonstration shows how HARMONIC coordinates multiple robots through distributed planning, where high-level strategic plans are divided into tactical commands for collaborative execution.

What You'll See:

  • Collaborative Activity Scripts: How team leaders and subordinates coordinate through meta-scripts
  • Distributed Plan Execution: How plans are divided between strategic and tactical layers
  • Multi-Robot Coordination: How UGV and drone work together in search missions
  • Behavior Tree Integration: How tactical layers execute collaborative plans
  • Real-time Communication: How robots coordinate and report findings

Ready to explore? Click "Next" or any step number above to begin!

1 Task Initiation: Danny's Request (M1)

DANNY: "I think I left my keys at home. Can you look around for them?"

What's happening: Danny initiates the search task by sending a message to the robot team. This triggers the team leader (UGV-U) to place a COLLABORATIVE-ACTIVITY on its agenda.

HARMONIC's response: The UGV-U recognizes this as a search task and begins the collaborative planning process:

@COLLABORATIVE-ACTIVITY (leader)
[INIT]
  *identify-team-members
[SELECT-PLAN]
  // Select a plan from available options.
  RUN *identify-candidate-plans
  RUN *select-plan
  // Selected: SEARCH-FOR-LOST-OBJECT

2 Information Gathering: Object Description (M2-M3)

UGV-U: "What do they look like?"
DANNY: "They are on a red keychain with a small flashlight."

What's happening: The UGV-U verifies preconditions for the SEARCH-FOR-LOST-OBJECT plan by gathering information about the object's appearance.

[PRECONDITIONS]
  RUN NEW @REQUEST-OBJECT-TYPE ✓
  RUN NEW @REQUEST-OBJECT-FEATURES ✓
  // Features: red keychain, small flashlight
  RUN NEW @REQUEST-LAST-SEEN-AT
  RUN NEW @REQUEST-LOCATION-CONSTRAINED

3 Location Information: Last Known Location (M4-M5)

UGV-U: "Do you recall where you last had them?"
DANNY: "I used them last night to open the front door, but they could be anywhere."

What's happening: The UGV-U gathers location information to prioritize search areas. The front door area becomes a priority location for the search sequence.

@SEARCH-FOR-LOST-OBJECT
  [SEARCH-ZONES]
    // Priority: FRONT DOOR (last used location)
    FOR #ZONE-1 IN #LOCATION-1.SEARCHABLE-ZONE
      RUN ASYNC AWAIT *search
      INTERRUPT WHEN #OBJECT-1.LOCATION KNOWN
      RUN *consider-reporting

4 Plan Distribution: Search Initiation (M6)

UGV-U: "Let's search the apartment."

What's happening: With preconditions met, the UGV-U moves to SUGGEST-PLAN and shares the domain plan with the drone through natural language dialog.

@COLLABORATIVE-ACTIVITY (subordinate)
[INIT]
  *identify-team-members
[WAIT-FOR-PLAN]
  AWAIT $.HAS-COLLABORATIVE-PLAN ISA @EVENT ✓
[RUN-PLAN]
  RUN NEW @SEARCH-FOR-LOST-OBJECT

5 Search Execution: Coordinated Exploration

What's happening: Both robots begin exploring their assigned areas using a waypoint strategy controlled by their individual tactical modules.

Strategic vs Tactical: The strategic (cognitive) module maintains awareness of area existence without directly guiding robot navigation, allowing for efficient local path planning while preserving high-level planning.

Both robots actively searching...

6 Communication and Coordination (M7)

UGV-U: "They aren't in the doorway."

What's happening: Throughout the search, robots report their findings to each other. When a robot fails to locate the keys in a searched area, it communicates this to its partner.

Coordination protocol: This continuous communication ensures efficient coverage and prevents redundant searching.

7 Object Detection: Keys Found

UGV-U: "I found them north of the couch."

What's happening: The UGV-U's Vision Meaning Representations (VMRs) widget displays object detection results that the strategic layer processes from sensing frames communicated by the tactical layer.

Object Located!

8 Task Completion: Mission Report

UGV-U: "I found them behind the couch."

What's happening: When the keys are found, the UGV-U reports the successful completion to Danny. Notice the difference in location description: "north of the couch" (M8) vs "behind the couch" (M9).

Context-appropriate communication: This demonstrates the cognitive agent's ability to generate context-appropriate language for different communication partners.

9 Mission Debrief: Team Coordination Summary

Mission Summary: The multi-robot search and retrieve mission demonstrates HARMONIC's distributed planning capabilities:

  • Collaborative Planning: Leader-subordinate coordination through meta-scripts
  • Distributed Execution: Strategic plans divided into tactical commands
  • Real-time Coordination: Continuous communication and status updates
  • Context-aware Communication: Different language styles for different audiences
  • Efficient Search: Coordinated coverage without redundancy
MISSION COMPLETE!

10 Watch Full Demo - Multi-Robot

Complete HARMONIC Multi-Robot Demonstration

Watch the full multi-robot demonstration showing HARMONIC's distributed planning and collaborative execution process with UGV and drone coordination.

Complete HARMONIC Multi-Robot Demo

(Click video to enlarge)