Language-Guided Object Search in Agricultural Environments

Advaith Balaji, Saket Pradhan, Dmitry Berenson
Autonomous Robotic Manipulation Lab1, University of Michigan
Paper Video
LOSAE in Action


Figure 1: LOSAE in Action. Our method allows the robot to find an unseen object using previously seen objects as instruments of reasoning. Here, the robot can understand that a tool like a drill is most likely located near similar tools like a screwdriver, chisel or shovel.

Abstract

Creating robots that can effectively assist in agricultural environments has the potential to significantly reduce the mental and physical workload experienced by farm workers. We present LOSAE (Language-Guided Object Search in Agricultural Environments), a novel method that enables robots to locate unseen target objects in loosely semantically organized farm environments. Unlike traditional indoor spaces with clear semantic hierarchies, agricultural settings lack well-defined room-level organization. LOSAE overcomes this challenge by leveraging object-to-object semantic relationships inferred by Large Language Models (LLMs) to guide the robot's search strategy.

Our method first explores the environment to build a memory of seen objects and their locations. When queried about an unseen target object, LOSAE uses an LLM to reason about semantic relationships between the target and previously seen objects, generating a probability distribution over potential locations. The robot then plans an efficient path that balances visiting high-probability locations while minimizing travel distance.

We deployed LOSAE on a Boston Dynamics Spot robot in a real farm environment, achieving an 80% success rate and a success weighted by path length (SPL) of 0.67. Our offline evaluations demonstrated 84% path efficiency, significantly outperforming baseline methods in both accuracy and efficiency.


System Diagram

LOSAE System Diagram


Figure 2: The robot is tasked with finding a target object xt based on a user query. Using an LLM for semantic reasoning, it generates a probability distribution P based on object-to-object relationships between a seen object xs in Xs and the target xt. This distribution helps compute waypoint scores s(wi) for each waypoint wi. The robot plans a path optimizing a cost function C, balancing high-scoring waypoints with short path lengths (see paper for details). It then navigates to a waypoint, inspects surrounding objects, and grasps the target if found; otherwise, it proceeds to the next waypoint and continues the search.

Robot Demo

Results

Comparison of LOSAE against baseline methods across multiple metrics. See paper for detailed discussion.

Method Success Rate ↑ SPL ↑
LOSAE (Ours) 0.80 0.67
Room Search 0.73 0.51
Method Avg. Path Efficiency (Std. Dev) ↑
LOSAE (Ours) 0.84 (0.27)
Room Search 0.72 (0.33)
Hottest Object 0.21 (0.32)
Hottest Waypoint 0.19 (0.41)

↑ indicates higher is better. SPL: Success weighted by Path Length

BibTeX

@article{balaji2025losae,
  author    = {Balaji, Advaith and Pradhan, Saket and Berenson, Dmitry},
  title     = {Language-Guided Object Search in Agricultural Environments},
  journal   = {International Conference on Robotics and Automation (ICRA)},
  year      = {2025},
}