PlanAgent

A multi-modal large language agent for closed-loop vehicle motion planning

PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning

PlanAgent introduces a multi-modal large language agent framework for closed-loop vehicle motion planning. This innovative approach leverages large language models to interpret complex driving scenarios, reason about traffic rules and safety constraints, and generate appropriate motion plans.

Key Features

  • Multi-modal Understanding: Integrates visual perception with natural language reasoning
  • Safety-First Planning: Incorporates traffic rules and safety constraints
  • Interpretable Decisions: Provides natural language explanations for planning choices
  • Adaptable Behavior: Can handle diverse driving scenarios and requirements

Technical Innovation

The system demonstrates several key advancements:

  • Integration of large language models with motion planning
  • Natural language-based safety constraint handling
  • Real-time adaptation to changing scenarios
  • Improved interpretability of autonomous decisions
<!--
  See https://www.debugbear.com/blog/responsive-images#w-descriptors-and-the-sizes-attribute and
  https://developer.mozilla.org/en-US/docs/Learn/HTML/Multimedia_and_embedding/Responsive_images for info on defining 'sizes' for responsive images
-->

  <source
    class="responsive-img-srcset"
    
      srcset="/assets/img/planagent_arch-480.webp 480w,/assets/img/planagent_arch-800.webp 800w,/assets/img/planagent_arch-1400.webp 1400w,"
      type="image/webp"
    
    
      sizes="95vw"
    
  >

<img
  src="/assets/img/planagent_arch.jpg"
  
    class="img-fluid rounded z-depth-1"
  
  
    width="100%"
  
  
    height="auto"
  
  
  
  
    title="PlanAgent Architecture"
  
  
  
    loading="eager"
  
  onerror="this.onerror=null; $('.responsive-img-srcset').remove();"
>

</picture>

</figure>

</div>

</div> –>

The architecture of PlanAgent, showing how language models are integrated with motion planning.

Applications

  • Autonomous Vehicle Planning: More robust and interpretable motion planning
  • Safety Verification: Natural language-based safety constraint checking
  • Human-AI Interaction: Better communication of planning decisions
  • Research Platform: Foundation for further research in language-guided planning

This work represents a significant step toward making autonomous vehicle planning more interpretable, safe, and adaptable to complex real-world scenarios.

References