宇航学报

宇航学报 ›› 2019, Vol. 40 ›› Issue (4): 435-443.doi: 10.3873/j.issn.1000-1328.2019.04.008

• 制导、导航、控制与电子 • 上一篇    下一篇

基于Sarsa(λ)强化学习的空间机械臂路径规划研究

徐帷，卢山   

  1. 1. 上海航天控制技术研究所，上海 201109；2. 上海市空间智能控制技术重点实验室，上海 201109
  • 收稿日期:2018-07-10 修回日期:2018-12-14 出版日期:2019-04-15 发布日期:2019-04-25
  • 基金资助:

    上海市科技人才计划（17XD1420700）；上海市自然科学基金（16ZR1415600）

Analysis of Space Manipulator Route Planning Based on Sarsa (λ) Reinforcement Learning

XU Wei, LU Shan   

  1. 1. Shanghai Institute of Spaceflight Control Technology, Shanghai 201109, China; 2. Shanghai Key Laboratory of Aerospace Intelligent Control Technology, Shanghai 201109, China
  • Received:2018-07-10 Revised:2018-12-14 Online:2019-04-15 Published:2019-04-25

PDF

513

摘要：

针对目标特性未知的在轨操作环境，研究了典型空间操作机械臂的路径规划策略。采用Sarsa(λ)强化学习方法实现目标跟踪及避障的自主路径规划与智能决策，该方法将机械臂系统的每节臂视为一个决策智能体，通过感知由目标偏差和障碍距离程度组成的二维状态，设计符合人工经验的拟合奖赏函数，进行各臂转动动作的强化训练，最终形成各智能体的状态-动作值函数表，即可作为机械臂在线路径规划的决策依据。将本方法应用于多自由度空间机械臂路径规划任务，仿真结果表明新算法能在有限训练次数内实现对移动目标的稳定跟踪与避障，同时各智能体通过学习所得的状态-动作值函数表，具备较强的后期在线自主调整能力，从而验证了算法较强的鲁棒性和智能性。

关键词: 强化学习, Sarsa方法, 空间机械臂, 路径规划

Abstract:

 Focusing on the on-orbit manipulating environment with uncertain target characters，the route planning strategy of a typical space manipulator is studied. The Sarsa(λ) reinforcement learning algorithm is used to achieve the goal of the autonomous route planning and intelligent decision for the tasks on target tracking and obstacle avoidance. This method considers each arm in a manipulator system as a decision agent, by means of percepting the two dimensional states consisting of the target deviation and the degree of obstacle distance, designing and fitting a reward function corresponding to the artificial experience，and the reinforced training on rotating action by each arm，the final state-action value function table of each agent can be used as a decision basis for the online manipulator route planning. We use this method on route planning task by a space manipulator with multi-degree of freedom, the simulation result shows that this new algorithm can achieve the requirement for stable tracking of a moving target and simultaneous obstacle avoidance within finite training times, meanwhile, the state-action value function table of each agent obtained from the reinforcement learning possesses the strong capacity of subsequent online autonomous adjustment, which validates the robustness and intelligence of this algorithm.

Key words: Reinforcement learning, Sarsa method, Space manipulator, Route planning

中图分类号: