The problem of low-predictability tasks in robotics.
In reinforcement learning for quadruped robots, there is a series of tasks that do not achieve convergence - despite long training. The reason is often early termination of the exploration trajectory: most attempts end before obtaining a useful gradient signal. Such tasks, called "narrow-viability", are particularly difficult to master with standard algorithms. Researchers identified that the problem does not lie in the lack of reward, but in the physical and dynamic limitations of the system, which prevent long explorations.
In particular, tasks such as "transitioning from walking on four legs to a handstand" are an example of such tasks. In these cases, even the most advanced models cannot learn because most attempts end in a fall or exceeding stability limits before any learning signal is obtained.
Dynamic actuator curriculum as a solution.
The new approach, called "Actuator Dynamics Curriculum", involves gradually reducing joint stiffness during training. Initially, high stiffness is set - which results in a higher natural frequency of feedback below critical damping - which increases the area of task predictability in the Markov decision process. As episodes become longer, stiffness is gradually reduced to a system-identified value.
Researchers used the cart-pole system as a representative example and showed that such a strategy expands the "kernel of monotonization" - i.e., the part of the state space from which the task can be performed. This allows for longer explorations without early termination of trajectories.
Enlarged imageClose zoomPrevious imageVerification on Boston Dynamics Spot robots.
The method was applied to the difficult task of transitioning from walking on four legs to a handstand on a Boston Dynamics Spot robot. Under standard training conditions with fixed stiffness, the model achieved maximum performance only slightly and never completed the transition. Using the dynamic actuator curriculum, the model was trained and demonstrated the ability to perform the transition in simulation on ten different random sets.
This result confirms that a reinforcement learning program with a dynamic actuator curriculum can solve problems that were previously insurmountable. The transition has also been transferred to hardware, indicating its realistic application.
Significance and limitations of the method
The new approach opens up new possibilities for legged robotics, especially in the case of tasks with a high degree of difficulty and limited predictability. It shows that actuator dynamics can be used as a novel type of curriculum - not only as a parameter to optimize, but as a tool to shape the exploration space.
At the same time, the method has limitations: it works best for tasks where the problem is early termination of trajectories, rather than a lack of reward. It is not a universal method for all reinforcement learning tasks. Furthermore, its effectiveness depends on the accurate identification of system dynamics parameters.
The significance of this method extends beyond individual tasks - it opens up new possibilities in designing learning systems for robots that must cope with dynamic and unpredictable environmental conditions. By gradually adjusting actuator dynamics, robots can explore the state space for longer, leading to better integration of learning signals and greater stability during training. This is particularly important in real-world applications, where it is impossible to simulate all extreme conditions. However, the method has its limits: it works most effectively where the problem is limited predictability of trajectories, rather than a lack of reward or low sensitivity of the model to the learning signal.
Therefore, its application should be tailored to the specific conditions of the task and requires precise identification of system dynamics parameters. In the future, it will be possible to extend this concept to other types of robots, such as humanoids or robots with multiple degrees of freedom, which could lead to more flexible and reliable autonomous systems. In the context of global robotics, such approaches can contribute to the realization of advanced exploration missions. mobile robots It is worth emphasizing that the dynamic actuator curriculum method is not a solution for all problems in autonomous robotics. Its effectiveness is limited to tasks where the main obstacle is early termination of the exploration trajectory, rather than a lack of reward or low sensitivity of the model to the learning signal. In practice, this means that this technique works best for tasks with high difficulty and limited predictability, such as transitions between walking and standing on hands, which require long exploration without loss of stability.
Actuator Dynamics Curricula for Narrow-Viability Tasks in Legged Robot Learning
