The problem of low-predictability tasks in robotics
In reinforcement learning for legged robots, there is a series of tasks that do not achieve convergence - even after long training. The reason is often early termination of the exploration trajectory: most attempts end before obtaining a useful gradient signal. Such tasks, called "narrow-viability," are particularly difficult to master with standard algorithms. Researchers have identified that the problem does not lie in the lack of reward, but in the physical and dynamic constraints of the system, which prevent long explorations.
In particular, tasks such as "transitioning from walking on four legs to a handstand" are an example of such tasks. In these cases, even the most advanced models cannot learn because most attempts end in a fall or exceeding stability limits before any learning signal is obtained.
Dynamic actuator curriculum as a solution
The new approach, called "Actuator Dynamics Curriculum," involves gradually decreasing joint stiffness during training. Initially, a high stiffness is set, which results in a higher natural frequency of feedback below critical damping, increasing the area of task predictability in the Markov decision process. As episodes become longer, the stiffness is gradually reduced to a system-identified value.
Researchers used a cart-pole system as a representative example and showed that this strategy expands the "monotonicity kernel" - which is the portion of the state space from which the task can be successfully performed. This allows for longer explorations without prematurely terminating trajectories.
Enlarged imageClose zoomPrevious imageVerification on Boston Dynamics Spot robots
The method was applied to the challenging task of transitioning from quadrupedal walking to a hand-standing state on a Boston Dynamics Spot robot. Under standard training conditions with fixed stiffness, the model achieved only limited success and never completed the transition. Using the dynamic actuator curriculum, the model was trained and demonstrated the ability to perform the transition in simulation across ten different sets of random trials.
This result confirms that reinforcement learning with a dynamic actuator curriculum can solve problems that were previously insurmountable. The transition was also transferred to hardware, indicating its realistic applicability.
Significance and limitations of the method
The new approach opens up new possibilities for legged robotics, especially for tasks with a high degree of difficulty and limited predictability. It shows that actuator dynamics can be used as a novel type of curriculum - not only as a parameter to optimize, but as a tool to shape the exploration space.
At the same time, the method has limitations: it works best for tasks where the problem is early termination of trajectories, rather than a lack of reward. It is not a universal method for all reinforcement learning tasks. Furthermore, its effectiveness depends on the accurate identification of system dynamics parameters.
The significance of this method extends beyond individual tasks - it opens up new possibilities in designing learning systems for robots that must cope with dynamic and unpredictable environmental conditions. By gradually adjusting actuator dynamics, robots can explore the state space for longer, leading to better integration of learning signals and greater stability during training. This is particularly important in real-world applications where it is not possible to simulate all extreme conditions. However, the method has its limits: it works most effectively when the problem is limited predictability of trajectories, rather than a lack of reward or too low sensitivity of the model to the signal.
Therefore, its application should be tailored to specific task conditions and requires precise identification of system dynamics parameters. In the future, it will be possible to extend this concept to other types of robots, such as humanoids or mobile robots with multiple degrees of freedom, which may lead to more flexible and reliable autonomous systems. In the context of global robotics, such approaches can contribute to the implementation of advanced exploration missions.
It is worth emphasizing that the method of dynamic actuator curricula is not a solution for all problems in autonomous robotics. Its effectiveness is limited to tasks where the main obstacle is early termination of the exploration trajectory, rather than a lack of reward or low sensitivity of the model to the learning signal. In practice, this means that this technique works best for high-difficulty and limited predictability tasks, such as transitions between motion and standing states, which require long exploration without loss of stability.
