Hopping Over Complex Terrain: Sample Efficiency through Reinforcement Learning for Step-Level Control

Navigation of legged robots on complex terrains requires them to achieve precise control of foot placement and speed as a slight miss-step or lack of momentum will lead to failure. Reinforcement Learning (RL) provides a viable alternative to achieve this objective. However, learning low-level joint torques, typically at 200+ Hz, scales poorly even for the simplest sys tems. For example, with n actuators, this would require learning 200n control parameters per second. This thesis takes an alternate approach. A suitable low-level controller that maps the joint torques to the sensor values is assumed. This low-level controller has a few parameters that are tuned once per step by the RL algorithm, typically at 5 Hz (assuming a step time of 0.2 sec). For example, with m free parameters, this requires tuning of only 5m parameters per second. Since 5m ≪ 200n, the proposed approach scales better than the traditional RL approach. The approach is demonstrated on a single-leg hopping robot with two actuators, a rotary actuator for hip swing, and a linear actuator for foot clearance and push-off. The low-level controller is a simple position derivative controller with two free parameters, a proportional gain, and a set-point. There are four sensor measurements at every step, the robot speed and height, and the obstacle height and distance. Here the free parameters m=2 and actuators n=2, hence 10 (5m) ≪ 400 (200n) ensuring scalability. Using proximal policy optimization the control pol icy learns 2 free parameters based on the 4 measurements in about 100,000 to 400,000 trials. The resulting control policy can achieve navigation of the hopper in novel scenarios without re-training or re-tuning.