Maximum Entropy RL and Policy Gradients: Why and What to Explore?

This presentation studies policy gradient algorithms that regularize entropy in their learning objective to encourage exploration, i.e., algorithms that optimize maximum-entropy reinforcement learning objectives. In the first part, the influence of entropy regularization on policy gradient performance is discussed in the light of numerical optimization theory. The focus lies on how it reshapes the learning objective and how it helps satisfy key assumptions used in convergence proofs. In the second part, a maximum-entropy objective that enforces exploration of future states and actions is discussed. A concrete off-policy policy-gradient algorithm is developed, discussed, and illustrated empirically.