Paper Title: Deep Q-network path planning with stratified sampling and prioritized replay
Authors: Yuxing Zhang, Setyawan Widyarto
Corresponding Author: Setyawan Widyarto (9212000501@student.unisel.edu.my)/ Malaysia
Abstract
Deep reinforcement learning (DRL) has been widely adopted for mobile robot path planning in unknown and complex environments. However, existing DRL-based approaches suffer from two main limitations: sparse reward, which provides meaningful feedback only at terminal states, and experience heterogeneity, which reduces the efficiency of uniform sampling from the standard experience replay buffer due to the diversity of transition qualities stored within it. To tackle both issues at once, we propose SS-PER-DQN, a novel path-planning approach in which stratified sampling and a prioritized experience replay mechanism are incorporated into the Double Deep Q-Network. The memory buffer is divided into three priority layers based on temporal-difference error levels, with a dynamically varying sampling ratio between layers, and importance-sampling compensation to balance the sampling-distribution bias. A densely shaped reward function is additionally employed to expedite early exploration in the presence of sparsely rewarded actions in the environment. Experiments performed on Grid World and ROS Gazebo simulations show that the SS-PER-DQN is able to successfully plan paths at success rates of 94.5% and 89.0%, with a 24.5% and 21.8% reduction in path lengths, respectively, when compared to the traditional DQN algorithm, after convergence in approximately 53% fewer training episodes. The analysis conducted during the ablation study reveals the specific roles of each contribution: stratified sampling helps the algorithm converge early by ensuring that informative state transitions guide the first phase of learning, and importance weighting influences the quality of gradient updates in later stages. It is also observed that the absolute planning time per decision step ranges from 5.0 ms (Grid World) to 5.3 ms (Gazebo) for SS-PER-DQN, representing an incremental overhead of approximately 1.9–2.2 ms per step relative to the classic DQN baseline, which remains well within the real-time requirements for robot navigation.
