Modeling of adaptive UAV route control based on reinforcement learning algorithms

Abstract

Subject matter is the reward function, action policy, and learning dynamics of the Proximal Policy Optimization (PPO) algorithm in the task of adaptive UAV navigation under dynamic airspace conditions and limited energy resources. Goal is to develop a simulation environment and a modified Proximal Policy Optimization (PPO) model for adaptive route management of a single UAV in 2D and 3D environments, considering the distance to the target, collision risk, and energy consumption. Tasks: to develop 2D and 3D simulation environments with different obstacle configurations and UAV motion parameters; to formulate a combined PPO reward function that incorporates distance to the target, collisions, and energy consumption; to implement and train PPO, DQN, and A2C algorithms in standardized navigation scenarios; to perform a comparative analysis of algorithm performance using key metrics: path length, number of collisions, reward, and energy consumption; to conduct statistical validation of the results using the t-test and confidence intervals; to analyze the influence of PPO hyperparameters on policy stability and learning convergence in 2D and 3D environments. Methods: deep reinforcement learning algorithms (PPO, DQN, A2C); two simulation models (2D and 3D) with randomly generated static obstacles were developed; a combined reward function was formulated, integrating distance-to-target progress, collision penalties, and an energy-related component; model performance was evaluated using average reward, path length, number of collisions, and total energy expenditure; statistical significance was assessed using the t-test and 95% confidence intervals. Results: The modified PPO model reduced the number of collisions in the 2D environment by 94,8% and shortened the route length by 94,3% compared to the baseline PPO, while exhibiting higher energy consumption due to more complex avoidance maneuvers. In the 3D environment, similar trends were confirmed, including improved navigation safety, more stable policy behavior, and statistically significant improvements across key metrics (p < 0,05). Conclusions: A unified 2D/3D simulation environment for adaptive UAV routing and a modified PPO model with a combined reward function were developed. In the 2D environment, the model achieved a ≈94,8% reduction in collisions, a ≈94,3% reduction in path length, and a ≈92,5% increase in average reward compared to the baseline PPO. In the 3D environment, analogous improvements and statistically significant gains (p < 0,05) were obtained. A relationship between avoidance aggressiveness and energy consumption was identified, enabling selection of an optimal policy for BVLOS scenarios.