REDQ-Agent Optimized Downlink Communication in 3D RIS-Assisted Wireless Systems

In this research study, a Reconfigurable Intelligent Surface (RIS)–assisted Multiple-Input Single-Output (MISO) downlink wireless communication system is implemented in three-dimensional (3D) space. Each entity—base station (BS), RIS, and user equipments (UEs)—has its own geographical coordinates (X, Y, Z). The distances between the fixed entities (BS and RIS) and the randomly located UEs are computed using the Euclidean norm. The BS–RIS and RIS–UEs links are modeled as rician fading channels, whereas the BS–UEs links are modeled as rayleigh fading channels. Joint beamforming at the BS and phase shift at the RIS is formulated to provide a better spectral efficiency to the UEs. To achieve this, the entire communication system is translated into a Reinforcement Learning (RL) framework with state, action, and reward. A Randomized Ensembled Double Q-Learning (REDQ) RL agent, which belongs to the model-free, off-policy RL family, is trained on this RL framework. REDQ has produced a better average spectral efficiency compared to other model-free, off-policy RL agents such as Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Soft-Actor Critic (SAC) in all experimental setups (signal power and RIS elements). This improvements is attributed to REDQ’s better update-to-data (UTD) ratio, ensemble of critics, and controlled Q-function variance with a common target value for all critics. These findings underscore the potential of REDQ in enhancing performance in RIS-assisted wireless communication systems.