Reinforcement Learning For Optimal Power Flow Control In Renewable-Integrated Microgrids
DOI:
https://doi.org/10.29284/cndtp863Keywords:
Deep Reinforcement Learning; Optimal Power Flow; Microgrid; Proximal Policy Optimisation; Renewable Energy; Battery Energy Storage; Voltage Regulation; Energy Management.Abstract
The rapid integration of stochastic renewable energy sources (RES) into distribution-level microgrids fundamentally undermines traditional optimisation-based optimal power flow (OPF) methodologies, which depend on precise system models and deterministic forecasts that are seldom accessible in practice. This research presents a deep reinforcement learning (DRL) framework using the Proximal Policy Optimisation (PPO) method to address the OPF problem in hybrid microgrids containing photovoltaic (PV) arrays, wind turbines, battery energy storage systems (BESS), and diesel backup generators. The proposed model learns a continuous, model-free control policy that simultaneously minimises operating costs, curtails voltage variances, and maximises renewable utilisation under time-varying load and generation uncertainty. The environment is constructed as a Markov Decision Process (MDP) utilising an IEEE 33-bus distribution test feeder enhanced with renewable generation profiles obtained from real-world irradiance and wind speed data. Comparative assessment with Model Predictive Control (MPC), Deep Q-Network (DQN), and Deep Deterministic Policy Gradient (DDPG) base values indicates that PPO achieves a 13.1% reduction in daily operating costs, minimises voltage violation occurrences by 79.4%, and increases renewable usage to 84.9%, exceeding all baselines across all important performance metrics while converging in fewer training events. Simulation results further indicate real-time practicality with a mean decision latency of 31 ms, making the proposed method appropriate for hardware deployment in practical microgrid energy management systems.
References
1. Ahmad, S., Shafiullah, M., Ahmed, C. B., & Alowaifeer, M. (2023). A review of microgrid energy management and control strategies. IEEE Access, 11, 21729–21757. https://doi.org/10.1109/ACCESS.2023.3251555
2. Ali, K. H., Abusara, M., Tahir, A. A., & Das, S. (2023). Dual-layer Q-learning strategy for energy management of battery storage in grid-connected microgrids. Energies, 16(3), 1334. https://doi.org/10.3390/en16031334
3. Liu, D., Zang, C., Zeng, P., Li, H., He, S., & Li, S. (2023). Deep reinforcement learning for real-time economic energy management of a microgrid system considering uncertainties. Frontiers in Energy Research, 11, Article 1163053. https://doi.org/10.3389/fenrg.2023.1163053
4. Li, Y., Yu, C., Shahidehpour, M., Yang, T., Zeng, Z., & Chai, T. (2023). Deep reinforcement learning for smart grid operations: Algorithms, applications, and prospects. Proceedings of the IEEE, 111(9), 1055–1096. https://doi.org/10.1109/JPROC.2023.3285589
5. Meng, Q., Zhang, Y., & Guan, Y. (2024). An online reinforcement learning-based energy management strategy for microgrids with centralized control. IEEE Transactions on Industrial Applications. https://doi.org/10.1109/TIA.2024.3351001
6. Moreno-Castro, J., et al. (2023). Microgrid management strategies for economic dispatch of electricity using model predictive control techniques: A review. Energies, 16(14), 5935. https://doi.org/10.3390/en16145935
7. Pan, X., Chen, M., Zhao, T., & Low, S. H. (2022). DeepOPF: A feasibility-optimized deep neural network approach for AC optimal power flow problems. IEEE Systems Journal, 17(1), 673–683. https://doi.org/10.1109/JSYST.2022.3201041
8. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimisation algorithms. arXiv preprint arXiv:1707.06347. https://doi.org/10.48550/arXiv.1707.06347
9. Shirkhani, M., et al. (2023). A review of microgrid decentralised energy/voltage control structures and methods. Energy Reports, 10, 368–380. https://doi.org/10.1016/j.egyr.2023.06.038
10. Yi, Z., Wang, X., Yang, C., Niu, M., & Yin, W. (2023). Real-time sequential security-constrained optimal power flow: A hybrid knowledge-data-driven reinforcement learning approach. IEEE Transactions on Power Systems, 39, 1664–1680. https://doi.org/10.1109/TPWRS.2023.3234025
11. Zhang, T., et al. (2023). A Bayesian deep reinforcement learning-based resilient control for a multi-energy microgrid. IEEE Transactions on Power Systems, 38(6), 5057–5072. https://doi.org/10.1109/TPWRS.2023.3239283
12. Zhao, B., Zhang, X., Li, P., Wang, K., Xue, M., & Wang, C. (2022). Optimal sizing, operating strategy, and operational experience of a stand-alone microgrid on Dongfushan island. Applied Energy, 113, 1656–1666. https://doi.org/10.1016/j.apenergy.2022.118654
13. Zhou, S., Hu, Z., Gu, W., Jiang, M., & Zhang, X. P. (2024). Deep reinforcement learning-based multi-objective energy management system for microgrids under a flexible energy market. Applied Energy. https://doi.org/10.1016/j.apenergy.2025.019531
14. Zhang, Y., Wang, J., & Chen, B. (2019). Deep reinforcement learning for real-time optimal energy management in microgrids. IEEE Transactions on Smart Grid, 10(5), 5637–5648.
15. Li, H., Wan, Z., & He, H. (2018). Real-time residential demand response using deep reinforcement learning. IEEE Transactions on Smart Grid, 10(4), 1–11.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
