Gym-ANM: Open-source software to leverage reinforcement learning for power system management in research and education

[en] Gym-ANM is a Python package that facilitates the design of reinforcement learning (RL) environments that model active network management (ANM) tasks in electricity networks. Here, we describe how to implement new environments and how to write code to interact with pre-existing ones. We also provide an overview of ANM6-Easy, an environment designed to highlight common ANM challenges. Finally, we discuss the potential impact of Gym-ANM on the scientific community, both in terms of research and education. We hope this package will facilitate collaboration between the power system and RL communities in the search for algorithms to control future energy systems.

Disciplines :

Energy
Computer science
Electrical & electronics engineering

Author, co-author :

Henry, Robin

Ernst, Damien ; Université de Liège - ULiège > Dép. d'électric., électron. et informat. (Inst.Montefiore) > Smart grids

Language :

English

Title :

Gym-ANM: Open-source software to leverage reinforcement learning for power system management in research and education

Publication date :

June 2021

Journal title :

Software Impacts

eISSN :

2665-9638

Publisher :

Elsevier, Amsterdam, Netherlands

Volume :

Peer reviewed :

Peer Reviewed verified by ORBi

Additional URL :

https://arxiv.org/abs/2105.08846
https://www.sciencedirect.com/science/article/pii/S2665963821000348

Available on ORBi :

since 21 May 2021

Statistics

Number of views

231 (17 by ULiège)

Number of downloads

145 (10 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

Bibliography

Gill, S., Kockar, I., Ault, G.W., Dynamic optimal power flow for active distribution networks. IEEE Trans. Power Syst. 29:1 (2013), 121–131.
McDonald, J., Adaptive intelligent power systems: Active distribution networks. Energy Policy 36:12 (2008), 4346–4351.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M., Playing atari with deep reinforcement learning. 2013 arXiv preprint arXiv:1312.5602.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature 518:7540 (2015), 529–533.
Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587), 2016, 484.
Vinyals, O., Babuschkin, I., Czarnecki, W.M., Mathieu, M., Dudzik, A., Chung, J., Choi, D.H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575:7782 (2019), 350–354.
Deisenroth, M.P., Neumann, G., Peters, J., et al. A survey on policy search for robotics. Found. Trend. Robot. 2:1–2 (2013), 1–142.
Kormushev, P., Calinon, S., Caldwell, D.G., Reinforcement learning in robotics: Applications and real-world challenges. Robotics 2:3 (2013), 122–148.
Kober, J., Bagnell, J.A., Peters, J., Reinforcement learning in robotics: A survey. Int. J. Robot. Res. 32:11 (2013), 1238–1274.
Gu, S., Holly, E., Lillicrap, T., Levine, S., Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017, IEEE, 3389–3396.
Sallab, A.E., Abdou, M., Perot, E., Yogamani, S., Deep reinforcement learning framework for autonomous driving. Electron. Imaging 2017:19 (2017), 70–76.
O'Kelly, M., Sinha, A., Namkoong, H., Tedrake, R., Duchi, J.C., Scalable end-to-end autonomous vehicle testing via rare-event simulation. Advances in Neural Information Processing Systems, 2018, 9827–9838.
Li, D., Zhao, D., Zhang, Q., Chen, Y., Reinforcement learning and deep learning based lateral control for autonomous driving [application notes]. IEEE Comput. Intell. Magaz. 14:2 (2019), 83–98.
Henry, R., Ernst, D., Gym-ANM: Reinforcement learning environments for active network management tasks in electricity distribution systems. 2021 arXiv preprint arXiv:2103.07932.
Zimmerman, R.D., Murillo-Sánchez, C.E., Thomas, R.J., MATPOWER: Steady-state operations, planning, and analysis tools for power systems research and education. IEEE Trans. Power Syst. 26:1 (2010), 12–19.
Thurner, L., Scheidler, A., Schäfer, F., Menke, J., Dollichon, J., Meier, F., Meinecke, S., Braun, M., Pandapower — An open-source python tool for convenient modeling, analysis, and optimization of electric power systems. IEEE Trans. Power Syst. 33:6 (2018), 6510–6521, 10.1109/TPWRS.2018.2829021.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W., Openai gym. 2016 arXiv preprint arXiv:1606.01540.
Gonzalez-Longatt, F.M., Rueda, J.L., PowerFactory Applications for Power System Analysis. 2014, Springer.
Langley, H., Wright, K., ERACS-a comprehensive package for PCs. IEE Colloquium on Interactive Graphic Power System Analysis Programs, 1992, IET p. 3/1-3/7.
Brown, K., Shokooh, F., Abcede, H., Donner, G., Interactive simulation of power systems: ETAP applications and techniques. Conference Record of the 1990 IEEE Industry Applications Society Annual Meeting, 1990, IEEE, 1930–1941.
TNEI, Interactive Power System Analysis (IPSA) software, https://www.ipsa-power.com, (Accessed on 05/12/2021).
PowerWorld Corporation, PowerWorld software, https://www.powerworld.com, (Accessed on 05/12/2021).
Milano, F., An open source power system analysis toolbox. IEEE Trans. Power Syst. 20:3 (2005), 1199–1206.
R. Lincoln, PYPOWER library, https://github.com/rwl/PYPOWER, (Accessed on 05/12/2021).
D. Ernst, Optimal decision making for complex problems course at the University of Liège, http://blogs.ulg.ac.be/damien-ernst/info8003-1-optimal-decision-making-for-complex-problems, (Accessed on 05/13/2021).