Lifelong control of off-grid microgrid with model-based reinforcement learning

Microgrid control; Optimization; Reinforcement learning; Developing countries; Learning algorithms; Renewable energy resources; Rural areas; Stochastic systems; System theory; Life span; Micro grid; Model-based reinforcement learning; Off-grids; Optimisations; Reinforcement learnings; Renewable energy source; Rural electrification; System Dynamics; algorithm; alternative energy; benchmarking; electrification; energy policy; rural electrification; sustainability

Abstract :

[en] Off-grid microgrids are receiving a growing interest for rural electrification purposes in developing countries due to their ability to ensure affordable, sustainable and reliable energy services. Off-grid microgrids rely on renewable energy sources (RES) coupled with storage systems to supply the electrical consumption. The inherent uncertainty introduced by RES as well as the stochastic nature of the electrical demand in rural contexts pose significant challenges to the efficient control of off-grid microgrids throughout their entire life span. In this paper, we address the lifelong control problem of an isolated microgrid. We categorize the set of changes that may occur over its life span in progressive and abrupt changes. We propose a novel model-based reinforcement learning algorithm that is able to address both types of changes. In particular, the proposed algorithm demonstrates generalisation properties, transfer capabilities and better robustness in case of fast-changing system dynamics. The proposed algorithm is compared against a rule-based policy and a model predictive controller with look-ahead. The results show that the trained agent is able to outperform both benchmarks in the lifelong setting where the system dynamics are changing over time. © 2021 Elsevier Ltd

Disciplines :

Energy

Author, co-author :

Totaro, S.; Departament de Tecnologies de la Informació i les Comunicacions, Artificial Intelligence and Machine Learning, Universitat Pompeu Fabra, Roc Boronat, Barcelona, 138 08018, Spain

Boukas, Ioannis ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science

Jonsson, A.; Departament de Tecnologies de la Informació i les Comunicacions, Artificial Intelligence and Machine Learning, Universitat Pompeu Fabra, Roc Boronat, Barcelona, 138 08018, Spain

Cornélusse, Bertrand ; Université de Liège - ULiège > Département d'électricité, électronique et informatique (Institut Montefiore) > Smart-Microgrids

Language :

English

Title :

Lifelong control of off-grid microgrid with model-based reinforcement learning

Publication date :

2021

Journal title :

Energy

ISSN :

0360-5442

eISSN :

1873-6785

Publisher :

Elsevier Ltd

Volume :

232

Peer reviewed :

Peer Reviewed verified by ORBi

Funding text :

This research is carried out in the framework of the project Dynamically Evolving Long-Term Autonomy (DELTA), a European research project funded under the CHIST-ERA scheme ( http://www.chistera.eu/ ). Anders Jonsson is partially supported by the grants TIN2015-67959 and PCIN-2017-082 of the Spanish Ministry of Science. The authors would like to thank Sergio Balderrama for the provision of measured data from the “El Espino” microgrid in Bolivia and Alessandro Davide Ialongo for the fruitful discussion.

Available on ORBi :

since 26 December 2022

Statistics

Number of views

62 (0 by ULiège)

Number of downloads

72 (2 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Balderrama, S., Lombardi, F., Riva, F., Canedo, W., Colombo, E., Quoilin, S., A two-stage linear programming optimization framework for isolated hybrid microgrids in a rural context: the case study of the “El Espino” community. Energy, 188, 2019, 116073, 10.1016/j.energy.2019.116073 03605442.
Bellemare, M.G., Dabney, W., Munos, R., A distributional perspective on reinforcement learning. Proceedings of the 34th international conference on machine learning, vol. 70, 2017, JMLR. org, 449–458.
Boukas, I., Ernst, D., Papavasiliou, A., Cornélusse, B., Intra-day bidding strategies for storage devices using deep reinforcement learning. 15th international conference on the European energy market, vol. 14, 2018.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W., OpenAI gym. 2016 arXiv preprint arXiv:1606.01540.
Bruni, G., Cordiner, S., Mulone, V., Sinisi, V., Spagnolo, F., Energy management in a domestic microgrid by means of model predictive controllers. Energy 108 (2016), 119–131, 10.1016/j.energy.2015.08.004 ISSN 0360-5442 https://www.sciencedirect.com/science/article/pii/S0360544215010488 Sustainable Energy and Environmental Protection 2014.
Dabney, W., Rowland, M., Bellemare, M.G., Munos, R., Distributional reinforcement learning with quantile regression. Thirty-second AAAI conference on artificial intelligence, 2018.
Dolara, A., Leva, S., Manzolini, G., Comparison of different physical models for PV power output prediction. Sol Energy 119 (2015), 83–99, 10.1016/j.solener.2015.06.017 ISSN 0038092X.
Dulac-Arnold, G., Mankowitz, D., Hester, T., Challenges of real-world reinforcement learning. 2019 arXiv preprint arXiv:1904.12901.
Foruzan, E., Soh, L.-K., Asgarpoor, S., Reinforcement learning approach for optimal distributed energy management in a microgrid. IEEE Trans Power Syst 33:5 (2018), 5749–5758.
François-Lavet, V., Taralla, D., Ernst, D., Fonteneau, R., Deep reinforcement learning solutions for energy microgrids management. European workshop on reinforcement learning, vol. 2016, 2016, EWRL.
Gajane, P., Ortner, R., Auer, P., A sliding-window approach for reinforcement learning in mdps with arbitrarily changing rewards and transitions. Proceedings of the 2nd ICML/IJCAI workshop on lifelong learning: a reinforcement learning approach, 2018, LLARLA 2018).
Imani, E., White, M., Improving regression performance with distributional losses. 2018 arXiv preprint arXiv:1806.04613.
Janner, M., Fu, J., Zhang, M., Levine, S., When to trust your model: model-based policy optimization. Advances in neural information processing systems, 2019, 12498–12509.
Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Campbell, R.H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., Levine, S., et al. Model-based reinforcement learning for atari. 2019 arXiv preprint arXiv:1903.00374.
Kofinas, P., Dounis, A., Vouros, G., Fuzzy q-learning for multi-agent decentralized energy management in microgrids. Appl Energy 219 (2018), 53–67, 10.1016/j.apenergy.2018.03.017 ISSN 0306-2619 https://www.sciencedirect.com/science/article/pii/S0306261918303465.
Kuh, A., Petsche, T., Rivest, R.L., Learning time-varying concepts. Advances in neural information processing systems, 1991, 183–189.
Kuznetsova, E., Li, Y.F., Ruiz, C., Zio, E., Ault, G., Bell, K., Reinforcement learning for microgrid energy management. Energy 59 (2013), 133–146, 10.1016/j.energy.2013.05.060 ISSN 03605442 https://www.sciencedirect.com/science/article/pii/S0360544213004817.
Lee, D., Powell, W.B., An intelligent battery controller using bias-corrected q-learning. 2012.
Lim, S.H., Xu, H., Mannor, S., Reinforcement learning in robust markov decision processes. Advances in neural information processing systems, 2013, 701–709.
Lombardi, F., Balderrama, S., Quoilin, S., Colombo, E., Generating high-resolution multi-energy load profiles for remote areas with an open-source stochastic model. Energy 177 (2019), 433–444, 10.1016/j.energy.2019.04.097 ISSN 03605442.
Lopes, M., Lang, T., Toussaint, M., Oudeyer, P.-Y., Exploration in model-based reinforcement learning by empirically estimating learning progress. Neural information processing systems, 2012, NIPS).
Nagabandi, A., Kahn, G., Fearing, R.S., Levine, S., Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. 2018 IEEE international conference on robotics and automation (ICRA), 2018, IEEE, 7559–7566.
Nilim, A., El Ghaoui, L., Robust control of markov decision processes with uncertain transition matrices. Oper Res 53:5 (2005), 780–798.
Nyong-Bassey, B.E., Giaouris, D., Patsios, C., Papadopoulou, S., Papadopoulos, A.I., Walker, S., Voutetakis, S., Seferlis, P., Gadoue, S., Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty. Energy, 193, 2020, 116622, 10.1016/j.energy.2019.116622 ISSN 0360-5442 https://www.sciencedirect.com/science/article/pii/S0360544219323175.
Parisio, A., Rikos, E., Tzamalis, G., Glielmo, L., Use of model predictive control for experimental microgrid optimization. Appl Energy 115 (2014), 37–46, 10.1016/j.apenergy.2013.10.027 ISSN 03062619.
Polydoros, A.S., Nalpantidis, L., Survey of model-based reinforcement learning: applications on robotics. J Intell Rob Syst 86:2 (2017), 153–173.
Rocchetta, R., Bellani, L., Compare, M., Zio, E., Patelli, E., A reinforcement learning framework for optimal operation and maintenance of power grids. Appl Energy 241 (2019), 291–301, 10.1016/j.apenergy.2019.03.027 ISSN 0306-2619 https://www.sciencedirect.com/science/article/pii/S0306261919304222.
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T., Silver, D., Mastering atari, go, chess and shogi by planning with a learned model. 2019.
Schulman, J., Levine, S., Moritz, P., Jordan, M., Abbeel, P., Trust region policy optimization. ICML 2015-proceedings of the 31st international conference on machine learning, 2015.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O., Proximal policy optimization algorithms. http://arxiv.org/abs/1707.06347, 2017 1-12.
Shuai, H., He, H., Wen, J., On-line scheduling of a residential microgrid via monte-carlo tree search and a learned model. 2020 arXiv preprint arXiv:2005.06161.
Sutton, R.S., Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Porter, B., Mooney, R., (eds.) Machine learning proceedings 1990, 1990, Morgan Kaufmann, San Francisco (CA) 978-1-55860-141-3, 216–224, 10.1016/B978-1-55860-141-3.50030-4 https://www.sciencedirect.com/science/article/pii/B9781558601413500304.
Sutton, R.S., McAllester, D.A., Singh, S.P., Mansour, Y., Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 2000, 1057–1063.
Sutton, R.S., Szepesvári, C., Geramifard, A., Bowling, M.P., Dyna-style planning with linear function approximation and prioritized sweeping. 2012 arXiv preprint arXiv:1206.3285.
Taylor, S.J., Letham, B., Forecasting at scale. Am Statistician 72:1 (2018), 37–45.
Tsymbal, A., The problem of concept drift: definitions and related work. Comput Sci Depart Trinity Coll Dublin, 106(2), 2004, 58.
van Hasselt, H.P., Hessel, M., Aslanides, J., When to use parametric models in reinforcement learning?. Advances in neural information processing systems, 2019, 14322–14333.
Watkins, C.J., Dayan, P., Q-learning. Mach Learn 8:3–4 (1992), 279–292.
Zhang, Y., Fu, L., Zhu, W., Bao, X., Liu, C., Robust model predictive control for optimal energy management of island microgrids with uncertainties. Energy 164 (2018), 1229–1241, 10.1016/j.energy.2018.08.200 ISSN 0360-5442 https://www.sciencedirect.com/science/article/pii/S0360544218317365.