Energy systems; Design and control; Renewable energy source; Reinforcement learning
Abstract :
[en] The decentralisation and unpredictability of new renewable energy sources require rethinking our energy system. Data-driven approaches, such as reinforcement learning (RL), have emerged as new control strategies for operating these systems, but they have not yet been applied to system design. This paper aims to bridge this gap by studying the use of an RL-based method for joint design and control of a real-world PV and battery system. The design problem is first formulated as a mixed-integer linear programming problem (MILP). The optimal MILP solution is then used to evaluate the performance of an RL agent trained in a surrogate environment designed for applying an existing data-driven algorithm. The main difference between the two models lies in their optimization approaches: while MILP finds a solution that minimizes the total costs for a one-year operation given the deterministic historical data, RL is a stochastic method that searches for an optimal strategy over one week of data on expectation over all weeks in the historical dataset. Both methods were applied on a toy example using one-week data and on a case study using one-year data. In both cases, models were found to converge to similar control solutions, but their investment decisions differed. Overall, these outcomes are an initial step illustrating benefits and challenges of using RL for the joint design and control of energy systems.
Disciplines :
Energy
Author, co-author :
Cauz, Marine; École polytechnique fédérale de Lausanne (EPFL), Institute of Electrical and Micro Engineering (IEM), Photovoltaics and Thin-Film Electronics Laboratory (PV-Lab), Neuchâtel, Switzerland ; Planair SA, Yverdon-les-bains, Switzerland
Bolland, Adrien ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
Miftari, Bardhyl ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science
Perret, Lionel; Planair SA, Yverdon-les-bains, Switzerland
Ballif, Christophe; École polytechnique fédérale de Lausanne (EPFL), Institute of Electrical and Micro Engineering (IEM), Photovoltaics and Thin-Film Electronics Laboratory (PV-Lab), Neuchâtel, Switzerland ; Centre Suisse d'Electronique et de Microtechnique (CSEM), PV-Center, Neuchâtel, Switzerland
Wyrsch, Nicolas; École polytechnique fédérale de Lausanne (EPFL), Institute of Electrical and Micro Engineering (IEM), Photovoltaics and Thin-Film Electronics Laboratory (PV-Lab), Neuchâtel, Switzerland
Language :
English
Title :
Reinforcement Learning for Joint Design and Control of Battery-PV Systems
Publication date :
2023
Event name :
36th International Conference on Efficiency, Cost, Optimization, Simulation and Environmental Impact of Energy Systems (ECOS 2023)
Event place :
Las Palmas de Gran Canaria, Esp
Event date :
25-06-2023 => 30-06-2023
By request :
Yes
Audience :
International
Main work title :
36th International Conference on Efficiency, Cost, Optimization, Simulation and Environmental Impact of Energy Systems, ECOS 2023
Publisher :
International Conference on Efficiency, Cost, Optimization, Simulation and Environmental Impact of Energy Systems
A. Perera and P. Kamalaruban, “Applications of reinforcement learning in energy systems,” Renewable and Sustainable Energy Reviews, vol. 137, p. 110 618, Mar. 2021. DOI: 10.1016/j.rser.2020.110618.
A. T. D. Perera, P. U. Wickramasinghe, V. M. Nik, and J.-L. Scartezzini, “Introducing reinforcement learning to the energy system design process,” en, Applied Energy, vol. 262, p. 114580, Mar. 2020, ISSN: 0306-2619. DOI: 10. 1016/j.apenergy.2020.114580. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0306261920300921 (visited on 10/24/2022).
S. Fazlollahi and F. Maréchal, “Multi-objective, multi-period optimization of biomass conversion technologies using evolutionary algorithms and mixed integer linear programming (MILP),” en, Applied Thermal Engineering, Combined Special Issues: ECP 2011 and IMPRES 2010, vol. 50, no. 2, pp. 1504-1513, Feb. 2013, ISSN: 1359-4311. DOI: 10.1016/j.applthermaleng.2011.11.035. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1359431111006636 (visited on 11/14/2022).
A. Majid, S. Saaybi, T. Rietbergen, et al., Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey. May 2021. DOI: 10.36227/techrxiv.14679504.v1.
H. Quest, M. Cauz, F. Heymann, et al., “A 3D indicator for guiding AI applications in the energy sector,” en, Energy and AI, vol. 9, p. 100 167, Aug. 2022, ISSN: 2666-5468. DOI: 10.1016/j.egyai.2022.100167. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2666546822000234 (visited on 11/03/2022).
H. M. Abdullah, A. Gastli, and L. Ben-Brahim, “Reinforcement Learning Based EV Charging Management Systems-A Review,” IEEE Access, vol. 9, pp. 41 506-41 531, 2021, Conference Name: IEEE Access, ISSN: 2169-3536. DOI: 10.1109/ACCESS.2021.3064354.
M. Dorokhova, Y. Martinson, C. Ballif, and N. Wyrsch, “Deep reinforcement learning control of electric vehicle charging in the presence of photovoltaic generation,” Applied Energy, vol. 301, p. 117 504, Nov. 2021. DOI: 10.1016/j. apenergy.2021.117504.
W. Shi and V. W. Wong, “Real-time vehicle-to-grid control algorithm under price uncertainty,” in 2011 IEEE International Conference on Smart Grid Communications (SmartGridComm), Oct. 2011, pp. 261-266. DOI: 10.1109/SmartGridComm.2011.6102330.
W. Uther, “Markov Decision Processes,” en, in Encyclopedia of Machine Learning, C. Sammut and G. I. Webb, Eds., Boston, MA: Springer US, 2010, pp. 642-646, ISBN: 978-0-387-30164-8. DOI: 10.1007/978-0-387-30164-8_512. [Online]. Available: https://doi.org/10.1007/978-0-387-30164-8_512 (visited on 03/10/2023).
Manu Lahariya, N. Sadeghianpourhamami, and Chris Develder, “Computationally efficient joint coordination of multiple electric vehicle charging points using reinforcement learning,” [Online]. Available: arXiv:2203.14078.
N. Sadeghianpourhamami, J. Deleu, and C. Develder, “Definition and Evaluation of Model-Free Coordination of Electrical Vehicle Charging With Reinforcement Learning,” IEEE Transactions on Smart Grid, vol. 11, no. 1, pp. 203-214, Jan. 2020, Conference Name: IEEE Transactions on Smart Grid, ISSN: 1949-3061. DOI: 10.1109/TSG.2019. 2920320.
A. Bolland, I. Boukas, M. Berger, and D. Ernst, “Jointly Learning Environments and Control Policies with Projected Stochastic Gradient Ascent,” en, Journal of Artificial Intelligence Research, vol. 73, pp. 117-171, Jan. 2022, ISSN: 1076-9757. DOI: 10.1613/jair.1.13350. [Online]. Available: https://www.jair.org/index.php/jair/article/view/13350 (visited on 03/07/2023).
R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” en, Machine Learning, vol. 8, no. 3, pp. 229-256, May 1992, ISSN: 1573-0565. DOI: 10.1007/BF00992696. [Online]. Available: https://doi.org/10.1007/BF00992696 (visited on 03/07/2023).
B. Miftari, M. Berger, H. Djelassi, and D. Ernst, “GBOML: Graph-Based Optimization Modeling Language,” en, Journal of Open Source Software, vol. 7, no. 72, p. 4158, Apr. 2022, ISSN: 2475-9066. DOI: 10.21105/joss.04158. [Online]. Available: https://joss.theoj.org/papers/10.21105/joss.04158 (visited on 03/07/2023).
Gurobi, Gurobi - The fastest solver, Library Catalog: www.gurobi.com, 2020. [Online]. Available: https://www.gurobi.com/ (visited on 03/25/2020).
M. Reuß, L. Welder, J. Thürauf, et al., “Modeling hydrogen networks for future energy systems: A comparison of linear and nonlinear approaches,” en, International Journal of Hydrogen Energy, vol. 44, no. 60, pp. 32 136-32 150, Dec. 2019, ISSN: 0360-3199. DOI: 10.1016/j.ijhydene.2019.10.080. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0360319919338625 (visited on 10/24/2022).
C. Sánchez, L. Bloch, J. Holweger, C. Ballif, and N. Wyrsch, “Optimised Heat Pump Management for Increasing Photovoltaic Penetration into the Electricity Grid,” Energies, vol. 12, p. 1571, Apr. 2019. DOI: 10.3390/en12081571.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal Policy Optimization Algorithms, arXiv:1707.06347 [cs], Aug. 2017. [Online]. Available: http://arxiv.org/abs/1707.06347 (visited on 03/12/2023).
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, High-Dimensional Continuous Control Using Generalized Advantage Estimation, arXiv:1506.02438 [cs], Oct. 2018. [Online]. Available: http://arxiv.org/abs/1506.02438 (visited on 03/12/2023).
