Demand response, or demand-side management, improves grid stability by increasing demand flexibility, and shifts peak demand towards periods of peak renewable energy generation by providing consumers with economic incentives. The future of demand response greatly depends on its ability to prevent consumer discomfort and integrate human feedback into the control loop. Reinforcement learning is a potentially model-free algorithm that can adapt to its environment, as well as to human preferences by directly integrating user feedback into its control logic. In the recent years, there has been an increase in popularity of the use of reinforcement learning for energy management applications, especially in fields such as electric vehicles, and HVAC control.

Figure 1. Articles using reinforcement learning by topic and year of publication.

We have reviewed all the literature about the use of reinforcement learning in urban energy systems and for demand response applications in the smart grid [1]. Our review shows that although many papers consider human comfort and satisfaction, most of them focus on single-agent systems with demand-independent electricity prices and a stationary environment. However, when electricity prices are modeled as demand-dependent variables, there is a risk of shifting the peak demand rather than shaving it. Therefore, there is a need to further explore the applicability of reinforcement learning in multi-agent systems, which can coordinate with each other to participate in demand response. We have observed that most of the studies are not easily reproducible, and so it is rather challenging to compare the performance of the controllers. Further standardization is needed in both the investigated control problems, and the used methods and simulation tools. We have proposed a basic framework to help in this standardization.

We have also introduced a new simulation environment that is the result of merging CitySim, a building energy simulator, and TensorFlow, a powerful machine learning library for deep learning. This new simulation environment has the potential for developing building energy scenarios in which machine learning algorithms, such as deep reinforcement learning, are applied to of the major problems and opportunities modern cities face, e.g., the increased demand for heating and cooling due to increasing populations [2].

This simulation environment allows to study model-free and self-tuning control algorithms, such as deep reinforcement learning (DRL) when integrating distributed renewable energy sources and storage devices into buildings. DRL can learn on-line and off-line from historical sensor data, and it can adapt to diverse changes in the system it controls on both the demand and the supply side. Its off-line learning feature allows it to be safely implemented with a back-up controller and operational constraints.

Figure 2. Simulation environment merging CitySim and TensorFlow.

To know more about this research, or ask us for any of these articles, here is the link to our project and our website.

References

[1] Vázquez-Canteli, J.R., and Nagy, Z., “Reinforcement Learning for Demand Response: A Review of algorithms and modeling techniques”, Applied Energy 235, 1072-1089, 2019 (published in a special section: Progress in Applied Energy – reserved to 3% of the articles).

[2] Vázquez-Canteli, J.R., Ulyanin, S., Kämpf J., and Nagy, Z., “Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities”, Sustainable Cities and Society, 2018.