Forecasting renewable power generation should not be that difficult, right?

Daniel Enthoven, Short Term Trade Desk Scientist

“It’s no rocket science“ is stated in an article I read about modelling the energy output of a solar asset. So how correct is this statement? Therefore, I would like to quote a well-known phrase - “The world isn’t black and white. It’s all grey“. A multi-perspective view must be applied to this statement, which triggered me to write this brief article.

This article introduces you to the current energy market dynamics and challenges. Furthermore, it aims to illustrate why forecasting renewable power generation is a very exciting and dynamic development field. Finally, I hope this will inspire and motivate others and spark interesting discussions.

The Renewable Energy Boom is here

EU wants to accelerate the take-up of renewables to contribute and reach the goal of reducing net greenhouse gas emissions by at least 55% by 2030. The energy sector is responsible for over 75% of the EU’s greenhouse gas emissions.

Increasing the share of renewable energy across the different sectors of the economy is a fundamental building block to reaching the EU’s energy and climate objectives [1]. The future promises further growth of the renewable energy industry, potentially aided by supportive policies from an administration focused on combatting climate change.

Volatility, high commodity prices, and renewable energy

The power price is determined by the “merit order” – the sequence in which power stations contribute power to the market. The merit order is based on the lowest marginal costs. The cheapest offer made by the power station with the smallest running costs setting the starting point of the merit order and is the first to be called upon to supply power, see Figure 1.

Power from renewable installations such as wind turbines and photovoltaic installations is bid in assuming zero marginal cost. As a result, these renewable assets push more expensive conventional power plants further up the merit order and lower the entrance price.

Why do we have such high energy prices and volatility in the energy market today when renewable energy lowers the entrance price? Firstly, energy prices today are still often set by conventional assets. Commodity prices (e.g. gas, coal, and carbon certificates) mainly determine the marginal costs of these assets. Therefore, the recent extreme increase in commodity prices resulted in higher power prices bid in by conventional power plants. Secondly, renewable energy production volumes are more volatile to e.g. changing weather conditions. Price volatility kicks in due to the price differences of the different producers (y-axis) and the volatile share of renewable power generation volumes bid into the market (x-axis).

With the growth of renewable installed capacity the need for precise predictions of how much power will be fed into the grid at any given moment is becoming increasingly important. Accurate weather forecasts provide the basis but grid operators, energy traders, and renewable power plant operators themselves now rely on a new industry specializing in predictions tailored to the sector.

It’s no rocket science

After this introduction, let us return to the statement ”It’s no rocket science”. Yes, it is no rocket science; in the sense that we can use physics to derive or measure a wind power curve or a solar power curve. The higher the windspeed or radiation the more power generation of your renewable asset. Of course, there are some constraints, but understanding the general relationship is no black box.

The inherent forecasting inaccuracy of an individual asset (for asset operators) or a specific region (for grid operators) might be a manageable risk, however specifically for the trading sector, the stakes are much higher. The imbalances as a result of forecasting inaccuracy will be strongly correlated with the renewable assets in that same balancing area. Correspondingly market prices will also correlate to these inaccuracies. This makes it very hard to manage as all market parties are having similar needs. That's why it becomes rocket science because the devil is in the details. It is a challenge to out-compete in terms of accuracy, reliability, and speed.

The name of the game

Machine learning is a hot topic and changing the world around us. Is it the key to our predictability challenge? Let’s finish this article with some domains that I and my colleagues are involved in, among other things.

Machine Learning

Machine Learning can add value to business questions, but we need to keep in mind that it is just math and therefore has its constraints. If you directly throw everything at the model and see what looks useful, your model will likely wind up overly complicated, expensive, and filled with unimportant features. If you try lots of features without a hypothesis, you will falsely believe these are relevant signals for your model [3].

Machine Learning is an important domain for increasing our predictability. But if we do not get the desired results, we sometimes benefit from looking further than just the kind of model.

Representativeness of your Data

Regardless of what we'd like to predict, data quality and representativeness are critical components for an accurate forecast. We can increase our predictive power with advanced machine learning set-ups, but in general, the representativeness of our data can suppress the locked value of our potential predictability increase.

Time horizons of forecasts

The forecast horizon is the length of time into the future for which forecasts are to be prepared. To optimize the usable available data flows we need to separate different time horizons. For example, the representativeness of certain features is only trustworthy enough at the ultra-short-term horizon (< several hours ahead). For a few days ahead you don’t want to overfit your model and be more generic.

Infrastructure

We live in an unbelievably rich ecosystem for creating present-day web applications. By contrast, modern Machine Learning doesn’t yet have that same ecosystem. This gives rise to the concept of MLOps, which follows a similar pattern to DevOps.

A last note

Hopefully, I have been able to describe in a concise narrative way why renewable power generation forecasting is an exciting field of research where innovation is essential in different areas.

Hopefully, you are just as enthusiastic as me and want to add value to the renewable energy boom. It’s not a one-man job!

References

[1]: Renewable energy targets (europa.eu)

[2]: https://finance.yahoo.com/news/more-volatility-expected-in-energy-sector-amid-transition-ing

[3]: https://developers.google.com/machine-learning/problem-framing/good

[4]: MLOps: Continuous delivery and automation pipelines in machine learning