Data & Code

I share data and code with my publications wherever possible. See links in specific papers for details and contact me if you have any problems. Here is a list of more general resources that might be useful.

Data

HEFTcom2024: data from the Hybrid Energy Forecasting and Trading Competition 2024 is available on the Zenodo. It includes generation from Hornsea 1 wind farm, solar production from east England, various field from the GB electricity market, historic weather forecast from two providers, competition submissions from all participants and more! A paper describing the competition is available here.

Control REACT: the outputs from this innovation project (2019-2021, with NGESO) include an extensive dataset of wind, solar and net-demand data for Great Britain, and code implementing state-of-the-art probabilistic forecasting methods using ProbCast. Available on Zenodo: Data & Code

ECMWF: Historic weather forecasts from ECMWF are now CC-BY and accessible via the MARS archive (free for academics in member countries)! Many datasets, including some operational products, are also freely available via web interface or API.

dynamical.org: are trying to make NWP data more accessible. Check it out!

GB Electricity Data: Metered data from all transmission connected and some embedded generators is available for research at 30-minute resolution from Elexon. Good knowledge of the BM is required to make sense of this data - take care! Increasingly, energy networks and operators in GB are making data available via their own data portals too.

Monash Time Series Repository: includes many time series including energy demand and wind power generation at high temporal resolution, details here.

SERL: The Smart Energy Research Lab is collecting smart meter data from thousands of GB consumers as well as survey responses from participating households.

NEED: The National Energy Efficiency Data-Framework (NEED) provides annual energy consumption for around 4 million households with accompanying metadata.

MIDAS and Historic UK Met Office Forecasts: I have worked extensively with the these dataset provided the British Atmospheric Data Centre which provides meteorological measurements from weather stations around the UK. This data is free to use for academic purposes.

FINO: The FINO project comprises three offshore platforms collecting atmospheric and oceanographic data, including wind speed and wave height measurements.

Marine Data Exchange:Data relating to UK offshore wind projects. Highlights include 30 year wind resource model outputs produced by the UK Met Office.

Wiki-Energy: A smart grid project in the US including domestic solar generation and electricity usage as recorded by smart meters installed in hundreds of homes.

Global Energy Forecasting Competitions 2012 & 2014: Data from the GEFCom2012 and GEFCom2014 is quality controlled and lots of state-of-the-art competitors to test yourself against.

Code

bulktrends is currently under development in collaboration with Defra, and available on GitHub. It is designed to monitor, forecast and hindcast UK import flows using large-scale trade datasets to support evidence-based monitoring and analysis of UK trade patterns. The central interest is to monitor and evaluate for impacts on Sanitary and Phytosanitary (SPS) import dynamics over time. The key objectives include producing robust forecasts with measures of uncertainty, identifying and analysing outlier events and anomalies in the time series, comparing traditional and more novel time series techniques when applied to large trade datasets used for monitoring and providing robust diagnostic tools to identify the optimal methods to be used in this case.

ProbCast is an R package created by myself and colleagues at the University of Strathclyde. It isn’t under active development at the moment and could do with an overhaul… It is a collection of functions for probabilistic forecasting (mainly wrappers for quantile and semi-parametric regression model fitting functions), cross-validation, evaluation and visualisation. Central to ProbCast is the data class MultiQR, for storing the results of multiple quantile regression, and methods for working with MultiQR objects. You can read more about the first release of ProbCast here, and access the package on GitHub.