Combining physics and artificial intelligence for high-resolution climate projections and sub-seasonal-to-seasonal forecasts

NIWA Data Scientist Neelesh Rampal is building a deep learning model that enhances the resolution of climate projections and weather forecasts. Running his model on two different clusters – Mahuika and Māui – meant learning to use an HPC platform job manager and new data upscaling techniques. NeSI team members provided both the digital skills training and technical troubleshooting support Neelesh needed to see his model become a reality.

Currently, New Zealand is experiencing combined damages of $840M per decade from floods and droughts alone [2] – which is expected to increase under climate change [1,2,3,4]

The effects of anthropogenic climate change will become more severe over the coming decades. New Zealand needs to adapt to and mitigate the effects of climate change to build a climate-resilient economy. To deliver the full information needed for societal decision-making, high spatial resolution climate projections are required.

Neelesh Rampal is a Data Scientist at the National Institute of Water and Atmospheric Research (NIWA). He’s working alongside with Meteorologists Tristan Meyers and Ben and Climate Scientist, Dr Peter Gibson to build a new deep-learning model that enhances the spatial resolution of climate projections and weather forecasts. Deep learning models are significantly adding value to the coarser resolution climate projections and weather forecasts [5] and is provided improved decision making utility.

Using models to inform decision-making

General Circulation Models (GCMs) are sophisticated physics-based models for simulating the response of global climate to increasing greenhouse gas emissions. Current GCMs are computationally expensive. This makes running high spatial resolution ensembles impossible, even on the world’s largest supercomputers. A similar issue also arises for sub-seasonal-to-seasonal (S2S) forecasts, which provide forecasts at time-scales from 2 weeks to 2 months. The resolution limits of existing GCMs are around 100 km and S2S forecasts are 50km This resolution is too coarse to deliver the information needed for societal level decision making at local and regional scales decision-making for weather and climate effects.

NIWA Scientists developed a deep learning model that learns how to reconstruct high-resolution rainfall observations from coarse resolution weather patterns [5]. This model is then applied to coarse resolution GCM outputs to produce high-resolution climate projections. NIWA Forecasters will also use this deep learning model to produce high spatial resolution of S2S drought forecasts. Neelesh used NIWA’s climate data archives to train the deep-learning model. Applying this deep learning model to climate projections and weather forecasts will require petabytes of data processing, which is only possible through automation and use of HPC infrastructure.

“The computational speed of deep learning is at least 1,000 times faster than the physics-based models currently used", says Neelesh.

Expanding capability through HPC

NeSI provided Neelesh with the opportunity to scale his current training operation to vastly larger archives of data, which was initially a barrier. He worked alongisde NeSI team members to build the automation pipeline, as his existing code could not handle this large amount of data. NeSI Data Science Engineer Maxime Rio and Application Support Specialist Wes Harrell helped Neelesh by identifying bottlenecks in his code and guiding him in parallel process problem solving.

“We were previously limited to under 100 GB of training data, which is why we came to NeSI. We needed a way to train models on vastly larger datasets. We needed to know alternative ways to go beyond memory limitations, but it hadn’t been done with climate data.”

Neelesh used both NeSI’s Māui and Mahuika platforms. Swapping between the platforms was necessary for his model training, but it presented unique challenges. Mahuika has several partitions that allow jobs with a range of CPU and RAM requirements. Meanwhile, Māui is a single-partition HPC. This meant the code had to recognise and adapt to the platform it was currently working on.

“Maxime and Wes enabled me to advance my design and automation workflows on the platforms. The advantage of learning how to create workflows on NeSI HPC, was we could create reproducible science. This enabled us to provide better collaboration amongst scientists.”

Neelesh worked with Maxime to incorporate the Slurm Workload Manager. Slurm is NeSI’s job scheduler and was responsible for assigning jobs to both Mahuika and Māui when Neelesh ran his model. While Neelesh was aware of Slurm, he was not sure how to best incorporate his jobs into the manager.

“A lot of learning was trial and error, but it was great to have the support to help us set up scripts. Maxime worked with us when we had questions on how to load Slurm jobs in the most efficient way.”

The result was a Python package for training deep-learning models on extremely large climate datasets. This improved the predictive ability of the model and overcame the hard limit that earlier models had on training data sizes.

Insights enabling better decision-making

There are significant implications for this research, and Neelesh's model can help create high-resolution climate projections and enable better societal decision-making in a climate change context.

These methodologies have international significance and can be leveraged in other nations as well, particularly in developing countries who cannot afford the computational cost of physics-based models.  

References:

[2] Frame, D., Rosier, S., Carey‐Smith, T., Harrington, L., Dean, S., & Noy, I. (2018). Estimating financial costs of climate change in New Zealand. New Zealand Treasury Commissioned Report. Access link: https://treasury.govt.nz/sites/default/files/2018‐08/LSF‐estimating‐financial‐cost‐of‐climatechange‐in‐nz.pdf.

[1] Bell, R. J. S. P., Lawrence, J., Allan, S., Blackett, P., Stephens, S., Hannah, J., ... & Davies, K. (2017). Coastal hazards and climate change: Guidance for local government. Ministry for the Environment.  

[3] Paulik, R., Stephens, S. A., Wadhwa, S., Bell, R., Popovich, B., & Robinson, B. (2019). Coastal flooding exposure under future sea‐level rise for New Zealand. NIWA Client Report 2019119WN, prepared for The Deep South Science Challenge, 2019.  

[4] Simonson, T., & Hall, G. (2019). Vulnerable: the quantum of local government infrastructure exposed to sea level rise. LGNZ Te Kāhui Kaunihera ō Aotearoa.

[5] Rampal, N., Gibson, P. B., Sood, A., Stuart, S., Fauchereau, N. C., Brandolino, C., ... & Meyers, T. (2022). High-resolution downscaling with interpretable deep learning: Rainfall extremes over New Zealand. Weather and Climate Extremes, 100525.

Do you have an example of how NeSI support or platforms have supported your work? We’re always looking for projects to feature as a case study. Get in touch by emailing support@nesi.org.nz.