Automating workflow capabilities

The below case study shares some of the technical details and outcomes of the scientific and HPC-focused programming support provided to a research project through NeSI’s Consultancy Service.

This service supports projects across a range of domains, with an aim to lift researchers’ productivity, efficiency, and skills in research computing. If you are interested to learn more or apply for Consultancy support, visit our Consultancy Service page.

Research background

Alena Malyarenko and Alexandra Gossart are research fellows at the Modelling Hub, Victoria University of Wellington, where they contribute to the Antarctic Science Platform MBIE funded project by simulating past and future Antarctic climate scenarios. Alena and Alex have developed a regional climate model, which couples atmosphere, ocean and sea ice, in particular to investigate how wind and air temperature impact on ocean salinity and vice-versa.

Attribution: 

The figure above shows how wind and ocean currents are coupled. Image provided by Alena Malyarenko and Alexandra Gossart, Victoria University of Wellington

Project challenges

Alena and Alex want to run their coupled model for five years at the end of the 20th century. Given that one month of simulation time takes about one day to run on the Māui supercomputer (480 processes), their simulation is expected to require several months of continuous execution time, which is much longer than the maximum wall clock time limit (24h) available on NeSI’s Māui platform. 

To overcome this limitation, the researchers have configured their code to periodically write checkpoint files, which allow the model to be restarted after being killed. The researchers anticipate that the model will need to be manually restarted about 60 times.
 

What was done

Research Software Engineer Alexander Pletzer and Data Science Engineer Maxime Rio have written a Cylc configuration file that automatically resubmits the job as often as is necessary to complete the simulation. The diagram below shows three cycles of the workflow – Cylc allows an infinite number of cycles to be executed.

Attribution: 

Main outcomes 

Faster results and a more efficient use of NeSI’s high performance platforms. 
 

Researcher feedback

"The workflow has enabled us to run long simulations without constant vigilance. We really appreciate the work that the NeSI team put into our model infrastructure." 

- Alena Malyarenko and Alexandra Gossart, Victoria University of Wellington

Do you want to bring your research to the next level? Send an email to support@nesi.org.nz.