Moving research data tools to the cloud to broaden access and impact

The below case study shares some of the technical details and outcomes of the scientific and HPC-focused programming support provided to a research project through NeSI’s Consultancy Service.

This service supports projects across a range of domains, with an aim to lift researchers’ productivity, efficiency, and skills in research computing. If you are interested to learn more or apply for Consultancy support, visit our Consultancy Service page.

Research background

With increasing numbers of extreme weather events, it is critical that communities understand and are managing land erosion. Alex Herzig (Senior Researcher - Geospatial Modeller) at Manaaki Whenua Landcare Research and colleagues use an in-house developed tool called LUMASS (the Land-Use Management Support System) to provide support for both land use impact assessment and spatial planning.

Some of the applications of LUMASS can be found on their website and include the Our Land and Water National Science Challenge, where LUMASS was applied in areas relating to achieving greenhouse gas emission targets and water quality objectives. LUMASS has also been used in the Smarter Targeting of Erosion Control (STEC) program, for example to model sediment concentrations in rivers.

One of the challenges around making LUMASS available to a wider audience is its build process, which is quite complex, requiring many dependencies to also be built and taking significant time and resources to complete. Currently Alex performs the builds himself on a local workstation but would prefer if the builds could be more automated and taken out of his hands, to improve the robustness of the process and allow him to focus more on the science.

NeSI’s Research Developer Cloud has recently been released to early-access users and Manaaki Whenua - Landcare Research’s (Manaaki Whenua) Head of Digital Strategy, Nick Spencer, is keen for Manaaki Whenua scientists to take advantage of this new platform. The NeSI cloud offers greater flexibility compared to their high performance computing platform, with the ability to choose your Linux distribution, the amount of compute power you need, easily install software and develop and manage custom interactive solutions, such as web-based dashboards that can be accessed from anywhere. However, the approach to using cloud-based platforms can be quite different compared to local resources (e.g. your laptop or workstation) and traditional research computing facilities (e.g. high performance computers) and so Manaaki Whenua are keen for their staff to get started early and build comfort on the new platform.

Project challenges

• A need to adopt different practices, such as devops and infrastructure-as-code, to make efficient use of cloud platforms can create a barrier to entry
• LUMASS has a complex build, with many dependencies, that will need to be automated in a robust manner
• The LUMASS build uses significant resources, running for over 45 minutes on an 8 core, 16 GB RAM machine and using many GBs of disk space

What was done

Through a Consultancy project, NeSI’s Research Software Engineer Chris Scott and Research DevOps Support Engineer Kahu Anderson, worked closely with the Manaaki Whenua team, including Alex Herzig, Nicoletta De Maio, Chris Wydra, Mike Cochrane and Nick Spencer, to build familiarity with NeSI’s cloud platform and develop a proof of concept for automating the LUMASS build.

A demo was developed to show how to provision a virtual machine (VM) on the research cloud in a programmatic manner, using terraform to provision the resources and ansible to configure the VM once it has been provisioned. The demo was walked through with MWLR research software engineers and DevOps team so they could familiarise themselves with this approach, thus enabling them to apply these approaches to other projects.

Following that, we investigated using Github Actions (GitHub’s continuous integration service) to automate the LUMASS build. Alex Herzig was able to provide a build script containing all the steps required to compile LUMASS and its dependencies, which greatly helped the process. The output of the build is an AppImage application (a portable, self-contained executable file that can be run on other Linux machines).

We then created a three step GitHub actions workflow:

• Use terraform to provision a VM on NeSI’s Research Developer Cloud and run an ephemeral self-hosted GitHub actions runner on it (this step runs a GitHub-hosted actions runner)
• Run the LUMASS build and upload the “AppImage” as an artefact of the build (this step runs on the self-hosted actions runner we created in step 1)
• Use terraform to destroy the resources we provisioned in step 1 (this step runs on a GitHub-hosted actions runner)

With this approach, we can provision a VM with enough resources to perform the LUMASS build and the VM will only be provisioned when it is needed and destroyed immediately after it has been used, so no resources are wasted. A blueprint showing how this could be applied to other projects can be found here.

Main outcomes 

• Built familiarity with NeSI’s cloud platform and related infrastructure-as-code tools among Manaaki Whenua research software engineers, enabling them to apply these approaches to other projects on the NeSI cloud and also to continue the LUMASS work

• Developed a proof of concept showing how GitHub Actions and NeSI’s research cloud platform could be used to automate the LUMASS build process; using this approach LUMASS builds can be initiated at the click of a button in the GitHub web interface and run seamlessly on NeSI cloud infrastructure, significantly reducing the amount of effort involved in building LUMASS

• Created a blueprint showing how a similar approach could be used in other projects

Researcher feedback

"The automated build process promises to greatly simplify the production of new releases including artifact management. It will provide a more stable build environment and reduce the reliance on my personal laptop for building LUMASS. Last, not least, It saves valuable time I can use for science."

- Dr Alex Herzig, Senior Researcher - Geospatial Modeller, Manaaki Whenua Landcare Research

“Manaaki Whenua have been pleased to have had the opportunity to begin understanding how these types of remote platforms can broaden the type of compute we can offer our researchers. The NeSI team has been excellent to collaborate with and both teams. This proof of concept provided the opportunity to learn more about researcher needs when migrating to these types of compute platform.”

- Nick Spencer, Head of Digital Strategy, Manaaki Whenua Landcare Research

The Manaaki Whenua team are looking forward to putting the process into practice and to start offering this LUMASS platform more widely. 

If you are interested in trying out NeSI’s cloud platform, documentation, including information about how to get started, can be found here: https://support.cloud.nesi.org.nz/

Do you want to bring your research to the next level? We can help. Send an email to support@nesi.org.nz to learn more about our Consultancy support or visit our Consultancy Service page.