War on wildings

The below case study shares some of the technical details and outcomes of the scientific and HPC-focused programming support provided to a research project through NeSI’s Consultancy Service.

This service supports projects across a range of domains, with an aim to lift researchers’ productivity, efficiency, and skills in research computing. If you are interested to learn more or apply for Consultancy support, visit our Consultancy Service page.

Research background

It has been estimated that over the next 30 years approximately 7.5M hectares of land in New Zealand will be at risk to be invaded by wilding conifers. It was also estimated that successful wilding control can therefore provide up to $6.3B of economic and environmental benefit and a large Wilding conifer control programme was initiated in 2016.

To understand wilding invasions better we needed to understand how wilding conifers spread across the landscape. As trees that rely on wind to spread their seeds, Scion and CSIRO developed a Flow Transport Model (FTM), an analytical model that enables seed dispersal modelling. The model includes the impact of the atmospheric turbulence caused by the complex terrain and vegetation canopy and it can be used to identify the spatial extent and distribution of the long-distance conifer seed dispersal from any chosen location. Such information is critical to provide additional information to the operational control and management programme and researchers so we can better manage existing infestations but also to plan ahead to identify areas that are vulnerable to conifer spread and manage them accordingly.

Project challenges

• The computational cost of running the simulations is quite high, in some cases taking a day or more to run a simulation, which can limit the number of scenarios that can be simulated

• FTM requires Python and some dependency python packages to be installed, which is not simple on all platforms

• The FTM user interface requires some knowledge of Python programming, which limits how accessible the code is to non-programmers

• Simulating a new region or making a simple change, such as wind direction, is also quite cumbersome and difficult to achieve without programming knowledge

What was done

NeSI Researcher Software Engineers Chris Scott and Alex Pletzer collaborated with Scion scientist Jiawei Zhang to improve the FTM model. NeSI focussed on improving the performance of the code and the user interface. At the beginning of the project, a continuous integration workflow was added that builds and tests the code, so we could be sure that other improvements made later didn’t affect the scientific accuracy of the code. Following this we profiled the code and optimised key routines using a combination of vectorisation and Numba (more details about Python optimisation techniques can be found in our performance optimisation training). We achieved a significant improvement, with performance dropping from 1 day to around 30 minutes for a realistic test case.

We also spent time improving the user interface, moving to a config file based approach to configuring the FTM model and also added a simple GUI (graphical user interface) as an alternative to the command line interface. Pyinstaller was used to build the FTM GUI program into a self contained Windows executable, which makes the program much more accessible to scientists. 

Meanwhile, Jiawei concentrated on improving the input module, making it much simpler to specify the region the simulation should run on and also allowing changes such as altering the wind direction to be made in a simple manner that doesn’t require programming knowledge. A general seed releasing module was also added to read and create line/point seed releases based on simple user input in the config file.

Main outcomes 

The main outcomes from the project included:

• Simulations running significantly faster (down from 1 day to 30 minutes for a real case)

• Refactored code, which should make it easier for new developers to contribute

• Reworked user interface, with two options:

  • Command line program with options specified in a config file

  • Prototype GUI program with options specified in a graphical user interface

• Added a generic input module that simplifies modelling different regions or wind directions as well as different seeding locations

• Built the program into a self-contained executable that can be easily distributed and doesn’t require the installation of other dependencies

• Added a continuous integration workflow that builds and tests the code automatically when changes are made and also builds the self contained executables

The improvements made during this project have already been put into operational use. The model has been used to study the seed dispersal in the Mt Hutt area to help provide useful information for an MPI wildings management operation. A cluster of 24 simulations were used in the study. Thanks to the optimisation, all these simulations were able to finish within 2 days which would have taken more than 3 weeks to finish previously.

Researcher Feedback

"Building a scientific model/theory to solve complex real world problems is one thing but scaling it up to solve the problems in a timely manner and utilizing all the modern computational resources is another. We have been working with our world-renowned collaborators to build the state-of-art modelling tools for the wildings control. When applying for the NeSI’s consultancy, we were at the stage where we had achieved our goal to build the model but was hoping to make it more efficient and user friendly. It was a great pleasure working with Chris and Alex from NeSI for this consultancy project. Chris, who was the main contributor to this consultancy project, has helped optimize the model with 20 times speed gain. He also went above and beyond to package the model and its runtime environment into executables with graphic user interface. I also enjoyed the experience working together with the consultancy team. Regular updates not only helped me understand all the technical details they have done but also made it possible for me to implement functionality modules into the model alongside them. With the optimisation and friendly user interface gained from this consultancy project, we are one step closer to bringing the tool to a wider community in New Zealand."

- Jiawei Zhang, Atmospheric Scientist, Scion

Do you want to bring your research to the next level? Send an email to support@nesi.org.nz.