Modelling water quality in catchment of the Hauraki Gulf
"A welcome aspect was the very patient supportive approach of the consultants in training novice NeSI users to actually learn and use NeSI..."
Subject:
Picture perfect HPC image analysis
Jingli Lu needed High Performance Computing (HPC) performance for hyperspectral image analysis. She had some HPC knowledge but when problems occurred, NeSI team members jumped in to help integrate new tools and troubleshoot her experience.
AgResearch creates new technologies for New Zealand’s agricultural sector to discover new products and improve land use. It also exports these goods and knowledge, making it one of New Zealand’s biggest exporters. Jingli Lu is an AgResearch software engineer who uses hyperspectral imaging to augment ground forage assessment for agricultural use.
Red, green, and blue (RGB) imaging is the common method for building images, assigning red, green or blue colouring to each pixel image. Hyperspectral images improve the image detail by breaking down the light of each pixel into a wide range of spectral bands, including visible, ultra-violent and infrared light and assigning these to each pixel. This can help identify objects based on the specific wavelengths of light they emit.
“Each hyperspectral image pixel identifies several hundred bands of light. Images can range from 300-900 megabytes,” said Jingli.
Jingli needed to collate many hyperspectral images to understand field ground cover. Different plant species cover livestock fields, but the traditional ways of finding the ratio of species require harvesting or monitoring by hand. These are time-consuming and inaccurate ways to understand ground cover. Automation that compares plant images to known structures breaks down the bottleneck. It allows researchers to analyse ground cover without the intensive labour involved.
Unlocking the capabilities of computational power
Jingli was familiar with the capabilities afforded through computational approaches from past experience using AgResearch’s local infrastructure. In partnership with NeSI, AgResearch is in the process of expanding its infrastructure even further with a new future-focused platform that will enable new styles of access and use of advanced research computing.
In the meantime, Jingli's need to analyse hyperspectral images using both spectral and spatial information demanded computational resources at a more powerful scale. That is why she approached NeSI to use its national platform of HPC resources and computational expertise for image analysis.
Moving to NeSI's shared HPC environment meant becoming familiar with new tools and adapting to a different infrastructure. Jingli said the NeSI user resource guide and support through NeSI's Consultancy service was invaluable for transitioning to the new platform.
“I found the user guide quite good, it has almost everything I needed. But the user guide is a long document, so it could be hard to find exactly what I’m looking for. Talking to the NeSI team members, they were able to guide me to the right pages and get my problems solved very easily.”
Tapping into NeSI expertise
AgResearch had a program to analyse RGB images, which Jingli wanted to convert to hyperspectral analysis. The software was in a Docker container, which Jingli was unfamiliar with. Containers are considered best practice for reproducibility and use on HPC architecture. They allow users to package code with all program dependencies and versions, for use on different systems. Jingli worked with NeSI Data Science Engineer, Maxime Rio to change the container format.
“I needed to change the Docker container to Singularity for use on the NeSI supercomputer. Maxime built a new container by definition for me to run on Mahuika.”
Getting started on a new-to-you HPC facility can be hard, but NeSI team members are available to help along the way, whether through workshops, online office hours, documentation or one-on-one consultations with our HPC experts (no question is too small or big!). NeSI's aim is to make the 'getting started' process as easy as possible.
As a relatively novice HPC user, Jingli called on NeSI team members to help her navigate the transition to working in a shared supercomputing environment.
“When you get stuck, ask for help. NeSI has a very professional and knowledgeable team. It is also useful to have a basic Linux command line knowledge and to get used to scheduling jobs,” she said.
Getting started with image analysis on NeSI
If you are interested in running image analysis on NeSI, or learning more about this and other data science capabilities on NeSI systems, check out these key links to help you get started:
Machine Learning 102 (Image analysis on NeSI) workshop
(email training@nesi.org.nz to inquire when our next session will happen)
If you have questions about NeSI, you can also email us at any time to speak to a member of our team.
