When asking for help leads to a 1300x speedup

A geographer has been able to dramatically reduce her processing times after asking for help from NeSI’s Computational Science team. For the Australian subset of the data, processing time was reduced from ~6.5 hours to 17 seconds.

This 1,376-fold speed increase was due to algorithmic changes suggested by Dr. Sung Bae. The researcher’s custom Python code used an inefficient search algorithm running largely in serial. Sung was able to work to improve both the search algorithm as well as the parallelism of the software. Those improvements compound to produce the impressive result.

Sung discusses what happened, “By optimising the search algorithm, we managed to reduce the search time significantly.” The original estimate for completion was at least three months. “This optimisation [and increasing parallelism] enabled us to complete the entire computation within 24 hours.”

This type of speedup leads to a much different research workflow. Not needing to wait for months for a research result enables the whole process of research to be conducted much faster. NeSI provides a unique combination of both excellent human support available and HPC platforms of international calibre.

What is the Computational Science team and what does it do?

The team works to boost research. Its members work collaboratively with research groups to find out where the bottleneck is and how to fix it. Some examples of what could be included in one of those collaborations:

• evaluating the performance of your software on each of NeSI’s HPC facilities
• assisting with parallelism, including making use of techniques such as MPI and OpenMP
• taking the time to make sure that you know what terms such as MPI and OpenMP mean
• improving I/O performance
• suggesting algorithmic changes
• implementing improvements in the programming language that you are familiar with

The team is actively looking for potential collaborations. This service is available to all researchers across the country who are accessing NeSI’s HPC Service via its Research allocation class.

The team is made up of members who have both strong research as well as computer science backgrounds. The team members’ profiles, including their disciplinary strengths, are available on NeSI’s website.

To explore the possibilities of collaborating, begin by talking to your project advisor. If you do not yet have a project with NeSI, email support@nesi.org.nz to let us know you are considering using our HPC Service and we will point you in the right direction. Asking for help is often the most difficult part of the process.