Automating workflow capabilities
"The workflow has enabled us to run long simulations without constant vigilance."
Subject:
Simulating black holes
"Alex and his team quickly pointed out optimisations that could be done to enhance the code further. The result is a more robust codebase that runs faster than before."
The below case study shares some of the technical details and outcomes of the scientific and HPC-focused programming support provided to a research project through NeSI’s Consultancy Service.
This service supports projects across a range of domains, with an aim to lift researchers’ productivity, efficiency, and skills in research computing. If you are interested to learn more or apply for Consultancy support, visit our Consultancy Service page.
Research background
In the theory of general relativity, the trajectories of particles follow geodesic lines in four dimensional (4D) space-time. In the absence of masses, this 4D space is rectilinear. However, large distortions, both in space and time, can occur in the presence of heavy masses such as black holes.
The warped structure of space-time can be obtained by solving Einstein’s equations, a set of nonlinear partial differential equations. The problem here involves solving the conformal field equations, a recasting of the Einstein equations to include points at infinity, as an initial boundary value problem.
Project challenges
A collaboration led by Professor Jörg Frauendiener at the University of Otago and Lecturer Chris Stevens at the University of Canterbury looks to dive a bit deeper into this space-time problem. Jörg, Chris and collaborators have developed a Python code, COFFEE, which applies finite differences in space and a Runge-Kutta method for time integration, combined with a pseudo-spectral method to compute angular derivatives. The code can take up to one day to run for the problem considered here, and, in order to accelerate the computations, the code can run in parallel using MPI and OpenMP.
What was done
As part of a Consultancy project, NeSI Research Software Engineer Alexander Pletzer and Application Support Specialist Albert Savary worked with the researchers on a number of tasks, including:
porting the model to NeSI platforms
applying memory checking tools to identify potential memory issues
upgrading the code to use the latest Python version and libraries
Main outcomes
With Jörg and Chris' code now running on NeSI's advanced computing platform, the researchers can study Bondi components, the angular momentum of the black hole, and find the Newman-Penrose constants. It also opens the door for future work to use a spinning black hole rather than a static one.
Attribution:
Figure: Speedup of the code on Mahuika's Milan processors. The best performance was obtained using 32 MPI tasks, eight CPUs per task and hyperthreading, yielding a speedup of 26x over the serial code.
Researcher feedback
"The NeSI consultancy was extremely useful for this project, where the code would work on local clusters but not on NeSI. This was originally then a specific NeSI problem, but Alex and his team quickly pointed out optimisations that could be done to enhance the code further. The result is a more robust codebase that runs faster than before."
- Chris Stevens, University of Canterbury
Do you want to bring your research to the next level? Send an email to support@nesi.org.nz.
