Optimising simulations to improve electronic materials design

The below case study shares some of the technical details and outcomes of the scientific and HPC-focused programming support provided to a research project through NeSI’s Consultancy Service.

This service supports projects across a range of domains, with an aim to lift researchers’ productivity, efficiency, and skills in research computing. If you are interested to learn more or apply for Consultancy support, visit our Consultancy Service page.

Research background

At the University of Auckland, as part of Dr Nicola Gaston's research group, PhD student Emma Vincent is investigating ways to create computing hardware that is more energy-efficient. She has been studying the emergent electronic properties of cluster assemblies, with a view to developing simulation methods that incorporate models of structural disorder in explaining the electron transport behaviour. Her work is part of a MacDiarmid Institute for Advanced Materials and Nanotechnology research programme, Hardware for Future Computing. 

The aim of Emma's project is to simulate the mechanics of atoms using a Kinetic Monte-Carlo (KMC) method. A standard workflow consists of running two python programmes - one which generates the crystal lattice, and a second which calculates the trajectories of atoms. The lattice may be hundreds or thousands of atoms in size. Usually there are 100 steps per trajectory and between 10,000 and 100,000 trajectories depending on the magnitude of the rates. For smaller average rates it takes more sampling for trends to emerge. 

Project challenges

As the researchers' simulations increased in size, it was becoming prohibitively difficult to complete work in a timely manner. Because trajectories are completely independent, the problem scales very well by running each simulation as its own job. There were some challenges in recombining the data into a single output file and integrating the code with the Cylc workflow engine. 

What was done

NeSI Research Software Engineer Alexander Pletzer and NeSI Researcher Support Analyst John Whiting modified the KMC code to take command line arguments. This allowed Alex and John to then write Cylc configuration files that can submit and track the status of multiple concurrent trajectory and voltage computations.

The result is a much reduced turn-around wall clock time for large simulations.

Main outcomes 

• Alex and John helped Emma and her collagues introduce good software development practices. These included:
  • Git source version control to ensure multiple developers can contribute to the code while preserving the history of the code development
  • Unit testing to ensure that code changes don’t break the software
  • Continuous integration builds on top of unit testing by running the test battery each time a developer pushes changes
• Introduced a more efficient file format for storing the trajectories
• The computation of many trajectories was split into Slurm jobs whose submissions are orchestrated by the Cylc workflow engine. The picture below shows an example where 600 trajectories were computed in slightly more than a minute, representing a 44x speedup.
• Visualisation of the trajectories

Below are the trajectories re-assembled from the 60 jobs submitted to the Mahuika platform.

Attribution: 

Researcher feedback

"I am grateful for the NeSI team's assistance - not only did they save me months of work by optimising a parallel implementation of my simulations for their supercomputers, but they also developed unit tests to ensure the code's functionality is maintained. With their support, managing data generation using a Cylc-based approach has been both swift and seamless. Additionally, they helped integrate GitHub repositories into our workflow for efficient version control."

- Emma Vincent, Doctoral Candidate, Faculty of Science, Physics, University of Auckland

Do you want to bring your research to the next level? We can help. Send an email to support@nesi.org.nz to learn more about our Consultancy support or visit our Consultancy Service page.