When getting back to basics is better than fancy new tools

Prof. Brendon Bradley of QuakeCORE (based at the University of Canterbury) has used a parallelised software, designed for 3D seismic wave propagation simulations on NeSI’s BlueGene/P supercomputer, to develop a sophisticated understanding of the effects of earthquake-induced ground shaking. This software, developed by US collaborator Rob Graves (US Geological Survey), enabled the team to simulate all the events in the Canterbury earthquake sequence. NeSI has been supporting this research from the early stages, and has been providing HPC and visualisation services to help develop an intuitive understanding of the model (read more about Prof. Bradley’s visualisation work here).

Prof. Bradley's team has recently completed the simulation of the Alpine Fault scenario using 8192 cores of BlueGene/P, and discovered that post-processing the output data was frustratingly slow, taking 5.5 days to complete (longer than the main computation itself). NeSI's algorithm specialist, Dr Sung Bae, embarked on an IO optimisation project to overcome this problem.

The problem has a historical basis – when Graves originally wrote the program in the ‘90s, MPI IO, a parallel IO library, was in its infancy, and he decided to opt for a conventional ‘file-per-process’ approach with each process written to a separate file. When 8192 parallel processes are used, 8192 output files are produced. This simple approach has its own merit, but requires an extra task of ‘merging’, so that the simulation can move on to the next step. This ‘merging’ was particularly slow with large-scale simulations, such as the Alpine Fault scenario.

Bae observed that each file has a large number of segments maintained in a very specific order, and ‘merging’ requires careful reassembly of segments from all individual output files following the implicit structure.

Attribution: 

Figure 1: Emphasising the complexity of merging when using the conventional ‘file-per-process’ approach.

Bae identified that the frequent jumps occurring during the reassembly were the main cause of the inefficiency, and was initially convinced that parallel file systems (GPFS) and parallel IO library would work wonders. However it was not to be and he was stuck with the lacklustre result. It was on a return to the basics that something struck Bae – "memory access is faster than disk”. He realised he had been blinded by the fancy tools, forgetting the basics.

Bae then devised an algorithm that reassembles a long contiguous segment on-the-fly in memory before writing to the disk. This approach eliminated the need for frequent jumps on the disk surface and showed a dramatic improvement. The new parallel solution using four POWER7 cores completes the merge in 12 minutes, which used to take 5.5 days – a 660-fold speedup. While it still took substantial work to materialise this idea, the solution, which yielded an outcome beyond expectations, relied primarily on simple programming concepts.

Prof. Bradley says, "The new approach that Sung Bae developed has been critical for our research workflow. We couldn't have continued with post-processing taking so long for the large computations, and it would have taken several months (if at all) for someone in our research team to learn the necessary MPI IO skills and develop a feasible solution. This is a classic example of the huge advances that are being achieved through our excellent collaboration with NeSI."

Prof. Bradley and QuakeCoRE accessed their compute hours and Dr Bae’s consulting support through a NeSI Merit Project. Merit Projects are designed to support high-quality, investigator-driven research which clearly aligns with New Zealand’s research priorities, providing free allocations of platform time and consulting, and high-priority status in the system. Learn more about how to access NeSI and its NeSI Merit Projects here.