Responding to evolving research needs

Attribution: 

Each year NeSI publishes an Annual Review to celebrate the work we do,

the people we work with, and the groundwork we're laying for future directions. 

Below is an excerpt from our 2023 Annual Review. 

 

NeSI’s national infrastructure responds to an ever-evolving eResearch landscape. Our platforms deliver a mix of traditional HPC offerings alongside new programmable infrastructure for advanced research solutions.

Supporting our core HPC user base 

Following an early access phase in early 2023, the Mahuika Extension (standard compute and high memory Milan nodes) opened to all NeSI users. Work is ongoing to ensure that all applications currently running on the original Mahuika Broadwell nodes can be run easily on the extension Milan nodes with a difference being the suite of compilers available on the new technology. We continue to manage a consistently high level of demand for storage resources.

Throughout 2023, we honed our user education efforts and operational tactics to ensure researchers' data on our systems is stored in the appropriate space (i.e. inactive data no longer on the highest performance media) to relieve pressure on storage capacity. NeSI’s Research Support is critical to the uptake of our computer services and is consistently acknowledged as central to enabling our users' success with our services.

Attribution: 

Programmable infrastructure for research data collaboration 

Flexible and high performance, NeSI's research cloud platform remains a significant area of development and opportunity for NeSI. Flexible HPC addresses needs beyond queue-based workloads, enabling NeSI and partners to collaborate and build solutions that are interactive, long running, and secure. The tenancy level isolation brings enhanced security, and allows tenants to build their own services for their wider research communities.

Flexible HPC tenancies in 2023:

• AgResearch’s institutional eResearch platform – More than 1PB of data was migrated into the AgResearch tenancy’s new primary eResearch Infrastructure data-store. We continue to welcome new researchers onto the platform, where a suite of software is available to use, supported by NeSI's Researcher Support Team. We completed significant upgrades for some key underlying components, showing the value in the containerised platform management of partner StackHPC. 
   
• Aotearoa Genomic Data Repository (AGDR) - Continued support and development of AGDR has enabled the project team to improve data ingestion processes. In 2023, 10 new data sets for taonga species were added to the system (see our Data Services section).
   
• Rakeiora Genomics Platform - Following a risk and vulnerability assessment through NeSI’s partnership with the University of Auckland Information Security team, the Rakeiora platform was approved for participant data and workflows. This is part of a layered security capability assuring the suitability of both NeSI’s hosting platform and any tenant for production research. As a final step in moving Rakeiora to NeSI’s Research Developer Cloud platform, this was a significant milestone in being ready to use primary and tertiary care data.
   
• Strong AI Lab (SAIL), University of Auckland – SAIL hosts their own infrastructure in the Flexible HPC platform. They work at the intersection of machine learning, reasoning and natural language understanding, with a focus on maximizing the near-term benefit of AI to entrepreneurs and business. We meet with the SAIL group regularly to stay attuned to their needs, from targeted training to exploring approaches that could support a broader set of AI researchers nationally. In 2023 they continued to make good use of their GPU investment hosted on the platform.

Attribution: 

Evolving services to embrace cloud-native approaches

In September 2023, NeSI launched its Research Developer Cloud, a suite of cloud-native services targeted at research groups building complex or unique solutions to computational challenges. Built atop our Flexible HPC infrastructure, the Research Developer Cloud’s first phase delivered basic building blocks for a development environment. Following the launch, a small group of pilot projects from Manaaki Whenua – Landcare Research, GNS, and AUGMENT (Auckland Bioengineering Institute, University of Auckland) were onboarded.

They are helping us identify user needs and make continuous improvements to the service before we widen access. All of these projects have commonalities around the themes of machine learning (ML), pipeline development for ML, and back-end development of applications.

The Research Developer Cloud is seen as the enabler for these projects to improve on their existing workflows via automation and DevOps. We’ve also been moving some of NeSI’s own services into the Research Developer Cloud space. A small cross-functional group of NeSI’s Support and Consultancy teams built a prototype cloud-based training environment. It is scalable, portable, and is one of our first exemplar blueprints showcasing automated deployment and configuration into the Research Developer Cloud.

At the 2023 eResearch Australasia Conference, we shared reflections and lessons learned from rolling out this new service with a developer end-user focus. During that event we also made early connections with the Australian Research Container Orchestration Service (ARCOS), who are building knowledge around Kubernetes and containers in the Australian research sector. We will continue to build that relationship in 2024.

Attribution: 

Logging into OnDemand on NeSI, our new interactive computing environment.

These are just some of the many activities and partnerships that kept us busy in 2023.
Click here to read more from our 2023 Annual Review.

If you'd like to receive a printed copy of our review, get in touch.