NeSI is refreshing its technology platforms to ensure Aotearoa's national eResearch infrastructure and services are more powerful, flexible, and responsive to the needs of researchers.
Through a series of case studies, we're taking a deeper look at how the new platforms lift existing NeSI services and how researchers can benefit as a result.
Here we're looking at our new data storage platform and what it means for researchers working with their research data.
Researchers critically depend on data storage and data management capabilities to achieve their research ambitions. With our gaze on the Aotearoa and Australian research sector, NeSI is working with partners to build a macro view of research data in Aotearoa.
The early insights are eye-opening! As at December 2022 nearly 45 PB of first-copy research data was being stored across 14 New Zealand research institutions. Based on the growth rates we've seen historically, this would suggest ~70PB today.
We've seen this growth in demand and contention for storage resources on our platforms. NeSI hosts nearly 24 PB of research data across types of storage, supporting high-performance research workloads, research team collaborations and sharing, and long-term storage of valued research data.
NeSI’s scalable HPC and research developer cloud platforms support active researchers using computational sciences, analytics, and artificial intelligence. We’re improving our support for research data management across the lifecycle and investing in a wider range of storage capacities and profiles.
As part of our platform refresh, we're modernising our data storage platform. By collaborating with industry leaders WEKA, Xenon, and Versity we now have a richer data platform we can mix and match to meet the diverse needs of the Aotearoa research sector, and better support research data throughout its lifecycle, including:
Initially, we’re implementing services that our existing users are familiar with while gaining immediate performance and usability improvements.
At Waipapa Taumata Rau University of Auckland, Professor Russell Snell leads research that analyses genetic information of living New Zealanders. Read more about his research here.
"Apart from providing valuable computation resources, the security of the NeSI platform is a critical contribution to meet the human ethics requirements of our project. The NeSI platform and the support has been exemplary contributing to our work for over 10 years. This major compute resource underpins virtually all our work in both agriculture and human health. The technical specialists are excellent, responding quickly and accurately."
At NIWA, Dr. Rose Pearson is a remote sensing scientist using Digital Elevation Models to study flooding impacts. Read more about her work here.
"I had never used a supercomputer before... It's a real treat to be able to work with the highly skilled group of individuals at NeSI. They are responsive to queries or upskilling me so I can better make use of NeSI and its power the next time around."
Scientific collaborations are a source of radical innovation over the long term, driving the initial developments of all the core advanced technologies we rely on today including the internet, the web, and even email. HPC was one of these foundational technologies, driven by science.
Today, we continue to see innovation in a vast array of advanced technologies driven by science, and we’re now seeing an increasing proportion of the technologies we use in science emerge from the global ecosystem of public cloud providers, users, and developers.
A major driver of this shift has been the global scales of data underlying public cloud computing, and the necessity of embracing machine learning and other forms of artificial intelligence to manage these global-scale platforms.
After all, the global reach of scientific collaborations such as the Large Hadron Collider and the disruptive scales of their data analysis drove innovation in many of the earlier generations of advanced computing and data technologies.
At NeSI we’ve been embracing and investing in cloud-native approaches for the last few years, including into a sovereign research cloud and through harnessing the latest scientific software designed for data sciences and web-based collaborations.
We're now modernising our core HPC platform and our underlying data platforms by investing into a modular set of storage technologies. We’re looking forward to delivering on the promises of this powerful research data platform over the years ahead.
Our new High Performance Storage platform from WEKA offers the most advanced performance and scalability in the market today. It delivers high performance and accelerated storage for HPC & AI workloads, and allows high security for sensitive data analysis.
We can dedicate storage to specific research collaborations and institutes. We can tier out to cheaper cloud storage or across to the similar object storage on our own research cloud platform. This new WEKA storage platform replaces our current IBM General Parallel File System (GPFS) system.
“We are proud that New Zealand eScience Infrastructure (NeSI) has chosen WEKA as their data platform partner," says Joe Hassell, Sr. Director, APAC Sales, WEKA. "Our collaboration and partnership are set to modernise their high-performance computing storage environment, delivering extreme performance to enable faster time to discoveries and insights for researchers across New Zealand.
"We're excited to see how the WEKA Data Platform will accelerate the research and innovation powered by NeSI and look forward to a fruitful partnership, pushing the boundaries of what's possible with high-performance computing and AI.”
We’re all experiencing demand for a lift in capability in research data across research communities and institutions. NeSI's new data storage platform is designed to respond to this diversity of needs and expectations.
Freezer, powered by Versity, is our completely redesigned long-term storage service to support highly valued research data. It replaces our Nearline service with an easier-to-use and scalable platform that we can also extend over time, even to build dedicated spaces for research institutions and teams.
Freezer is tape-backed storage much like the “Glacier” storage services now common across public clouds, providing cost-effective data retention. Freezer is compatible with the common S3 cloud protocol and existing tools such as those used for accessing Amazon Web Services (AWS) S3 service.
“NeSI has chosen the new best-of-breed AI/HPC architecture for their core infrastructure," says Bruce Gilpin, CEO, Versity. "Pairing the Versity solution for managing large quantities of long term data with WEKA’s high performance storage delivers the best mix of value, capacity, and performance. We are delighted to be working on this cutting edge system for NeSI and the country of New Zealand.”
XENON have been providing HPC services and systems for 28 years and are providing the Versity solution.
“NeSI’s new Freezer brings XENON’s Versity deployments in ANZ to over 200PB in the last 12 months," says Dragan Dimitrovici, CEO, XENON. "We are excited to bring this powerful, robust solution to New Zealand’s research community as protecting and managing research data is critical to scientific practice.”
Ultimately, pairing different hardware and technology solutions means we can now be more responsive to researchers' data needs. With this new data storage platform now coming into service you can expect to see NeSI's range of storage service offerings evolve and expand. For example, we're considering how to diversify our allocation approaches.
We're also considering a dedicated mid-tier of lower performance storage catering for less active data that still needs to be always available online. We're exploring how complementary services, such as NeSI's Research Developer Cloud, can use services like our object storage directly for compute workloads. If any of these are of interest, we’re always looking to learn more about researchers’ needs.
We'll continue to share updates as our work on the platform refresh progresses. If you have questions, get in touch or join our online Office Hours to chat with our team.
