NeSI launches national scientific Data Transfer platform

NeSI’s national Data Transfer platform ⁠— operated in partnership with Globus, REANNZ and multiple NZ institutions ⁠— is now available for users to move data to and from NeSI quickly and easily.

Since 2014, NeSI has partnered with Globus to offer a high-speed option for transferring large and distributed data nationally and internationally. Over that time, NeSI has been working with New Zealand research institutions to facilitate data transfer to and from existing and new Globus Data Transfer Nodes (DTNs).

Last year, NeSI reviewed its data transfer offering and has now implemented a new and improved service designed for use with NeSI’s national HPC platforms.

NeSI data transfer nodes
Attribution: 

NeSI has plumbed Globus DTNs directly into its new infrastructure platform, enabling access to data on both Māui and Mahuika HPC systems hosted at NIWA, as well as at data storage and research facilities at AgResearch, the University of Auckland, and the University of Otago.

These DTNs act as an interface between Globus’ worldwide network of other endpoints.

“Bringing this new platform online has truly been a collaborative effort, involving collaborations with international partner Globus, national advanced network provider REANNZ, and several innovative research institutions across the country,” says Nick Jones, Director of NeSI.

It didn’t take long for the platform’s performance to be tested. In May 2018, tens of millions of files and hundreds of terabytes of data were moved over REANNZ's network as the first wave of NeSI users’ research data was migrated from the Pan cluster at the University of Auckland to the new Mahuika supercomputer in Wellington, hosted at NIWA. It was a record-setting moment for REANNZ and a testament to the enhanced capabilities and performance of NeSI’s new data transfer platform.

“We know that research today is increasingly data-intensive — NeSI is making it easier for New Zealand researchers to access, transfer, and share large and distributed datasets” says Jones. “Through Globus we’re enabling researchers to collaborate and approach science at new scales. With these datasets now more readily available, alongside our local partners we’re exploring opportunities to add value to this data through the data management features the Globus platform offers.”

Globus is particularly useful for users who need to move large data sets (more than a few gigabytes at a time). For example, New Zealand researchers working on Deep South Challenge earth system modelling projects have been among some of the data transfer platform’s early adopters, using it to access and share their large climate modelling data sets.

The platform’s potential as a national framework for data sharing and access also has applications in NeSI’s collaboration with Genomics Aotearoa, where discussions are underway to address broader issues in data management, custodianship, and access management requirements for Māori and other sensitive genomics data.

To begin using NeSI’s new data transfer platform, you simply a Globus account, an active NeSI account, and a web browser to begin using the service. Click here for detailed instructions on how to get started.

To learn more about whether NeSI’s new data transfer platform would be useful for your research, join our introductory Q&A webinar on Wednesday 22 May. Click here to register.

If you have questions or would like more information, visit our User Support Centre or contact support@nesi.org.nz.

 

What can I do with NeSI’s Data Transfer platform?

  • Move data, big or small:
    From kilobytes to terabytes, you can efficiently, reliably, and securely move your research data to and from Maui and Mahuika, or to any other endpoint in Globus’ worldwide network.

  • Share files with others:
    All you need is an email address, a NeSI account, and a web browser to share data with colleagues – Globus manages authentication and access.

  • Take advantage of Globus' data management features:
    Your data is transferred directly between the source and destination systems while Globus tunes performance parameters, maintains security, monitors progress, and validates correctness.

 

Who can use NeSI’s Data Transfer platform?

  • Institutions and research communities across New Zealand can use NeSI’s data transfer service to access, transfer, and share their data sets

  • If you’d like to set up a Globus Data Transfer Node (DTN) at your institution, contact support@nesi.org.nz.


 

 

Topic: 
Community