NeSI exploring cloud-native services for research developers
As more domains integrate computational and data-focused approaches to the way they work, we're seeing research communities' needs evolve beyond traditional models of queue-based High Performance Computing (HPC).
NeSI has responded by investing in infrastructure and services that support more diverse capabilities and interactive approaches to computing. Our latest offering is NeSI's Research Developer Cloud, a suite of cloud-native services targeted at research groups building complex or unique solutions to computational challenges.
Launched at the 2023 Research Software Engineering Conference in September and built atop our Flexible HPC infrastructure, the Reserach Developer Cloud enables easier and more adaptable collaboration around research data.
Lifting research capabilities and collaboration
The Research Devleoper Cloud's initial offerings range from compute resources to identity management, and all are programmable via public API. See the full feature list here.
Developers can build solutions for researchers that are scalable, portable, and interoperable. Research communities seeking a locally owned and sovereign cloud computing platform can develop tools and platforms that support their research data collaborations.
As more groups begin using the Research Developer Cloud, we will share blueprints of solutions created so that others can learn from them and apply similar approaches to their projects or build upon the solutions developed to deliver expanded benefits.
DevOps in research
DevOps is a set of practices and principles you can apply to your development cycle. It embraces iterative development, collaboration, visibility, and rapid delivery of solutions.
In the coding stage, you might use tools like Github or GitLab for version control and to develop solutions collaboratively. Interally, NeSI has been using Github actions to trigger a continuous integration / continuous delivery deployement cycle, which enables us to regularly build, test, and deploy updates to the Research Developer Cloud.
We're also using tools like Terraform and Ansible to interact with the platform through code and automate our deployment cycle steps. Along the way, we're building our knowledge in Kubernetes and other tools to operate, montior, and scale the services that we develop, channelling our learnings back into our planning, design, and rapid development approaches.
We want to work with you to strengthen Aotearoa's research developer ecosystem and roll out future cloud services that are tailored to meet research DevOps needs. Want to be part of our journey? Let's talk.
Security as a shared responsibility
Within NeSI’s cloud-native infrastructure, security is a shared responsibility and considered at all phases of development and deployment.
Shared Responsibility emphasises the kaitiakitanga, the collaboration, and the coordination essential to its maintenance and to protecting the research communities it enables. This shared way of working is becoming more common, yet there is plenty to learn.
We’re advancing our security posture, policies, processes, and practices through continual improvement and development. Click here to learn more about this approach.
Partnership-led approaches
NeSI is building valuable expertise through collaborations with our inaugural platform tenants:
- Aotearoa Genomic Data Repository - As a national data repository for genome sequencing data of taonga species, the Aotearoa Genomic Data Repository wanted to host its data onshore, integrate FAIR principles (Findable, Accessible, Interoperable, and Reusable), and embed kaitiakitanga into its data access processes. The solution we've built incorproates Gen3 (Open source genomic domain data repository solution, University of Chicago), Kubernetes (for platform management, health and scaling), Google Forms / Zendesk (for application process).
Read more about this partnership...
- AgResearch - This has been a larger, institutional level collaboration to co-design and deploy an eResearch Infrastructure platform for AgResearch. Among the tools and services we've co-designed and deployed as part of this partnership, we've built expertise in using Keycloak (for identity management), Open OnDemand (an interface layer for end users to access various services), and Slurm appliance (enabling deployment of Slurm and queue-based job submission on the cloud).
Read more about this partnership...
Collaboration and learning are among the pillars of DevOps practices. With that in mind, we're keeping our roadmap open as we explore opportunities to engage with other developers in the research space.
More information
We hosted a Lunch 'n Learn webinar in early November to share an overview of our Research Developer Cloud. We covered:
- the cloud-native building blocks available for use
- the partnership-led approaches NeSI is using to build the tools and manage the platform
- how we applied DevOps practices to the system
- examples of interacting with the cloud-native platform via code
If you missed the session, you can watch the embedded video below or watch on NeSI's YouTube channel.
If you have questions about the Research Developer Cloud or NeSI's future plans in this space, get in touch.
