NeSI explores innovative uses of cloud services for advanced computing

NeSI improves research productivity by adapting platforms to meet these needs, while building critical mass and supporting innovation. A good example of this is how NeSI is starting to integrate public cloud technology with our on-premise high performance computing platforms to provide greater capacity, potentially more services, and mitigate risk of infrastructure failure.

In 2016 NeSI created a seamless integration from one of our platforms to two public cloud vendors offering diverse services. This pilot project was successful, resulting in a working capability to burst from the NeSI platform to both cloud services, and building our knowledge and experience in cloud integration.

Why?

NeSI’s purpose is to grow the computing capability of researchers, which we deliver by providing advanced computing services and by contributing directly to sector capability development and capacity growth through computational consulting and training in digital skills. Underpinning our services are our current platforms which focus on high performance computing, are currently well utilised, and provide a broad range of benefits across many research projects.

Integration of public or private cloud computing with these large on-premise platforms will allow NeSI to be more responsive to researcher needs. This could be delivered by supporting cloud bursting, providing access to specialised computing environments, and/or development of virtual labs.

Cloud bursting can provide more capacity at peak times by moving certain workloads onto the cloud, and in some cases mitigates the risk of service unavailability due to infrastructure failure. Access to specialised computing environments can enable researchers to apply technologies and solutions not already deployed on NeSI. Development of virtual labs provides a well-curated community-centric solution for meeting common data workflow, processing, and visualisation needs.

Attribution: 

Figure 1: bursting into the cloud to provide increased capacity.

One key insight gained from our work to date is that each cloud service has its own sweet spot in terms of price and value. To meet our various goals, it is likely NeSI will need to integrate and adapt solutions from multiple providers.

Pilot project – cloud bursting

A pilot project was seen as useful to test how cloud burst integration could be done and refine our thinking before moving to a full cloud implementation. The pilot was designed to test the following ideas:

• Can a job run in the cloud? To make use of cloud services, a researcher’s job is first analysed to determine if it can be burst to the cloud for execution. This analysis applies business rules which evaluate technical criteria and ensure data sovereignty requirements are met; jobs satisfying both criteria may be executed in the cloud or otherwise are queued to run on NeSI’s platforms.
• Can cloud bursting be made transparent to the user? This means moving the job to and from the cloud is transparent to the researcher, and that jobs executed in the cloud deliver scientifically identical results to the same job executed on NeSI’s platforms. This will provide more flexibility and scalability for NeSI researchers in the future when implemented in production.

The pilot involved designing two different solutions to evaluate the feasibility of offloading workloads to the cloud. They were based on Infrastructure as a Service (IaaS) and Software as a Service (SaaS) models. In our pilot, the IaaS solution was tightly integrated with NeSI’s Pan platform, putting NeSI in control of service provisioning, including workload scheduling, data migration, and software package deployment. In comparison, the SaaS solution required little integration, though service provisioning was limited.

The pilot confirmed that NeSI can integrate cloud technology with our existing platforms in a seamless way. It became clear through the project that the design and implementation of the IaaS solution was more straightforward and would lead to a better experience for the researcher. Both solutions could form part of NeSI’s future infrastructure.

In the first half of 2017 we will take the learnings from the 2016 pilot and implement and trial cloud services with a set of researchers open to exploring this new way of working. In addition to the seamless cloud burst capability we will be looking at the life-cycle aspects of working with cloud environments, to ensure our future integrated infrastructure supports appropriate policy, reporting, and accounting.