Digital support in a national crisis

Managing data around the Christchurch earthquakes

In the hours following both Christchurch earthquakes, gigabytes of data needed to be downloaded, stored, processed and delivered to emergency management response teams both at the Beehive and the frontline in Christchurch.

The eResearch community was involved in both emergencies. In the first quake, the community helped to create a pathway for data sharing and downloading via KAREN and the BeSTGRID DataFabric. After the second quake, BeSTGRID became a key coordination mechanism, enabling distributed data coordination across government and emergency agencies.

In both cases, eResearch-based computer technology was key to the recovery effort, with relay via the super-fast KAREN network and housing on BeSTGRID enabling multiple, simultaneous accesses that skirted broken links in Canterbury.

September 2010

The community helped link a diverse group of agencies and providers across the globe on a high-speed network so that first-response agencies could understand the magnitude of the event.

Between 7am and 8am on 4 September 2010, GeoEye-1 passed over Christchurch, capturing urgently needed images. GeoEye-1 orbits around 680km above the earth, sending back some of the world’s highest resolution satellite imagery.

On the ground, the New Zealand Defence Force’s Geospatial Intelligence Organisation (GIO) in Auckland was approached by the Ministry of Civil Defence & Emergency Management for imagery to help agencies assess the extent of the quake’s damage.

GeoEye-1’s images showed some badly-damaged buildings and large areas of liquefaction, but also confirmed that there wasn’t widespread devastation.  This imagery would provide frontline agencies with a regional overview of the quake’s damage, supplementing local reports, news feeds and photography and video from helicopters.

Sharing the data

GeoEye provided the imagery free-of-charge for use in the disaster response, but the size of the files led to some data-logistics challenges.

“The first challenge was obtaining the imagery quickly from GeoEye in the US for initial processing and analysis. Downloading over 50 GB of files through a standard internet connection was simply going to take too long”.

“The second was how to on-supply the imagery and associated analytical products to both the National Crisis Management Centre in the Beehive and to the emergency response agencies in Christchurch,” says Richard Wells, Deputy Director Production at GIO.   

ORCON solved the initial download problem, offering a direct international link through their server in Auckland.  The Centre for eResearch was then approached to assist with the New Zealand-wide data distribution.

“Staff at BeSTGRID were able to make use of fast networking and large shared storage facilities to create a nationally-distributed ‘Data Fabric’ that acted as a collation point for several of the datasets used in the subsequent relief effort, including satellite images, LiDAR elevation data and landcover maps.  This Fabric greatly simplified the effort involved in moving massive data files around the country,” says Centre for eResearch Director, Professor Mark Gahegan. From BeSTGRID the files were distributed via the super fast KAREN network linking New Zealand’s leading universities and research organisations.

In addition to satellite imagery, GIO was also liaising with New Zealand Aerial Mapping (NZAM), a company specialising in aerial photographic and LIDAR surveys.  A survey was flown the following day and large geospatial datasets were uploaded to BeSTGRID and distributed via KAREN.

February 2011

The impact of the 22 February 2011 earthquake was much greater than the September event.  This was mirrored by the scale of the data-logistics challenge.  In September, eResearch and BeSTGRID supported some 35 GB of data. In 2011, that increased to around 350 GB.

“Despite this tenfold increase in data, eResearch found the DataFabric massively reduced the logistical effort because it was available in a share space,” says Nick Jones, Co-Director Centre for eResearch and Director, BeSTGRID.

“The eResearch community coordinated its own response team in providing the infrastructure across a large number of science agencies (Auckland, Canterbury and Lincoln universities, and GNS and Landcare Research) and research infrastructure groups (BeSTGRID, KAREN, Australian Research Collaboration Service) incorporating different teams at each site and a core response team of 10 IT experts. The team was supported by industry groups, such as SNAP internet and Orcon”.

The magnitude of this earthquake was such that the initial response team worked extremely long days and nights through the first two weeks following the quake to address performance and routing issues and provide support to a wider range of agencies than in September 2010.

The community supported users from across New Zealand including the Defence Force, National Crisis Management Centre, Police and Fire Service, Land Information New Zealand, Landcare Research, GNS and commercial geospatial data providers Koordinates (NZ) and Google (US, Aus). Much of the data loading and redistribution to the Defence Force took place in the Centre for eResearch labs at The University of Auckland.

“The network also provided on-site drop-in support, allowing response teams to come on-site at Auckland, Lincoln, Landcare Palmerston North and Victoria to get direct access to very large data sets, which they couldn’t have accessed quickly through standard internet connections,” says Nick Jones.

What We Learnt

The importance of timeliness

When the quakes hit, information was a priority, firstly to establish the extent of the damage and whether there had been loss of life and, secondly, to help emergency response agencies manage the crisis. At the same time, other agencies such as GIO, GNS, Landcare, and LINZ were alerted.

“Agencies require post-event data and imagery quickly. Christchurch highlighted the need to have pre-existing access to BeSTGRID’s DataFabric for data processing and high-speed networks such as KAREN for data-dissemination in emergency response situations,” says Richard Wells.

The need for faster delivery channels

“We’ve moved past radio communication and a truck with an antenna. Disaster response now requires and generates massive amounts of data with hundreds of gigabytes streaming daily,” says Centre Director Professor Mark Gahegan.

“Much of the data comes in from US satellite data providers and is too large to bring across the Pacific in reasonable time. We've now established a working large-scale data pipeline with Argonne National Labs in Chicago that has reduced delivery time from at least 24 hours to within 30 minutes.  

“What we’ve learnt from Christchurch is that New Zealand has the e-infrastructure to get first-response agencies connected when there is a disaster of national magnitude. Now, we need to put in place a formal structure including BeSTGRID and KAREN to manage future national emergencies.”

Less formal delivery channels also sprang up during the second quake. The Student Army used Facebook to mobilise its forces and science-blogger Vicki Hyde set up her laptop and a printer on the street to provide printed updates for residents. Mr Jones says groups like this also have a role to play in gathering and disseminating information because of their ability to maximise the scale of distributed communities and related technology.

Data storage

In the two weeks immediately following the February earthquake, eResearch handled more than 660 GB of data storage and distribution requirements of various agencies and groups across New Zealand through the BeSTGRID DataFabric.

By mid-April, the DataFabric of the eResearch Centre was storing almost 300 GB of data directly related to the Christchurch earthquake.

Sharing data

Some of the data sets have been licensed as open source and are now widely available for public use and application, which is in-line with the Government Open Access and Licensing framework (NZGOAL).

“NZGOAL recognises agencies have vast quantities of data and non-copyright material that may have significant creative, economic, cultural and sustainability potential,” says Professor Gahegan.

Post-quake analysis

In the days and weeks following the quake, additional imagery and datasets were distributed via KAREN.

The February earthquake showed agencies need high-resolution, post-event imagery and large geospatial datasets and that these create data-logistics challenges in both processing and distribution.

KAREN’s key advantage is its ability to move very large amounts of data around, enabling multiple agencies simultaneous access, while the BeSTGRID DataFabric provides access to data sharing and extensive data processing capacity.

In the longer-term, post-event, there will also be a role for BeSTGRID and KAREN as scientists continue with their analysis and seek to share the resulting data with multiple agencies.