Interoperable controlled-access big data transfer for ELIXIR - expanding EGA collaboration

Building upon the existing ELIXIR based collaboration within the EMBL-EBI in the UK and the Center for Genomic Regulation (CRG) in Spain, this project addressed limitations on computing, network bandwidth, and storage buffer areas that affected the EGA data delivery, and developed a protocol for secure data transfer from the EGA archive to CRG.

The project tested a number of data transfer protocols, (FTP, UDT, Aspera and Globus), optimised the EBI hardware and network access to the archive in support of large scale re-encryption processes, and created necessary monitoring tools for validating data integrity at the CRG storage. In total, the project successfully transferred one petabyte of data using Aspera.

The new protocol now allows us to further optimise each step with the aim of automating the entire process by the end of 2015. The study is now finished, the end report is available here.

Further Implementation Studies:

Representatives from EMBL-EBI present a webinar on the Local EGA demo setup from later work (See also this link)

Duration: 

1 January 2014 to 30 November 2014

Nodes involved: 

Platform/Community: