+44 40 8873432 [email protected]

Data Mover Challenge

The movement of data on the internet is a major challenge for researchers in BIG Sciences. Over the past decade, we can see that it is not only in BIG Science that generates large amount of data, but also Internet of Things (IoT) as standards of the data resolution being captured continues to evolve.

We have also seen increases of network bandwidth, from mega to Giga, and 100s of Gbps.  Despite this high bandwidth capacity, researchers are still not experiencing the corresponding increase in the “speed” of data transfer.  This is due to multiple factors: server, transport protocol, etc.

The Data Mover Challenge (DMC) aims to bring together experts from industry and academia, in a bid to test their software across servers located in various countries (Australia, Japan, Singapore, USA) that are connected by 100G international networks.

The Challenge

  • Each team will be given 1 week to deploy the software on globally distributed nodes.
  • The transport protocol must minimally support a data transfer rate of 10Gbps.

Objective of the
data mover challenge

  • To showcase challengers by having them compete in deploying the best software tools on Data Transfer Nodes (DTNs) that are set up within current international Research and Education Networks(RENs) across the globe.
  • To bring together a community of experts in DTN deployments and operations.

Scope

Network Infrastructure

  • Each team will leverage on the global REN infrastructure for the Data Mover Challenge.
  • The deployed DTNs will connect all locations, including Australia, Japan, Singapore and USA.
  • Teams must determine the connection type that will be used, for example, point-to-point, multipoint or distributed.
  • Multiple Virtual Local Area Networks (VLANs) will be set up, per location, to provide inter-connectivity.
  • Network monitoring will be set up to capture data transfer statistics.

Server Hardware sponsorship

  • Purpose-built DTN servers will be provided by partners based on baseline minimum specifications
    – e.g. 100/10 Gb capability, local storage, connection to high performance file systems
  • Existing DTN systems at each partner sites must be used.
  • Singularity container software will be used on the DTN servers.

Applications

  • Data transfer tools & software will be provided by the competing team.
  • Teams will need to implement their DTN transfer tools & software within Singularity.

Data Mover Challenge Topology

Eligibility

  • This Data Mover Challenge is open to all organisations, companies, research institutions, academia, researchers, post-graduate students and undergraduate students.
  • Representatives of all partnering organisations for the challenge shall not participate as team members for this competition.

Judging Criteria

  • A scoresheet with specific data transfer scenarios will be provided during the competition.
  • Judges will consider the following factors:
    1. Maximum sustainable transfer rate from disk to disk
    2. Usability of the software
    3. Cost
    4. Scalability
  • Innovativeness in usage. The competitor can use an application to demonstrate their software across multiple sites.

Result Announcement

  • The winning team will be announced at the SupercomputingAsia (SCA) 2019 Conference in March 2019 in Singapore, and will have the opportunity to showcase a live demonstration of their winning solution.

Winners (Updated: 13 March 2019)

 PRESS RELEASE – Data Mover Challenge – WINNERS Announced!

(Left-Right: Andrew Howard, Cloud Team Manager, National Computational Infrastructure Australia, Dr Chin Fang, Founder & CEO, Zettar Inc., Prof Tan Tin Wee, Chief Executive, National Supercomputing Centre (NSCC) Singapore)

(Left-Right: Andrew Howard, Cloud Team Manager, National Computational Infrastructure Australia, Joe Mambretti, Director, International Center for Advanced Internet Research at Northwestern University, Prof Tan Tin Wee, Chief Executive, National Supercomputing Centre (NSCC) Singapore)

Winning team will receive

Return coach airfare for team leader of the winning team to Singapore from the country he/she is based in

Subsistence Allowance (including accommodation) for SCA19

Complimentary all-access conference pass for SCA19

Recognition of achievement at SCA19

Registration

Registration is now closed. Thank you for your participation.

 

Please contact [email protected] for any questions or enquiries.

Important Dates

  • 30 NOV

    2018

    Deadline for Online Submission of Entries

  • 4 jan

    2019

    Teams will be Informed of their Selection

  • 14 JAN

    24 FEB

    2019

    Individual Teams Deployment

  • 12 MAR

    2019

    Announcement of Winner at SCA19

  • 12 MAR

    14 MAR

    2019

    Winning Team’s Demo at SCA19

Supporting Partners