The sharing and transfer of huge amounts of data across borders and continents are key foundations for the advancement of global science. Such data is essential in advanced, leading-edge research that run extreme data-intensive calculations to solve today’s most complex research puzzles for tomorrow’s innovations. Applications for the data are wide-ranging including the Internet of Things (IoT), Artificial Intelligence (AI), advanced manufacturing, drug discovery, weather monitoring, big data analysis and many others.

Over the past decade, such transfers have progressed beyond research and into applications, and at increasingly larger data amounts as innovations and standards of the data resolution being captured continues to evolve. The network bandwidth to support the growing demand has gone from Mega to Giga, and to hundreds of Gbps in just a few years. Despite these advancements in high bandwidth capacity, scientists and researchers are still not experiencing the corresponding increase in the “speed” of data transfer. This is due to multiple factors, including server and storage hardware limitations, operating system tuning, and data transfer tools used, to name a few.

The Data Mover Challenge (DMC), which is organised by the National Supercomputing Centre (NSCC), Singapore brings together experts from industry and academia, to push the envelope in high speed, big data transfer capabilities, and across vast distances. Participants for DMC19 included international teams from USA, Japan, Taiwan, Indonesia, Thailand, Vietnam and South Korea. The Data Transfer Nodes (DTNs) for DMC19 that were set up globally in Singapore, Australia, USA, South Korea and Japan were connected by 100Gbps international research and education networks.

The next edition of the Data Mover Challenge, DMC20, has been launched. This year’s edition will challenge participants to send test data effectively around the globe!

Submissions for DMC20 are open from 01 June 2019 till 31 July 2019.

The lead of the winning team will have the opportunity to showcase the team’s solutions at the SupercomputingAsia conference in Singapore, which includes return coach airfare and free admission passes to the event that gathers the best of HPC in Asia!

DMC20 Participating Teams

Organisation Country/Countries Team name Solution Title
Computer Canada / Calcul Canada Canada Compute Canada DTN-as-a-Service
Gauss Centre for Supercomputing (GCS) e.V. Germany GCS Unicore UFTP
iCAIR / Starlight United States Starlight 2020 Starlight DTN-as-a-Service
JAXA: Japan Aerospace Exploration Agency, NICT: National Information of Communications and Technology Japan Team Musashino High-speed satellite data transferring over the world from JAXA / NICT via HpFP protocol
National Centre for High performance Computing / NARLabs Indonesia, Japan, Vietnam, Taiwan, Thailand and Unites States SEAIP Data Movers SEAIP-DTN-as-a-Service
National Institute of Informatics Japan NII MMCFTP
Visuality Systems Israel NQ NQ line of products – SMB Implementation

Dmc20 – “Data For Science”

The capability to effect the sharing of large amounts of research data quickly and securely between collaborating entities or computational resources will rise in demand as science becomes more data-intensive with more precise and larger instruments, and global research cooperation increases.

The theme for DMC20, “Data for Science”, places an emphasis on the need for better coordination and speed in moving large amounts of data between multiple sites with Data Transfer Nodes (DTNs) as well as to HPC centres around the globe.

DMC20 will focus on optimising point-to-point data transfer between sites – a crucial step forward in advancing research collaboration and sharing. DMC20 will extend the network testbed and location of DTNs to cover the entire world. The intention will be for the test data to circle the globe. Partners in Asia, Europe and USA will be supporting the competition.

The inaugural DMC19 Overall Winner: Zettar Inc.

(Left-Right: Andrew Howard, Cloud Team Manager, National Computational Infrastructure Australia, Dr Chin Fang, Founder & CEO, Zettar Inc., Prof Tan Tin Wee, Chief Executive, National Supercomputing Centre (NSCC) Singapore)

Dmc20 – At SupercomputingAsia 2020

The winner of DMC20 will be announced at SupercomputingAsia 2020 (SCA20)!

SCA20 will be held from 24-27 February 2020 in Singapore (www.sc-asia.org). The lead of the winning team will be invited to SCA20 to receive their award and to present the team’s winning solution at the conference.

our Scope

  • Network Infrastructure
    • Each team will leverage on the global NREN infrastructure for the Data Mover Challenge.
    • The deployed DTNs will connect all locations, including Australia, Europe, Japan, Singapore and USA.
    • Teams must determine the connection type that will be used, for example, point-to-point, multipoint or distributed.
    • Multiple Virtual Local Area Networks (VLANs) will be set up, per location, to provide inter-connectivity.
    • Network monitoring will be set up to capture data transfer statistics.
  • Server Hardware sponsorship
    • Purpose-built DTN servers will be provided by partners based on baseline minimum specifications – e.g. 100/10 Gb capability,
      local storage, connection to high performance file systems
    • Existing DTN systems at each partner sites must be used.
    • Singularity container software will be used on the DTN servers.
  • Applications
    • Data transfer tools & software will be provided by the competing team.
    • Teams will need to implement their DTN transfer tools & software within Singularity.
    • A separate note will be sent to the shortlisted DMC20 participants at the start of the challenge with details on how to access the nodes, and other operational processes.

Data Mover Challenge topology

  • DMC20 is open to all organisations, companies, research institutions, academia, researchers, post-graduate students and undergraduate students.
  • The final number of Participating Teams for the challenge will be assessed and shortlisted at the discretion of the DMC20 organising committee.
  • The organiser, co-organisers and representatives involved in organising DMC20 are not eligible to participate in the challenge.
  • Error free data transfer performance (average transfer rate and time taken) of the data set
    • Memory to Memory Test for both directions (Weightage 20%)
    • Disk to Disk Test for both directions (Weightage 40%)
  • Technological Innovation base on solution submission and interview with judges (Weightage 20%)
  • End-user Experience including user interface, ease of installation, licensing model, and integration with services such as access federation etc. (Weightage 20%)

The Data Mover Challenge will run from Aug 2019 to Jan 2020 (excluding Oct and Nov 2019).

Each Participating Team will be given 5 days for the challenge.

Monday SGT (UTC+8) 0000Hrs – Wednesday SGT (UTC+8) 2359Hrs: Software Setup

Thursday SGT (UTC+8) 0000Hrs – Friday SGT (UTC+8) 2359Hrs: Running and demonstrating of Software to judges (Statistics and results will be captured)

The winning team’s leader will be invited to attend SupercomputingAsia 2020 (SCA20) from 24-27 Feb 2020 in Singapore for an award presentation and solution showcase.

Download Participants’ information here. DMC20 Participants Info

Registration

Update: Submission for DMC20 entries has closed. Thank you

For enquiries about the challenge or any other details, please contact [email protected]

Important dates

  • June 01 2019

    Launch of DMC20

  • June 01 - July 31 2019

    Submission Period of entries for DMC20

  • July 31 2019

    Close of entry submissioins for DMC20

  • Aug 31 2019

    Announcement Of DMC20 Shortlisted Participants

  • Aug 26 - Jan 232019   2020

    Start challenge to begin

  • Jan 23 2020

    End challenge to conclude

  • feb 24 - 27 2020

    SupercomputingAsia 2020 (SCA20) Gathering the best of HPC in Asia!

  • Feb 26 2020

    DMC20 Award Ceremony Announcement Of winner(s)

Organised By:

Supporting Partners:

DO YOU HAVE A QUESTION ABOUT SCA20?

You can contact us at [email protected]

Contact Us