The sharing and transfer of huge amounts of data across borders and continents are key foundations for the advancement of global science. Such data is essential in advanced, leading-edge research that run extreme data-intensive calculations to solve today’s most complex research puzzles for tomorrow’s innovations. Applications for the data are wide-ranging including the Internet of Things (IoT), Artificial Intelligence (AI), advanced manufacturing, drug discovery, weather monitoring, big data analysis and many others.

Over the past decade, such transfers have progressed beyond research and into applications, and at increasingly larger data amounts as innovations and standards of the data resolution being captured continues to evolve. The network bandwidth to support the growing demand has gone from Mega to Giga, and to hundreds of Gbps in just a few years. Despite these advancements in high bandwidth capacity, scientists and researchers are still not experiencing the corresponding increase in the “speed” of data transfer. This is due to multiple factors, including server and storage hardware limitations, operating system tuning, and data transfer tools used, to name a few.

The Data Mover Challenge (DMC), which is organised by the National Supercomputing Centre (NSCC), Singapore brings together experts from industry and academia, to push the envelope in high speed, big data transfer capabilities, and across vast distances. Participants for DMC19 included international teams from USA, Japan, Taiwan, Indonesia, Thailand, Vietnam and South Korea. The Data Transfer Nodes (DTNs) for DMC19 that were set up globally in Singapore, Australia, USA, South Korea and Japan were connected by 100Gbps international research and education networks.

The next edition of the Data Mover Challenge, DMC20, has been launched. This year’s edition will challenge participants to send test data effectively around the globe!

Submissions for DMC20 are open from 01 June 2019 till 31 July 2019.

The lead of the winning team will have the opportunity to showcase the team’s solutions at the SupercomputingAsia conference in Singapore, which includes return coach airfare and free admission passes to the event that gathers the best of HPC in Asia!

—•—

Data Mover Challenge 2020 (DMC20)

Announcing the Winners of DMC20!

The Data Mover Challenge (DMC), which is organised by the National Supercomputing Centre (NSCC), Singapore brings together experts from industry and academia, to push the envelope in high speed, big data transfer capabilities, and across vast distances. Data Transfer Nodes (DTNs) for DMC20 connected by 100Gbps international research and education networks were set up globally in Singapore, Australia, Europe, USA, South Korea and Japan.

The DMC20 competition attracted seven international teams from USA, Europe, Japan, Taiwan, Indonesia, Thailand, Vietnam and South Korea
from both the public and private sector.

The challenge focused on optimising point-to-point data transfer between sites – a crucial step forward in advancing research collaboration and sharing. The challenge tested the expertise and skills of each of the teams to produce the best solutions for data transfer.

After extensive deliberations and reviews, the panel of international judges decided that there would not be a single overall winner for DMC20. They have instead decided to present not one but four awards to the teams that had shown unique and innovative solutions in various areas. 

Read more about the DMC20 and the winners here in the Media Release.

Congratulations to all the winning teams!*

  Team

  Award Name

  Citation

  National Institute of Informatics (NII)

  Most Innovative & Novelty
  Award

Team NII’s solution of MMCFTP to transfer datasets and accelerate automatic archiving and de-archiving of small files received the Most Innovative & Novelty Award

  iCAIR / Starlight

  Best Speed & Science
  Integration Award

Team iCAIR / Starlight’s DTN-as-a-Service solution produced high performance data access with NVMe over Fabric with strong integration with Jupyter notebooks received the Best Speed and Science Integration Award

  Gauss Centre for Supercomputing (GCS) e.V.

  Most Systemic Approach
  Award

GCS enhanced the DTN service using an under the hood feature to optimize data transfer performance using UFTP integrating with various application received the Most Systemic Approach Award

  Japan Aerospace Exploration Agency (JAXA) /
  National Institute of Information and
  Communications Technology (NICT)

  Experimental Excellence
  Award

JAXA / NICT’s well engineered HCP tool based on their original high performance flexible protocol with GUI integration produced a wide set of tests in challenging environments received the Experimental Excellence Award

—•—

The international panel of judges:

  Andrew Howard  – Chief Judge

Cloud Systems Manager,
National Computational Infrastructure (NCI)

  Cees de Laat

Professor,
Informatics Institute, Faculty of Science,
University of Amsterdam

  Eric Pouyoul

Network Engineer,
Energy Science Network
(ESnet)

  Francis Lee Bu Sung

Associate Professor,
School of Computer Science and Engineering,
Nanyang Technological University (NTU)

  Lawrence Wong

Emeritus Professor,
Department of Electrical & Computer Engineering,
National University of Singapore (NUS)

  Tim Chown

Network Development Manager,
Jisc

 —•—

DMC20 Partners

 

*Note – In light of the cancellation of the SupercomputingAsia 2020 (SCA20) conference, the announcement of the winners and the DMC20 Awards Ceremony, which was scheduled to be held on 26 February in Singapore will not be proceeding. The organisers will arrange to present the Awards separately to the winning teams.


DMC20 Participating Teams

Organisation Country/Countries Team name Solution Title
Computer Canada / Calcul Canada Canada Compute Canada DTN-as-a-Service
Gauss Centre for Supercomputing (GCS) e.V. Germany GCS Unicore UFTP
iCAIR / Starlight United States Starlight 2020 Starlight DTN-as-a-Service
JAXA: Japan Aerospace Exploration Agency, NICT: National Information of Communications and Technology Japan Team Musashino High-speed satellite data transferring over the world from JAXA / NICT via HpFP protocol
National Centre for High performance Computing / NARLabs Indonesia, Japan, Vietnam, Taiwan, Thailand and Unites States SEAIP Data Movers SEAIP-DTN-as-a-Service
National Institute of Informatics Japan NII MMCFTP
Visuality Systems Israel NQ NQ line of products – SMB Implementation

Dmc20 – “Data For Science”

The capability to effect the sharing of large amounts of research data quickly and securely between collaborating entities or computational resources will rise in demand as science becomes more data-intensive with more precise and larger instruments, and global research cooperation increases.

The theme for DMC20, “Data for Science”, places an emphasis on the need for better coordination and speed in moving large amounts of data between multiple sites with Data Transfer Nodes (DTNs) as well as to HPC centres around the globe.

DMC20 will focus on optimising point-to-point data transfer between sites – a crucial step forward in advancing research collaboration and sharing. DMC20 will extend the network testbed and location of DTNs to cover the entire world. The intention will be for the test data to circle the globe. Partners in Asia, Europe and USA will be supporting the competition.

The inaugural DMC19 Overall Winner: Zettar Inc.

(Left-Right: Andrew Howard, Cloud Team Manager, National Computational Infrastructure Australia, Dr Chin Fang, Founder & CEO, Zettar Inc., Prof Tan Tin Wee, Chief Executive, National Supercomputing Centre (NSCC) Singapore)

Dmc20 – At SupercomputingAsia 2020

The winner of DMC20 will be announced at SupercomputingAsia 2020 (SCA20)!

SCA20 will be held from 24-27 February 2020 in Singapore (www.sc-asia.org). The lead of the winning team will be invited to SCA20 to receive their award and to present the team’s winning solution at the conference.

our Scope

  • Network Infrastructure
    • Each team will leverage on the global NREN infrastructure for the Data Mover Challenge.
    • The deployed DTNs will connect all locations, including Australia, Europe, Japan, Singapore and USA.
    • Teams must determine the connection type that will be used, for example, point-to-point, multipoint or distributed.
    • Multiple Virtual Local Area Networks (VLANs) will be set up, per location, to provide inter-connectivity.
    • Network monitoring will be set up to capture data transfer statistics.
  • Server Hardware sponsorship
    • Purpose-built DTN servers will be provided by partners based on baseline minimum specifications – e.g. 100/10 Gb capability,
      local storage, connection to high performance file systems
    • Existing DTN systems at each partner sites must be used.
    • Singularity container software will be used on the DTN servers.
  • Applications
    • Data transfer tools & software will be provided by the competing team.
    • Teams will need to implement their DTN transfer tools & software within Singularity.
    • A separate note will be sent to the shortlisted DMC20 participants at the start of the challenge with details on how to access the nodes, and other operational processes.

Data Mover Challenge topology

  • DMC20 is open to all organisations, companies, research institutions, academia, researchers, post-graduate students and undergraduate students.
  • The final number of Participating Teams for the challenge will be assessed and shortlisted at the discretion of the DMC20 organising committee.
  • The organiser, co-organisers and representatives involved in organising DMC20 are not eligible to participate in the challenge.
  • Error free data transfer performance (average transfer rate and time taken) of the data set
    • Memory to Memory Test for both directions (Weightage 20%)
    • Disk to Disk Test for both directions (Weightage 40%)
  • Technological Innovation base on solution submission and interview with judges (Weightage 20%)
  • End-user Experience including user interface, ease of installation, licensing model, and integration with services such as access federation etc. (Weightage 20%)

The Data Mover Challenge will run from Aug 2019 to Jan 2020 (excluding Oct and Nov 2019).

Each Participating Team will be given 5 days for the challenge.

Monday SGT (UTC+8) 0000Hrs – Wednesday SGT (UTC+8) 2359Hrs: Software Setup

Thursday SGT (UTC+8) 0000Hrs – Friday SGT (UTC+8) 2359Hrs: Running and demonstrating of Software to judges (Statistics and results will be captured)

The winning team’s leader will be invited to attend SupercomputingAsia 2020 (SCA20) from 24-27 Feb 2020 in Singapore for an award presentation and solution showcase.

Download Participants’ information here. DMC20 Participants Info

Registration

Update: Submission for DMC20 entries has closed. Thank you

For enquiries about the challenge or any other details, please contact [email protected]

Important dates

  • June 01 2019

    Launch of DMC20

  • June 01 - July 31 2019

    Submission Period of entries for DMC20

  • July 31 2019

    Close of entry submissioins for DMC20

  • Aug 31 2019

    Announcement Of DMC20 Shortlisted Participants

  • Aug 26 - Jan 232019   2020

    Start challenge to begin

  • Jan 23 2020

    End challenge to conclude

  • Feb 24 - 27 2020

    SupercomputingAsia 2020 (SCA20) in Singapore CEREMONY CANCELLED

  • Feb 26 2020

    DMC20 Award Ceremony Announcement Of winner(s)

Organised By:

DO YOU HAVE A QUESTION ABOUT SCA20?

You can contact us at [email protected]

Contact Us