The AmLight Team participated in multiple Network Research Exhibition (NRE) demonstrations at the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC18) held in Dallas, TX. NRE participants are invited to share the results of their demonstrations and experiments that display innovation in emerging network hardware, protocols, and advanced network-intensive scientific applications from the preceding year’s conference as part of the Innovating the Network for Data-Intensive Science (INDIS) workshop.
NRE-16: Global Petascale to Exascale Science Workflows Accelerated by Next Generation SDN Architectures and Applications
We will demonstrate several major advances in software defined and Terabit/sec networks, intelligent global operations and monitoring systems, workflow optimization methodologies with real-time analytics, and state of the art long distance data transfer methods and tools and server designs, to meet the challenges faced by leading edge data intensive experimental programs in high energy physics, astrophysics, climate science including the Large Hadron Collider (LHC), the Large Synoptic Space Telescope (LSST), the Linac Coherent Light Source (LCLS II), the Earth System Grid Federation and others. Several of the SC18 demonstrations will include a fundamentally new concept of “consistent network operations,” where stable load balanced high throughput workflows crossing optimally chosen network paths, up to preset high water marks to accommodate other traffic, provided by autonomous site-resident services dynamically interacting with network-resident services, in response to demands from the science programs’ principal data distribution and management systems. This will be empowered by end-to-end SDN methods extending all the way to autoconfigured Data Transfer Nodes (DTNs), including intent-based networking APIs combined with transfer applications such as Caltech’s open source TCP based FDT which have been shown to match 100G long distance paths at wire speed in production networks. During the demos, the data flows will be steered across regional, continental and transoceanic wide area networks through the orchestration software and controllers, and automated virtualization software stacks developed in the SENSE, PRP, AmLight, Kytos and other collaborative projects. The DTNs employed will use the latest high throughput SSDs and flow control methods at the edges such as FireQoS and/or Open vSwitch, complemented by NVMe over fabric installations in some locations.
Download the final NRE-16 demo submission (PDF)
See the official LSST press release online.
NRE-17: Large Synoptic Survey Telescope (LSST) Real Time Low Latency Transfers for Scientific Processing Demonstrations
At SC18 in Dallas, Texas we plan to experiment with data transfer rates, using 100Gig FIONA Data Transfer Nodes (a.k.a. DTNs) in Chile and Illinois. The demos aim to achieve three goals: First, we will demonstrate real time low latency transfers for scientific processing of multi-Gigabyte images from the LSST base station site in La Serena, Chile, flowing over the REUNA Chilean National Research & Education Network (NREN), as well as ANSP and RNP Brazilian national circuits and the AmLight-ExP Atlantic and Pacific Ring through AMPATH2 to Starlight and NCSA. Second, we will simulate operational and data quality traffic to SLAC, Tucson and other sites including the Dallas show floor. Third, we will stress test the AmLight ExP network to simulate the LSST annual multi-petabyte Data Release from NCSA to La Serena at rates consistent with those required for LSST operations.
Download the final NRE-17 demo submission (PDF).
NRE-18: Americas Lightpaths Express and Protect Enhances Infrastructure for Research and Education
Americas Lightpaths Express and Protect (AmLight ExP) enables research and education amongst the people of the Americas through the operation of production infrastructure for communication and collaboration between the U.S. and Western Hemisphere science and engineering research and education communities. AmLight ExP supports a hybrid network strategy that combines optical spectrum (Express) and leased capacity (Protect) that provides a reliable, leading-edge diverse network infrastructure for research and education.
Download the final NRE-18 demo submission (PDF)
More details about the SC18 can be found here