SAACC Meeting May 18, 2018 Report

[1], remote operations for Gemini telescope, remote observing for Southern Astrophysical Research (SOAR) telescopes, significant data transfer for LSST, data transfer for Carnegie Observatory in La Serena, NRAO/ALMA telescope (Santiago) and pending the Giant Magellan Telescope (GMT). Smaller users include SMARTS, PROMPT (x8 now), GONG, ALO, WHAM, LCOGTN, KASI/KMTnet, ASAS-SN, mEarth (Harvard), “EvryScope”/Prompt, and T80S (Brazil).

The key use cases from the technical perspective are data transfer (high bandwidth), remote observing (high QoS), robotic telescopes (high reliability), coordination (real-time and other QoS), data access and archiving (bandwidth and QoS), and distributed processing and analysis. Examples of outstanding scientific discoveries made possible, based on the collaborations over the network, are the Dark Energy (2011 Nobel Prize)[2], Gravitational Wave (2017 Nobel Prize)[3], and the Discovery & Study of the Exoplanets. Solid cross-disciplinary foundations in astroinformatics in Chile are supported by current initiatives from Center for Mathematical Modeling (CMM) and National Laboratory for High-Performance Computing (NLHPC), and a new emerging ones by University of La Serena “Big Data Initiative” and the Ministry of Economy/CORFO: Astroinformatics Initiative (possible Institute for Astroinformatics & Economic Development), and initiatives in the area of Bioinformatics.

 

National Radio Astronomy Observatory (NRAO)
The NRAO team presented updates on data transfer from ALMA to North America. A correlator upgrade is scheduled for 2022 and will allow up to eight-times more channels, and a twice wider bandwidth. The expected data rate increase is about a factor of four, corresponding to a data rate of 1PB/yr. (not all projects will need the extra channels, though most will use the wider bandwidth). Some increase in data rate can be expected in the buildup to this as the network at the AOS is upgraded (but should only be modest, ~10% overall). In summary, the ramp-up of the ALMA data rate has been slower than anticipated, allowing to stay ahead of the curve. The two-stream data collection in Cycles 3-5 is artificially boosting the data rate, and it is assumed that this will no longer be the case in Cycle 6 (Oct 2018 onwards). The bidirectional data flow will continue to be needed to support data processing at the ARCs, and a learning process is in place of how the network performs when transferring ~1TB/day in multiple parallel streams. A desire was expressed of how to establish a link with 1Gb/s available bandwidth (out of a 10Gb/s pipe) within the next 1-2 years to improve our transfer speed to and from Chile for bulk reprocessing, and to help with occasional large data and metadata transports (e.g., a DB export). New developments on ~5yr timescale (e.g., the correlator upgrade) can probably be accommodated with a modest (x4) increase the data rate.

 

Atacama Large Millimeter/Submillimeter Array (ALMA) Communication Infrastructure
ALMA Communication Infrastructure updates on system performance, downtime events, and availability were presented for the period May 2017 to April 2018. Three major fiber cuts have affected the availability for this period: 1) Aug 2017 fiber cable Santiago REUNA damaged by road work, 2) Dec 2017 splice box on the Santiago ESO fiber was damaged, and 3) Feb 2018 Several days long AOS-CALAMA cable damaged by Altiplanic winter[4] had a limited user impact as ALMA was in engineering maintenance anyhow and the AOS node disabled. The presence of a fiber cable in the Chajnantor area that connects to Calama and the Chilean communication backbone and the availability of the policy for Non-ALMA projects and experiments approval in the ALMA concession was presented as the key enablers for an improved connection. The intention is to create a REUNA Point of Presence at AOS will create the possibility of fiber-based class communication for Astronomical Facilities in Chajnantor.

 

Large Synoptic Survey Telescope (LSST)
Updates on multiple sites and connectors to LSST was presented:

• Accepted fibers: from the Summit to AURA Gatehouse, from the AURA Gatehouse to La Serena, and from La Serena to Santiago
• Installed Dense Wave Division Multiplex (DWDM) equipment on Cerro Pachon, in La Serena, and in Santiago
• Conducted First Optic Light Demonstration December 2017
• Activated 100 Gbps Ring from Santiago, Chile to Miami, USA

During the First Optic Light demonstration in December 2017, the LSST Network Engineering Team (NET) completed the first successful transfer of digital data over LSST/AURA fiber optic networks from the Summit Site on Cerro Pachon, Chile to the Base Site in La Serena, Chile and on to the Archive Site at NCSA in Champaign. A set of 6 x 10 Gbps Network Interface cards on Data Transfer Nodes (DTN) configured with iPerf3 generated a sustained data rate of approximately 44 gigabits per second, over a period of 24 hours. This exceeded the test target of 40 gigabits per second.

Installation of Cisco ACI-based Summit network, DWDM procurement for 100 Gbps Spectrum link, the design of the Base Data Center in La Serena, and an agreement with ESnet (to provide 2 x 100 Gbps links Atlanta – Chicago) is in progress.

The remaining major milestones for 2018 are:

• Initial Summit network
• Summit Computer room server installations
• Base LAN Design completion
• LSE-239 Base Data Center Requirements updated/rebaselined
• 100 Gbps test between Chile and NCSA
• Activation of Spectrum Link (Sao Paolo to Boca Raton)
• Completion of ESNet Agreement

 

Americas Lightpaths Express and Protect (AmLight ExP[5])
Network Services updates presented by AmLight ExP included:

• 100G ring Miami-Fortaleza, Fortaleza-Sao Paulo, Sao Paulo- Santiago, Santiago-Miami
• Panama being added Q2 2018
• 10G ring from Miami-Sao Paulo- Miami for protection
• 10G Miami-Santiago for protection
• 100G and 10G rings are diverse, operating on multiple submarine cables
• Total upstream capacity presently at 230Gbps

Currently, each AmLight PoP has a 10G perfSonar node with two NICs (BWCTL and OWAMP), and two 100G network taps installed in Miami to support the IRNC AMIS project for monitoring and measurements. An additional 100G network tap is being installed between Sao Paulo and Santiago. Also, a Maddash portal available online.

Future planning for 2018 includes the Spectrum activation planned for Year 4: 1) Two 100G channels via Fortaleza (RNP and ANSP) (Q3 2018), and 2) One direct 100G channel from Miami to Sao Paulo (ANSP) (Q4 2018). Panama will be added to the 100G Protect ring in Q2 2018 (Miami, Fortaleza, Sao Paulo, Santiago, Panama, Miami). A draft network path for LSST to Chicago was presented including dedicated, secondary, and backup paths.

 

AmLight ExP SDN/SDX updates
AmLight became an SDN network in 2014, and since then researchers can use virtualization to prototype their own application (current testbeds are ONOS/CasTOR, FIBRE, Kytos E-Line, Awave-SDX). As new scientific instruments are deployed, the need for large real-time data transfer is increased. In response to the growing demand, the AtlanticWave-SDX[6] (Awave-SDX) project is building a distributed intercontinental experimental SDX[7]. The goal of the SDX is to support end-to-end services capable of spanning multiple SDN domains, use application provisioning of end-to-end Layer 2 circuits, and providing network programmability. Current features of the AtlanticWave-SDX are:

• Layer 2 Point-to-Point and Point-to-Multipoint circuits with bandwidth reservation and MAC learning
• Web UI and REST calls customized per user profile
• REST supporting HTML and JSON replies
• Support for complex data plane pipelines and arbitrary advanced rules
• Support for Inbound NSI requests for inter-domain provisioning
• Support for Docker and Vagrant images
• Shibboleth

For the 2018 deployment plans for 2018, an installation of SDX switches in Brazil and Chile is planned, and a L2VPN will be created connecting SOX to AMPATH via Internet2. The L2VPNs will connect AMPATH, AndesLight, and SouthernLight’s SDX switches. An Awave-SDX prototype will connect all sites and astronomers, and domain scientists will be invited to try it out.

A cooperative troubleshooting use case of the data transfer between ALMA and NERSC with perfSonar was presented. The NRENs involved in the path were ALMA, REUNA, RedCLARA, AmLight, ESNet, and NERSC. A detailed diagram of the all NRENs network component was presented along with a dashboard with packet lost measurements before and after a bypass circuit was provisioned. This use case showcased the difficulty the network teams from different NRENs are facing when faced with troubleshooting multinomial path and as a possible solution and SDN/SDX solution can be implemented.

 

National Network for Research and Education in Chile, REUNA
Current status of the REUNA backbone was presented. On April 2018, the first 100G DWDM 700 km optical path from La Serena to Santiago containing 4x100G lambda was launched. At the lightening up the LSST fiber optic network transfer data from Pachon/LSST to NSCA Illinois, USA, a set of 6x10Gbps NIC in DTN generated and sustained a data rate of 48Gbps for a 24h period.

A Pilot communication test in collaboration with Simons Observatory and ALMA REUNA’s PoP in Chajnantor demo was also presented. The goal of the demonstration was to simulate PoP at AOS to bring connectivity to other observatories using part of the existing network.

The future planning for REUNA’s connectivity included:

• Link: La Serena – Temuco: 100G DWDM network (2018)
• Link: La Serena – Antofagasta – Calama – Arica (2019)
• Link: Temuco – Puerto Montt (2020)
• Potential integration with FOA (2021)

 

Brazil’s academic network – Rede Nacional de Ensino e Pesquisa (RNP)
RNP presented updates on agreements signed with the Electrical Power Utility Companies to expand the RNP scalable 100G infrastructure. Two of the 100G links are operational, and five more are excepted to be activated by the end of the year. The goal is to have seven PoP connected in a 100G ring. Future plans for the RNP backbone include projects involving Northeast, Southeast, and Southern regions of Brazil. To support the LSST data transfer demand, RNP is working closely with the AmLight team to light up a 100G channel in Monet cable system in 2018. Expected to be operational by Q3 in 2018 are the links from Miami to Boca Raton to Fortaleza and from Fortaleza to Santana do Parnaíba/SP – Barueri/SP – São Paulo/SP.

The path to support LSST data transfer from Fortaleza to Porto Alegre will use the network infrastructure required by the BELLA-T project[8]. Outside the RNP backbone, from Porto Alegre, Brazil to Buenos Aires, Argentina, the path is estimated to be finalized by the end of 2018 and operational by June 2019. The next section of the path, from Buenos Aires to Santiago, Chile, is ready to receive the 100G transponders; however, the DWDM/DCI equipment is not selected yet.

 

RedCLARA
RedCLARA presented a future plan for the terrestrial network evolution for the next two years, and the strategy followed to increase the bandwidth in Latin America. Most of the current links are at 10Gbps capacity, and few are 5Gbps and 2.5Gbps. Few links will be eliminated because of cost, and a new link will be added (Venezuela). Panama link is planned to be added by the end of 2018. New Pacific Ocean Link at 10Gbps connecting Santiago, Guayaquil, Panama, and Barranquilla, and a new connection between Buenos Aires, Argentina, and Montevideo, Uruguay. The link between Argentina and Chile is in advance progress of negotiation too and could be done probably by the end of 2018. An alternative link is considered from Chile to Brazil to Colombia in the north part of Latin America. RedCLARA is in the process of building a redundancy network rings in every country in Latin & Central America to join the universities and research institutions with the end goal to have all countries connected with 100Gbps. Also, a private-public association has been discussed among the partners to support the operations and future development.

 

Internet2 (I2)
The science users are relying on the network ecosystem to support their unique research needs. I2 Core network provides support for various scientific workflows to support science collaborations. The required trends identified by I2 to support the growing network demands are tighter end-to-end collaboration, better standardization, new skills, tools & software adoption, and improved capacity in the sense of reduction of unnecessary infrastructure and improved cost/bit ratio.

Two 100G connection sites (Jacksonville and Pensacola) were emphasized as a possible path in Florida to connect to Sothern Crossroads (SoX) in Atlanta, GA. The discussion continued with questions for the LSST parties implying dedicated wave or backbone service and what are the expectations from I2 once the telescope is operational.

[1] DECam is a high-performance, wide-field CCD imager mounted at the prime focus of the Blanco 4-m telescope at CTIO http://www.ctio.noao.edu/noao/node/1033

[2] Hammel, H. (2011). AURA salutes the winners of the 2011 Nobel Prize in Physics, awarded for research conducted in part at AURA facilities. [online] Aura-astronomy.org. Available at: http://www.aura-astronomy.org/news/news.asp?newsID=272&newsType=1 [Accessed 21 May 2018].

[3] Berger, E., Kasliwal, M., Troja, E., Michaud, P., & Lifson, S. (2017, November 03). Astronomers Feast on First Light From Gravitational Wave Event. Retrieved from http://www.gemini.edu/node/12719

[4] https://alma-telescope.jp/en/news/mt-altiplanic_winter

[5] NSF Award #ACI-1451018 – IRNC: Backbone: AmLight Express and Protect (ExP), https://www.nsf.gov/awardsearch/showAward?AWD_ID=1451018&HistoricalAwards=false

[6] NSF Award # 1451024 – IRNC: RXP: AtlanticWave-Software Defined Exchange: A Distributed Intercontinental Experimental Software Defined Exchange (SDX) https://www.nsf.gov/awardsearch/showAward?AWD_ID=1451024&HistoricalAwards=false

[7] Florida International University (FIU), and Georgia Institute of Technology (GT) are implementing AtlanticWave-SDX, in collaboration with other exchange points supporting SDN

[8] BELLA-T (terrestrial network) – Currently, tenders are in course for completing the new DWDM 100G network linking South American NRENs by 2020