On Sunday of April 5, 2020, only a few days after last week route leaks an AS7552 belonging to Viettel - according to Wikipedia the largest telecommunication service provider in Vietnam - was leaking routes for more than 3 hours in a row.
The leak affected 4825 network prefixes from 326 operators, spreading from AS7552 upstreams: AS3491 and AS4637 towards AS1273 - Vodafone, which helped spread it to almost all major Tier-1 ISPs. Most of all Vietnamese, Cambodian and Australian networks were affected, with more than 25% of ISPs in the first two countries.
country | prefix_count | origin_count | origins_in_country ----------------------------------+--------------+--------------+-------------------- Australia | 1469 | 107 | 2630 Viet Nam | 1932 | 104 | 365 Cambodia | 892 | 35 | 128 Hong Kong | 23 | 15 | 987 United States | 49 | 14 | 27589 Lao People's Democratic Republic | 64 | 9 | 29 India | 20 | 8 | 2574 Taiwan, Province of China | 8 | 7 | 336 Thailand | 5 | 4 | 527 New Caledonia | 12 | 3 | 14 Singapore | 4 | 3 | 554 Timor-Leste | 28 | 2 | 12 New Zealand | 2 | 2 | 643 Indonesia | 2 | 2 | 1755 Bangladesh | 2 | 1 | 986 China | 8 | 1 | 1715 France | 3 | 1 | 1873 Korea, Republic of | 1 | 1 | 924 Malaysia | 4 | 1 | 278 Myanmar | 4 | 1 | 99 Nepal | 1 | 1 | 114 Pakistan | 1 | 1 | 198 Philippines | 239 | 1 | 454 United Kingdom | 1 | 1 | 2822 Vanuatu | 1 | 1 | 13 Brazil | 1 | 1 | 7691
Interestingly enough, this case proves what we always see - the provider believes the customer - and the leak goes all the way through to the core of the internet with the help of AS7552 current upstreams. During the incident investigation, we also wanted to check how many Tier-1 operators accept the route containing another Tier-1 on a not adjacent place in AS-PATH, and it turned out that all of them do, except for NTT and GTT. Every other Tier-1 accepted this leak through peering with Telia - AS1299. We thought that Telia would drop such a route because maintaining such a filter from customer direction is relatively easy. And yet, it didn’t.
This is a part of the data from the RIPE analyze tool that illustrates routing history where we see that one of the prefixes that had only been seen locally, however, at the time of the leak, it became available globally: https://stat.ripe.net/widget/routing-history#w.resource=118.107.68.0%2F23.
The BGPlay widget also highlights what exactly happened with this prefix during the incident.
As the quarantine measures continue almost across the globe, the pressure under which networks continue to work only rises. It is now only a matter of time when the next networking incident occurs because the human factor is actually helping to cope with the consequences of rapidly growing traffic amounts. Engineers would still need to make configuration changes, and yes, sometimes it could go wrong.
We could only, again, point to the necessity of monitoring networks in real-time, because even hour-long outages could cause severe damages and failure cascades for entire businesses.