January 27 of the year 2021 was marked with quite a peculiar route leak. AS61666 - GLOBO started announcing prefixes of its upstream provider MHNET - AS28146 to its another provider ALGAR - AS16735. In three minutes GLOBO leaked 1330 prefixes, and the whole routing incident lasted for 8 minutes - a time that was enough to create 1435 conflicts in 21 countries with 265 ASNs, mainly in Brazil (194 ASNs), United States (22 ASNs) and Venezuela (7 ASNs).
One would expect 2021 to start somewhat differently compared with chaos of the previous year. In Qrator.Radar, we also hoped for the better. Unfortunately, as soon as January 6 - today, we proved wrong.
AS203, belonging to what was formerly known as "Level3", acquired by "CenturyLink" in 2016, latter rebranded as "Lumen" in 2020, is a frequent visitor within the incident reports of the Qrator.Radar team. We are not here to blame anyone, but such occurrence of routing incidents for a single organization is worrying - we hope this article would help you to understand how even a small event could reach enormous impact with specific prerequisites met.
On Tuesday, September 29, 2020 AS1221 - Telstra announced 472 prefixes in a BGP hijack event that affected 266 other ASNs in 50 countries, with the most damage rendered to the U.S. and U.K. based networks. Worldwide it affected more than 1680 IPv4 prefixes, creating almost 2000 path challenge conflicts.
On Sunday, August 30, 2020, it all started with a simple question: “What’s happening?”
Approximately around 10 UTC, the global Internet started experiencing a very specific state of connectivity - inside the network of one of the largest Tier-1 operators in the world, CenturyLink (primary AS3356), something bad was undoubtedly going on.
Yesterday, on August 24, 2020, Qrator.Radar BGP monitoring saw a rather large route leak originating from the AS42910 - Premier DC, containing 1403 prefixes mainly from the United States (571) and, peculiarly, Akamai. And then almost all the Western Asia region countries.
Before we start investigating what is happening with the Internet within and outside of Belarus, let us quote a couple of sentences we are repeating in annual National Reliability Research & Report:
“Strictly speaking, when the BGP and the world of interdomain routing were in the design stage, the creators assumed that every non-transit AS would have at least two upstream providers to guarantee fault tolerance in case one goes down. However, the reality is different; over 45% of ISP’s have only one connection to an upstream transit provider. A range of unconventional relationships among transit ISPs further reduces reliability. So, have transit ISPs ever failed? The answer is yes, and it happens with some frequency. The more appropriate question is — under what conditions would a particular ISP experience service degradation? If such problems seem unlikely, it may be worth considering Murphy’s Law: “Anything that can go wrong, will.”
Why are we repeating this rather than start with the facts and timesteps as usual? Because this is precisely the case, from our point of view, with Belarus’ internet segment. Let us take a look at two diagrams representing a BGP network of Belarus a month ago, at the beginning of July 2020:
On the border of July 29 and 30, depending on where in the world you were, a routing anomaly occurred. Following the NANOG question regarding what exactly was happening, Qrator.Radar team loaded the researching instruments and dived into the investigation. Nevertheless, before we start, let us take a general overview of that play's main actors.
In the morning of Tuesday, July 21 a Brazilian AS 264462 belonging to “Comercial Conecte Sem Fio Ltda me” as it is stated in the whois record for this particular ASN, leaked massive 13046 network prefixes in a networking incident that lasted for 1 hour and 23 minutes, starting at 9.15 UTC and ending at 10.38.
On April 23, 2020, an AS205310 leaked routes from one of its upstreams to another (from AS8220 to AS15943), affecting 90 000 prefixes.
In some cases, such an incident could lead to massive network degradation across dozens of ISPs. However, it did not. Why?
Because some companies install and maintain their filters properly. And even taking into regard the fact that AS15943 is directly connected to Tier-1 ISPs, they didn’t even notice the incorrect routes. They simply never reached Tier-1s, shrinking in size after each hop.