One would expect 2021 to start somewhat differently compared with chaos of the previous year. In Qrator.Radar, we also hoped for the better. Unfortunately, as soon as January 6 - today, we proved wrong.
One would expect 2021 to start somewhat differently compared with chaos of the previous year. In Qrator.Radar, we also hoped for the better. Unfortunately, as soon as January 6 - today, we proved wrong.
AS203, belonging to what was formerly known as "Level3", acquired by "CenturyLink" in 2016, latter rebranded as "Lumen" in 2020, is a frequent visitor within the incident reports of the Qrator.Radar team. We are not here to blame anyone, but such occurrence of routing incidents for a single organization is worrying - we hope this article would help you to understand how even a small event could reach enormous impact with specific prerequisites met.
On Tuesday, September 29, 2020 AS1221 - Telstra announced 472 prefixes in a BGP hijack event that affected 266 other ASNs in 50 countries, with the most damage rendered to the U.S. and U.K. based networks. Worldwide it affected more than 1680 IPv4 prefixes, creating almost 2000 path challenge conflicts.
On Sunday, August 30, 2020, it all started with a simple question: “What’s happening?”
Approximately around 10 UTC, the global Internet started experiencing a very specific state of connectivity - inside the network of one of the largest Tier-1 operators in the world, CenturyLink (primary AS3356), something bad was undoubtedly going on.
Yesterday, on August 24, 2020, Qrator.Radar BGP monitoring saw a rather large route leak originating from the AS42910 - Premier DC, containing 1403 prefixes mainly from the United States (571) and, peculiarly, Akamai. And then almost all the Western Asia region countries.
Before we start investigating what is happening with the Internet within and outside of Belarus, let us quote a couple of sentences we are repeating in annual National Reliability Research & Report:
“Strictly speaking, when the BGP and the world of interdomain routing were in the design stage, the creators assumed that every non-transit AS would have at least two upstream providers to guarantee fault tolerance in case one goes down. However, the reality is different; over 45% of ISP’s have only one connection to an upstream transit provider. A range of unconventional relationships among transit ISPs further reduces reliability. So, have transit ISPs ever failed? The answer is yes, and it happens with some frequency. The more appropriate question is — under what conditions would a particular ISP experience service degradation? If such problems seem unlikely, it may be worth considering Murphy’s Law: “Anything that can go wrong, will.”
Why are we repeating this rather than start with the facts and timesteps as usual? Because this is precisely the case, from our point of view, with Belarus’ internet segment. Let us take a look at two diagrams representing a BGP network of Belarus a month ago, at the beginning of July 2020:
On the border of July 29 and 30, depending on where in the world you were, a routing anomaly occurred. Following the NANOG question regarding what exactly was happening, Qrator.Radar team loaded the researching instruments and dived into the investigation. Nevertheless, before we start, let us take a general overview of that play's main actors.
In the morning of Tuesday, July 21 a Brazilian AS 264462 belonging to “Comercial Conecte Sem Fio Ltda me” as it is stated in the whois record for this particular ASN, leaked massive 13046 network prefixes in a networking incident that lasted for 1 hour and 23 minutes, starting at 9.15 UTC and ending at 10.38.
On April 23, 2020, an AS205310 leaked routes from one of its upstreams to another (from AS8220 to AS15943), affecting 90 000 prefixes.
In some cases, such an incident could lead to massive network degradation across dozens of ISPs. However, it did not. Why?
Because some companies install and maintain their filters properly. And even taking into regard the fact that AS15943 is directly connected to Tier-1 ISPs, they didn’t even notice the incorrect routes. They simply never reached Tier-1s, shrinking in size after each hop.
Today, on April 22, 2020, in the world of BGP routing, a thing that usually occurs in rare circumstances, happened. A year and 11 days ago, on April 11 2019, we wrote our first incident report about a thing that has never been observed before - a hijack by, with the highest probability, BGP optimizing software. Later that year, in summer, Cloudflare was brutally hit by the same type of incident. And today, a year after the first incident with AS263444 belonging to Open X Tecnologia Ltda, the same autonomous system… no, you guessed wrong.
Today it leaked 9328 prefixes from 1250 autonomous systems including all your favorite names: Akamai, Cloudflare, Vodafone, NTT, Amazon, NVIDIA and many others.
leaker | min_start_time | max_end_time | duration | prefix_count | origin_count | min_avg_max_propagation | max_duration
--------+------------------------+------------------------+----------+--------------+--------------+-------------------------+--------------
263444 | 2020-04-22 01:25:00+00 | 2020-04-22 01:47:00+00 | 00:22:00 | 9328 | 1250 | 2, 21, 176 | 00:22:00