(Yet another one) CenturyLink BGP incident and the blinking Internet
Radar

On Sunday, August 30, 2020, it all started with a simple question: “What’s happening?”

A downdetector.com screenshot at the beginning of the incident

Approximately around 10 UTC, the global Internet started experiencing a very specific state of connectivity - inside the network of one of the largest Tier-1 operators in the world, CenturyLink (primary AS3356), something bad was undoubtedly going on.

Read more
AS42910 leaking hundreds of prefixes, affecting Akamai and Western Asia region
Radar

Yesterday, on August 24, 2020, Qrator.Radar BGP monitoring saw a rather large route leak originating from the AS42910 - Premier DC, containing 1403 prefixes mainly from the United States (571) and, peculiarly, Akamai. And then almost all the Western Asia region countries.

Read more
What is happening with the BY internet segment in terms of BGP and IPv4/IPv6
Radar

Before we start investigating what is happening with the Internet within and outside of Belarus, let us quote a couple of sentences we are repeating in annual National Reliability Research & Report

“Strictly speaking, when the BGP and the world of interdomain routing were in the design stage, the creators assumed that every non-transit AS would have at least two upstream providers to guarantee fault tolerance in case one goes down. However, the reality is different; over 45% of ISP’s have only one connection to an upstream transit provider. A range of unconventional relationships among transit ISPs further reduces reliability. So, have transit ISPs ever failed? The answer is yes, and it happens with some frequency. The more appropriate question is — under what conditions would a particular ISP experience service degradation? If such problems seem unlikely, it may be worth considering Murphy’s Law: “Anything that can go wrong, will.”

Why are we repeating this rather than start with the facts and timesteps as usual? Because this is precisely the case, from our point of view, with Belarus’ internet segment. Let us take a look at two diagrams representing a BGP network of Belarus a month ago, at the beginning of July 2020:

Read more
AS10990 and the peerless CDN combined with routing optimization tale
Radar

On the border of July 29 and 30, depending on where in the world you were, a routing anomaly occurred. Following the NANOG question regarding what exactly was happening, Qrator.Radar team loaded the researching instruments and dived into the investigation. Nevertheless, before we start, let us take a general overview of that play's main actors.

 

Read more
264462 massive route leak
Radar

In the morning of Tuesday, July 21 a Brazilian AS 264462 belonging to “Comercial Conecte Sem Fio Ltda me” as it is stated in the whois record for this particular ASN, leaked massive 13046 network prefixes in a networking incident that lasted for 1 hour and 23 minutes, starting at 9.15 UTC and ending at 10.38.

 

Read more
A different route leak species
Radar

On April 23, 2020, an AS205310 leaked routes from one of its upstreams to another (from AS8220 to AS15943), affecting 90 000 prefixes.

In some cases, such an incident could lead to massive network degradation across dozens of ISPs. However, it did not. Why?

Because some companies install and maintain their filters properly. And even taking into regard the fact that AS15943 is directly connected to Tier-1 ISPs, they didn’t even notice the incorrect routes. They simply never reached Tier-1s, shrinking in size after each hop.

Read more
AS263444 hitting the headline again
Radar

Today, on April 22, 2020, in the world of BGP routing, a thing that usually occurs in rare circumstances, happened. A year and 11 days ago, on April 11 2019, we wrote our first incident report about a thing that has never been observed before - a hijack by, with the highest probability, BGP optimizing software. Later that year, in summer, Cloudflare was brutally hit by the same type of incident. And today, a year after the first incident with AS263444 belonging to Open X Tecnologia Ltda, the same autonomous system… no, you guessed wrong.

 

Today it leaked 9328 prefixes from 1250 autonomous systems including all your favorite names: Akamai, Cloudflare, Vodafone, NTT, Amazon, NVIDIA and many others.

 leaker |     min_start_time     |      max_end_time      | duration | prefix_count | origin_count | min_avg_max_propagation | max_duration 
--------+------------------------+------------------------+----------+--------------+--------------+-------------------------+--------------
 263444 | 2020-04-22 01:25:00+00 | 2020-04-22 01:47:00+00 | 00:22:00 |         9328 |         1250 | 2, 21, 176              | 00:22:00
Read more
Weekend route leak by AS7552
Radar

On Sunday of April 5, 2020, only a few days after last week route leaks an AS7552 belonging to Viettel - according to Wikipedia the largest telecommunication service provider in Vietnam - was leaking routes for more than 3 hours in a row.

 

The leak affected 4825 network prefixes from 326 operators, spreading from AS7552 upstreams: AS3491 and AS4637 towards AS1273 - Vodafone, which helped spread it to almost all major Tier-1 ISPs. Most of all Vietnamese, Cambodian and Australian networks were affected, with more than 25% of ISPs in the first two countries.

 

Read more
This is how you deal with route leaks
Radar

That, we must say, is the unique story so far.

Here’s the beginning: for approximately an hour, starting at 19:28 UTC on April 1, 2020, the largest Russian ISP — Rostelecom — was announcing prefixes belonging to prominent internet players: Akamai, Cloudflare, Hertzner, Digital Ocean, Amazon AWS, and other famous names.

Before the issue was resolved, paths between the largest cloud networks were somewhat disrupted — the Internet blinked. The route leak was distributed quite well through Rascom (AS20764), then Cogent (AS174) and in a couple of minutes through Level3 to the world. The issue suddenly became bad enough that it saturated the route decision-making process for a few Tier-1 ISPs.

Read more
Serious Times — Serious Leaks
Radar

At 17:13 UTC on March 31, 2020, the AS50048 (NEWREAL-AS) leaked, in total, 2658 IPv4 network prefixes to the Tier-2 transit provider Transtelecom. Those prefixes included Orange, Akamai, Rostelecom and more than 300 other companies’ networks.

Read more