Yesterday, on February 19 Internet observed yet another demonstration of a handy Noction feature that is probably supposed to get you rich but is more likely to make you infamous.

Starting from 09:48 UTC, we saw around 200 thousand routes of previously non-existent prefixes with broken AS_PATH. But first things first.

 The day started with a rather harsh and buzzing sound of email notifications for critical routing events, which, as you can see, are cut off on such a high threshold that we consider those to be global.

We found out that all these global incidents have one common feature - affected prefixes have only local visibility. They were observed by only one of our speakers and appeared instantaneously.

Almost all of the routes were containing similar parts - “44393 14570 40676”. AS44393 - Securebit Route Collector, AS14570 - I Am A Bad Actor, LLC (we appreciated the imagination) and AS40676 - Psychz Networks (we loved the soundtrack). 

Example (Note: absent in RIPEDB https://stat.ripe.net/widget/routing-status#w.resource=189.203.175.128%2F25&w.min_peers_seeing=0 ):

For almost all of the prefixes, there was a corresponding correct route object (12000 different ISPs as origin ASN in total). Still, the number of prefixes with invalid ROA was larger than the number of valid ones. Someone (or somewhat) dumped a bunch of sub prefixes for valid prefixes and inserted accurate an ASN for them in AS_PATH.

Most of the miscreants (90%) were /25’s of valid  /24’s - a familiar sign of the routing “optimization” at work.

Although some extraordinary features let us alerted. Unlike common optimizer routes, almost all of the new routes were leading only to bad prefixes. But it could be explained because AS44393 is a Route Collector by itself and already has better routes to good prefixes.

More strangely - AS_PATHs to bad_prefixes contain more than 4000 new links that we have never seen before. So, unlike typical optimizers, as_path for new prefixes were not taken from valid ones but were crafted in place with valid ASN inserted at the end of AS_PATH (to pass ROA validation in case of max length threshold absence).

We wrote letters to people responsible for AS44393 and AS14570. They told us that they are on the receiving end, not the originating one, which leaves us with only AS40676. We were astonished to see AS40676 on an exciting webpage: https://www.noction.com/clients. That AS name (Psychz Networks) is listed as a customer of a BGP optimizer developing company. So, in the end, the root of the problem turned out to be it again.

Final question remains - why all the new prefixes did not spread over the world? The answer is simple: a bit of luck; since these bad routes were received by the route collector, which stopped their further spreading. Otherwise, by the time this article came out, the consequences are already breaking global news headlines.

What can we say? Route optimization is always a possible way to shoot yourself in the foot in the most inappropriate moment. And if you don’t use ROA with max_length, you would not notice that someone hurt you with this gun without third-party monitoring. But the happy final thought: you know the list of all gun owners.

Also, we want to alert all regional *NOGs members: please, if you are aware that in your region, someone is using BGP optimizer in day-to-day operations, make sure that you have all the necessary tools to prevent the consequences of such manipulation from your neighbour.