BGP event sends European mobile traffic through China Telecom for 2 hours
Traffic destined for some of Europe’s biggest mobile providers was misdirected in a roundabout path through the Chinese-government-controlled China Telecom on Thursday, in some cases for more than two hours, an Internet-monitoring service reported. It’s the latest event to stoke concerns about the security of the Internet’s global routing system, known as the Border Gateway Protocol.
The incident started around 9:43am UTC on Thursday (2:43am California time). That’s when AS21217, the autonomous system belonging to Switzerland-based data center colocation company Safe Host, improperly updated its routers to advertise it was the proper path to reach what eventually would become more than 70,000 Internet routes comprising an estimated 368 million IP addresses. China Telecom’s AS4134, which struck a network peering arrangement with Safe Host in 2017, almost immediately echoed those routes rather than dropping them, as proper BGP filtering practices dictate. In short order, a large number of big networks that connect to China Telecom began following the route.
The result: much of the traffic destined for telecommunications providers using the affected IP addresses passed through China Telecom equipment before either being sent to their final stop or being dropped during long waits caused by the roundabout paths. Traceroutes taken by Doug Madory, a security analyst at Oracle who first reported the leak, show just how circuitous the paths were. The following screenshot shows traffic starting at a Google Cloud server in Virginia passing through China Telecom’s backbone network before finally reaching its intended IP address located in Vienna, Austria.
A second screenshot shows a similar route between an Oracle data center in Toronto and an affected IP address in France.
Leak or hijacking?
It’s not clear if the mishap was an accidental leak or at least in some part an intentional hijacking. Some of the affected IP address blocks were smaller and more specific than those listed in legitimate announcements. Besides increasing the likelihood the modified announcement overrides the legitimate ones, the more specific routes may indicate use of route optimizers, which are designed to improve network traffic but can sometimes inadvertently result in the kind of route leaks seen on Thursday. What’s more, Safe Host is widely regarded as a trustworthy provider making it unlikely its faulty announcement was made intentionally.
On the other hand, China Telecom has a habit of accepting and propagating BGP announcements that later turn out to be improper. Last November, for instance, when a major African ISP updated tables in the Internet’s global routing system to improperly declare that its AS37282 was the proper path to reach 212 IP prefixes belonging to Google, the Chinese telecom accepted the route and announced it worldwide. The event intermittently made Google’s search and other services unavailable to many users and also caused problems for Spotify and other Google cloud customers. China Telecom has been especially suspect since last November, when Oracle’s Madory reported that it improperly misdirected big chunks of Internet traffic through its backbone for more than two years. As a result, traffic passing from California to Washington DC often traveled to Shanghai first. That incident involved China Telecom incorrectly handling the routing announcements of AS703, Verizon’s Asia-Pacific autonomous system.
“It’s hard to say definitively,” Rob Ragan, a principal security researcher at security consultancy Bishop Fox, told Ars in assessing whether Thursday’s routing incident was intentional. “It’s suspicious. Either way, that’s not good.”
Much of today’s Internet traffic is encrypted and that makes it difficult, if not impossible, for people who intercept it to read or modify its contents. Still, some security researchers theorize that BGP hijackers may in some cases be able to exploit weak encryption ciphers or use fraudulently obtained TLS certificates or other means to decrypt some of the traffic passing through their networks.
Such abilities may be the reason behind a series of previously reported BGP hijackings that, over the years, has routed the traffic of of financial institutions, government agencies, and network providers linkword through Russia.
Networks affected by Thursday’s event included Switzerland-based Swisscom’s AS3303, Netherlands-based telecom KPN’s AS1136, and AS1130 and AS21502, belonging to French telecommunications providers Bouygues Telecom and Numericable-SFR respectively. KPN later blamed the incident for causing a service outage that prevented many Dutch consumers from making debit card transactions. Some traffic for the Facebook-owned WhatsApp messaging service was also affected, researchers at network intelligence service ThousandEyes said.
Time for China Telecom to learn some MANRS
Some of the improper routes lasted for only minutes. Others stretched out for more than two hours. The unusually long timespan compounded the effects of the incident and also opened China Telecom up to criticism.
In a post detailing the incident Madory, who is director of Internet analysis of Oracle’s Internet intelligence team, wrote:
Today’s incident shows that the Internet has not yet eradicated the problem of BGP route leaks. It also reveals that China Telecom, a major International carrier, has still implemented neither the basic routing safeguards necessary both to prevent the propagation of routing leaks nor the processes and procedures necessary to detect and remediate them in a timely manner when they inevitably occur. Two hours is a long time for a routing leak of this magnitude to stay in circulation, degrading global communications.
A great place for any telecom to start improving their routing hygiene is to join the Internet Society’s Mutually Agreed Norms for Routing Security (MANRS) project.
Attempts to reach China Telecom officials for comment were unsuccessful. Safe Host representatives didn’t respond to an email. On Twitter, they wrote: “We are still investigating with our hardware supplier and CT on yesterday’s BGP leak, there was no configuration change on our side that triggered the issue.”
We are still investigating with our hardware supplier and CT on yesterday’s BGP leak, there was no configuration change on our side that triggered the issue.
— Safe Host SA (@swisscolo) June 7, 2019
Intentional or not, the incident underscores a fundamental weakness in BGP, which is the global routing table that allows an IP address belonging to one AS to locate an IP address belonging to a different AS. Decades ago, when the Internet was the province of hobbyists and researchers who largely knew each other, it was sufficient for the system to run on implicit trust. Nowadays, it’s clear that BGP has yet to adapt to an Internet that serves a much larger number of users, including profit-seeking criminals and nation-sponsored hackers.
And that means it’s up to individual networks to continuously police the address space allotted to them.
“This incident shows how ridiculously easy for a simple error to dramatically alter the service delivery landscape in the Internet,” Alex Henthorn-Iwane, vice president of product marketing at ThousandEyes, told Ars. “If you can’t see what’s happening, you can’t hold providers accountable and solve problems.”
The headline for this post was changed. Previously, it read: “BGP mishap sends European mobile traffic through China Telecom for 2 hours.”
