Services from Google on Monday became unavailable for up to two hours as user traffic followed a tortuous path through operators in Russia and Nigeria before hitting the Great Firewall of China.
This was the effect of an unintended anomaly that changed the normal traffic route towards some IP prefixes belonging to Google. At the heart of the issue was Nigerian ISP (AS37282) MainOne Cable Company, which leaked the prefixes to China Telecom, a government-owned provider.
To travel across the world, the internet traffic hops between multiple networks to reach the destination. The Border Gateway Protocol (BGP) is used to exchange the routing of the information based mostly on network policies and rules configured by a network administrator.
Network monitoring company ThousandEyes noticed problems when its offices could no longer connect to G Suite products from Google, and all the traffic dropped when it reached an edge router part of China Telecom's infrastructure.
Concern increased when they saw that Russian ISP TransTelecom (AS 20485) was also on the path, prompting the company to investigate the matter.
"Our analysis indicates that the origin of this leak was the BGP peering relationship between MainOne, the Nigerian provider, and China Telecom. MainOne has a peering relationship with Google via IXPN [Internet Exchange Point] in Lagos and has direct routes to Google, which leaked into China Telecom," informs ThousandEyes.
BGPmon, whose services focus on network monitoring and routing events, said on Twitter that the trouble started when Nigerian ISP MainOne leaked 212 IP prefixes to China Telecom; this led to redirecting the traffic and dropping it.
Appears that Nigerian ISP AS37282 'MainOne Cable Company' leaked many @google prefixes to China telecom, who then advertised it to AS20485 TRANSTELECOM (russia). From there on others appear to have picked this up.— BGPmon.net (@bgpmon) November 12, 2018
As ThousandEyes says, the "incident at a minimum caused a massive denial of service" to affected services. Furthermore, Google uses encryption to protect its traffic.
Traffic to Google was not the only one affected by the apparent misconfiguration at MainOne. BGPmon says that the Nigerian Autonomous System (AS) did the same mistake with IP addresses owned by Cloudflare, resulting in traffic taking a different route.
BGP is a solution from the 80s to help traffic reach its destination via autonomous systems (networks that handle their own block of IP addresses). It relies on a chain of trust, with routes being advertised by AS and trusted and accepted by its peers.
Although this automates routing the information over the internet, it also leaves room for traffic hijacking that can be malicious when intentional; as it seems to be the case now, the consequence was disrupting Google service due to improper configuration.
BGP's fragility is well known by the security industry, along with the possibilities of abuse from parties interested in traffic from specific regions of the world. A verification and filtering mechanism that would protect against both intentional and unintentional mishaps would be a solution for this decade-old problem.