The Internet broke today: Facebook, Verizon, and more see major outages

Enlarge / The Internet this week, more or less

Last week, Verizon caused a major BGP misroute that took large chunks of the Internet, including CDN company Cloudflare, partially down for a day. This week, the rest of the Internet has apparently asked Verizon to hold its beer.

Cloudflare went down again for half an hour yesterday, and this time, it was the company’s own fault—we’re still waiting on a full post-mortem, but the short version is that a firewall regular expression rule targeting malicious Javascript spiked the firewalls’ CPU usage, crippling throughput and causing widespread HTTP 502 errors. Microsoft’s Office365 also seems to have experienced a multi-hour partial outage yesterday, with the service working over some ISPs and routes but not others for about four hours.

Enlarge / Office365’s partial outage on July 2 was reported by third-party users at downdetector.com

Facebook and its properties WhatsApp and Instagram have suffered widespread outages relating to image display for most of today. The problem seems to be bad timestamp data being fed to the company’s CDN in some image tags; when I looked into the broken images littering my own Facebook timeline, I discovered different timestamp arguments embedded in the same URLs. Loading an image from fbcdn.net with bad “oh=” and “oe=” arguments—or no arguments at all—results in an HTTP 403 “Bad URL timestamp”.

If you dig into the code, Facebook refers to the same image URL with different arguments in its anchor tags and its image tags; in the posts with broken images, the URLs in the anchor tags work, but the ones in the image tags don’t. (On the plus side, now everybody is suddenly noticing the AI-driven hidden alt tags the company has been embedding in its images since 2017.)

Twitter has suffered some as-yet-unexplained hiccups in its direct messaging service today as well. The outages appear to be mostly in the Eastern United States and Europe, with few or no reports shown in other regions on several third-party outage-tracking sites.