Typo in Cloudflare Server Source Code Leaks Customer Info, Cookies, Passwords

A typo in the source code of a Cloudflare component has exposed the personal information of users visiting sites protected by Cloudflare's service, along with potentially more sensitive details such as cookies, passwords, authentication tokens, API keys, and others.

CloudFlare CTO John Graham-Cumming acknowledged the issue yesterday and said it fixed the underlying problem.

One character caused this entire ruckus

According to the company's incident post-mortem, the bug was caused by a typo in the source code of the Cloudflare HTML Parser component, a module the company uses to read a website's source code, and then pass it over to other modules that rewrite its content, based on the user's account settings.

The typo that caused this issue was that someone used ">=" instead of "==". This led to a situation where the HTML parser caused a buffer overflow, which then dumped the content of Cloudflare server memory into the client's HTTP requests.

According to Cloudflare investigation, the issue appeared only when customers enabled three settings in their account named Email Obfuscation, Automatic HTTPS Rewrites and Server-side Excludes.

Bug active for five months

A later investigation revealed the typo was introduced in the HTML Parser component on September 22, 2016. Cloudflare resolved the issue on February 18, 2017, when a Google engineer reported the problem.

"The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests)," said Cloudflare's Graham-Cumming.

The bug came to light last Saturday when Google Project Zero researcher Tavis Ormandy noticed irregular data output in mundane data processing operations.

A further investigation of this data revealed that certain sites, which he later identified as protected by Cloudflare's reverse proxy system, were including more information than usual in their HTTP requests.

The researcher says he reached out to CloudFlare via Twitter and immediately canceled his weekend plans due to the bug's severity.

Issue mitigated 47 minutes after disclosure

Cloudflare started working on the issue minutes after receiving details from Ormandy, and 47 minutes later had disabled Email Obfuscation for all customers, and later the Automatic HTTPS Rewrites feature.

Six hours after, the company identified the typo and reached out to all major search engines, as some of the leaked data had been cached in search results.

In his Project Zero bug report, Ormandy posted three screenshots of the leaked data, portraying HTTP requests for services such as Uber, OKCupid, and Fitbit. This was done to illustrate the bug's impact.

"I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings," Ormandy wrote about just a fraction of the data he was finding via the Cloudflare leak. "We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything."

The bug's impact is noticeable because Cloudflare is one of the largest web firewall and reverse proxy systems on the market.

To assess the bug's reach, a user has started compiling a list of sites protected by Cloudflare's service. At the time of writing, the list includes 4,287,625 possibly affected domains. BleepingComputer is on this list, but as the three features were not enabled, the site was not affected.

Small chance the issue was exploited

Ormandy, Cloudflare, and many security researchers doubt that someone exploited Cloudflare's bug.

"I don't know if this issue was noticed and exploited, but I'm sure other crawlers have collected data and that users have saved or cached content and don't realize what they have," Ormandy said.

While Cloudflare has tried to downplay the issue's impact, Ormandy doesn't agree with the company's stance.

"It contains an excellent postmortem, but severely downplays the risk to customers," the researcher said about Cloudflare's blog post.