Ormandy continued: A while later, we figured out how to reproduce the problem. It looked like that if an html page hosted behind cloudflare had a specific combination of unbalanced tags, the proxy would intersperse pages of uninitialized memory into the output (kinda like heartbleed, but cloudflare specific and worse for reasons I’ll explain later). My working theory was that this was related to their “ScrapeShield” feature which parses and obfuscates html – but because reverse proxies are shared between customers, it would affect *all* Cloudflare customers.
We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users. Once we understood what we were seeing and the implications, we immediately stopped and contacted cloudflare security.
This situation was unusual, PII was actively being downloaded by crawlers and users during normal usage, they just didn’t understand what they were seeing. Seconds mattered here, emails to support on a friday evening were not going to cut it. I don’t have any cloudflare contacts, so reached out for an urgent contact on twitter, and quickly reached the right people.
Could someone from cloudflare security urgently contact me.
— Tavis Ormandy (@taviso) February 18, 2017
After I explained the situation, cloudflare quickly reproduced the problem, told me they had convened an incident and had an initial mitigation in place within an hour.
“You definitely got the right people. We have killed the affected services”
In an update published later, Ormandy took issue with the post Cloudflare published. “It contains an excellent postmortem, but severely downplays the risk to customers,” he wrote. In a Twitter message, Ormandy said Cloudflare customers affected by the bug included Uber, 1Password, FitBit, and OKCupid. 1Password said in a blog post that no sensitive data was exposed because it was encrypted in transit.
Graham-Cummings, the Cloudflare CTO, has ruled out the possibility that secret keys for customers’ transport layer security certificates were exposed in the leaks. Still, he said end-user passwords, authentication cookies, OAuth tokens used to log into multiple website accounts, and encryption keys Cloudflare uses to protect server-to-server traffic were all at risk of being exposed. Cloudflare customers should at a minimum strongly consider changing passwords. Security researcher Ryan Lackey has other security advice here.
Cloudflare researchers have identified 770 unique URIs that contained leaked memory and were cached by Google, Bing, Yahoo, or other search engines. The 770 unique URIs covered 161 unique domains. Graham-Cummings said Thursday’s disclosure came only after the leaked data was fully purged, with the help of the search engines. Google cache, however, appeared to show data remained exposed by the bug, as evidenced by links such as this one, and social media threads including this one.