Google HQ

A year ago, several Google engineers got together and lay the foundation of Operation Rosehub, a project during which Google employees used some of their official work time to patch thousands of open source projects against a severe and widespread Java vulnerability.

Known internally at Google as the Mad Gadget vulnerability, the issue was discovered at the start of 2015 but came to everyone's attention in November 2015 after security researchers from Foxglove Security showcased how it could be used to steal data from WebLogic, WebSphere, JBoss, Jenkins, and OpenNMS Java applications.

Mad Gadget flaw sprawled across the entire Java ecosystem

According to researchers, the vulnerability was present in seven "gadget" classes inside the Apache Commons Collections library, versions 3.0, 3.1, 3.2, 3.2.1, and 4.0.

These seven vulnerable classes handle Java object deserializations, a basic operation, along with serialization, used to convert data from one format to another and transport it across a network.

The library, extremely popular among Java developers, was embedded in many closed and open sourced projects. A quick GitHub search shows the popularity of this library among Java developers. A study carried out by SourceClear in December 2015 found vulnerable versions of the library in over 70 other projects.

Big companies patched but the open-source world fell behind

After Foxglove's blog post, which also included proof-of-concept code, the Java world took notice, with Apache, Oracle, Cisco, Red Hat, Jenkins, VMWare, IBM, Intel, Adobe, HP, and SolarWinds, all issuing security alerts and patching their products.

"Unlike big businesses, open source projects don’t have people on staff to read security advisories all day and instead rely on volunteers to keep them informed," Justine Tunney, Software Engineer for Google's TensorFlow said.

"It wasn’t until five months later that a Google employee noticed several prominent open source libraries had not yet heard the bad news," she added. "Those projects were still depending on vulnerable versions of [Apache] Collections."

Google employees rally to help Java open-source community

Initially, that first Google employee started to patch the vulnerable open source projects on her own. This happened in March 2016. It didn't take the engineer long to discover how popular and widespread the problem was.

It was then when the employee started asking co-workers for help. As more and more employees joined in, they created a dedicated mailing list to coordinate their actions.

The group reached nearly 50 employees, and even their bosses joined in. They did this on company time because Google allows employees to spend 20% of their work day(s) for other projects.

Over 2,600 projects patched

"We felt that the severity of the vulnerability and its presence in thousands of open source projects were extenuating circumstances," Tunney said. "We recognized that the industry best practices had failed.

"Action was needed to keep the open source community safe. So rather than simply posting a security advisory asking everyone to address the vulnerability, we formed a task force to update their code for them. That initiative was called Operation Rosehub."

It took them months to patch over 2,600 open source projects they discovered on GitHub alone, even if some patches were mere one-line changes.

Mad Gadget still affects closed source projects

Despite this, the Rosehub team fears they haven't stamped out Mad Gadget from all places just yet.

"Going forward, we believe the best thing to do is to build awareness," Tunney says. "We want to draw attention to the fact that the tools now exist for fixing software on a massive scale, and that it works best when that software is open."

By revealing the work the Rosehub team did during the past year, Tunney wants companies and developers that work with closed source software, where Google couldn't reach, to patch the Mad Gadget flaw on their own.

This is important, as Mad Gadget was how a hacker took control of the San Francisco Municipal Railway system back in November 2016 and installed ransomware that impacted the city's public transportation for at least a day.

The same flaw was also found in the systems deployed at PayPal, most likely loaded through one of the vulnerable libraries.

The actions of Google's engineers show how vulnerable the open source community is to security flaws, and how important is it to notify project owners of any issues, instead of waiting for them to read security advisories (from other projects).