On March 1 of last year, Ariel Gabizon was tidying up a presentation he was preparing to deliver the following day at a financial cryptography conference on the Caribbean island of Curaçao when he spotted a seemingly small mathematical mistake that could, he realized, jeopardize billions of dollars in capital.
An engineer for the Zerocoin Electric Coin Company, a startup known for creating the privacy-oriented cryptocurrency Zcash, Gabizon had identified an error in a seminal cryptography paper that served as a foundation for a host of virtual coins, including Zcash’s. The flawed paper described the mathematical underpinnings of certain “zero knowledge” proofs, a cryptographic breakthrough that enables the privacy features of Zcash as well as those of other digital currency projects. Ultimately, an attacker could have exploited the vulnerability to mint an infinite amount of counterfeit Zcash—as well as any other cryptocurrency that relied on its cryptographic technology—and no one would have been the wiser.
Bryce “Zooko” Wilcox, CEO and cofounder of the Zcash company, told Fortune on a call that his team patched the security hole in October, roughly eight months after its initial discovery. “We don’t believe that there was any exploitation of the vulnerability” on the Zcash blockchain, he said, noting that very few people knew the cryptography well enough to have discovered and exploited it. Moreover, the Zcash team has not seen any anomalously large transfers of Zcash cryptocurrency which might suggest nefarious conduct, he said.
The Zcash team, which conceded that it cannot be absolutely certain that the vulnerability wasn’t exploited, said it sought to balance security concerns against the risk of leaks in the lead-up to a coordinated disclosure Tuesday morning. The team limited the number of people in the know, used encrypted communications, and carefully selected confidantes to prevent rogue insiders, spies, or hackers from gaining knowledge of the vulnerability, which they could have exploited for their personal gain, Wilcox said.
While Zcash and a couple of the other top affected cryptocurrencies have patched their systems, not every project susceptible to the bug had a heads up. Indeed, some projects appear still to be vulnerable, raising questions about the proper way to handle vulnerability disclosure in the cybercoin era.
Coordinating a fix
When Zcash researchers first discovered the counterfeiting vulnerability, they faced a dilemma. They could disclose the bug immediately, inciting chaos and panic and opening a number of cryptocurrency-related projects—including theirs—to attack, or they could keep the bug between themselves, create a fix, and sneak it into a planned network upgrade, only later quietly looping in other affected parties
The team opted for the latter approach. A core group of four secret-keepers—all Zcash insiders—slipped a fix into Zcash’s so-called Sapling update on October 28, unbeknownst to anyone else, as far as they’re aware, they say. The initial confidantes were Gabizon, another Zcash researcher named Sean Bowe, Wilcox, and Wilcox’s brother Nathan, who is now the Zcash company’s chief technology officer.
The flaw is similar in outcome, though not in makeup, to other bugs that have affected Bitcoin as well as lesser-known projects, such as Monero, another privacy-focused cryptocurrency. Industry insiders have referred to these as “inflation bugs,” because they pose the risk of drastically increasing a cryptocurrency’s monetary supply through the minting of unlimited counterfeit cryptocurrency.
In this case, the vulnerability involved a faulty method for constructing “zk-SNARKs,” a particular implementation of zero knowledge proofs. By switching to another method for producing these proofs, the team was able to remove the poisoned code from Zcash, they said. (Some projects that implemented a popular, related cryptographic code library, called “libsnark,” a toolkit favored by the likes of Ethereum and others, were unaffected, they said.)
The vulnerability did not expose anyone’s private data, Wilcox said. Further, it did not impact the work done by some collaborators, such as the crew at JPMorgan Chase, which had partnered with the Zcash company on privacy technology, he said.
On November 13, a couple weeks after implementing the fix, Zcash researchers alerted security contacts at two other affected projects about the vulnerability. The groups were chosen in part because they represented the highest total market valuations for affected cryptocurrencies next to Zcash: Komodo, whose KMD tokens today amount to $72 million in total market value, and Horizen, formerly known as ZenCash, whose ZEN tokens amount to $22 million.
(Zcash, whose total market value exceeded $1 billion at the time of the vulnerability’s discovery, has since fallen to $150 million amid a global cryptocurrency market rout.)
“We’d like to thank the Zcash team for disclosing their technical concerns and for the coordination work,” said Maurizio Binello, a Horizen team member, noting that a software upgrade completed by January 18 resolved the issue. “We see this as an important sign of maturity for the whole industry.”
All three projects groups have since patched their code, but smaller projects remain vulnerable as of the publication of this article, including, apparently, Bitcoin Private, whose virtual coins boast a total market value of $18 million.
Left behind
Bitcoin Private and Zcash have a tangled, contentious history.
Bitcoin Private spun out from a Zcash-derived project, ZClassic, in February 2018. ZClassic itself split from Zcash a year earlier with the intention to create a rival, privacy-focused cryptocurrency that nixed Zcash’s so-called founders reward, a cryptocurrency payout that has helped to fund the Zcash company’s operations.
Sean Bowe, a Zcash researcher who helped discover the infinite counterfeiting vulnerability alongside Gabizon, told Fortune that the Zcash company saw “no way for us to responsibly disclose” the issue to the Bitcoin Private team prior to the public disclosure date. He said the Zcash team had initiated a 90-day deadline for public disclosure after notifying Komodo and Horizen of its finding, and that the team did not wish to share the details more broadly due to security concerns.
Bowe pointed to recent controversy embroiling Bitcoin Private as contributing to Zcash reluctance. Questions have swirled around the rival project since a report from Coin Metrics, a cryptocurrency research outfit, unearthed reportedly shady activity surrounding the origins of the cryptocurrency. Specifically, analysts have pointed to data indicating that someone involved in the project seems to have engaged in a covert pre-mine. (The team behind Bitcoin Private has agreed with the substance of the findings, but has claimed not to know who was responsible.)
(Fortune has reached out to Bitcoin Private; we will update this story when we hear back.)
Asked why he left some projects, like Bitcoin Private, out of the disclosure conversations, Wilcox cited security precautions. “We didn’t want to disclose to more parties until the majority of the exposed market cap had already been protected,” he said.
Outside counsel
The prospect of vulnerability disclosure can put security researchers in tricky situations.
When Heartbleed—one of the first big, branded web vulnerabilities—came to light, controversy erupted over who knew what when, and why some teams, caught scrambling in the patching mayhem, were not briefed sooner. The complications are amplified in the cryptocurrency industry, where you’re never sure who to trust and where financial incentives are directly baked into the technology itself.
Bruce Schneier, a well-known cryptography expert, said there’s no hard-and-fast rulebook when it comes to recovering from—and coordinating the release of details related to—sensitive security incidents. “There’s no ‘follow procedure 17-D,'” he said.
Emin Gün Sirer, a Cornell University professor who specializes in blockchain research, proposed an adaptation of the Hippocratic Oath as an overriding guideline. “The principle of utmost importance is to minimize harm and to minimize financial loss,” he said.
Neha Narula, executive director of the Digital Currency Initiative, a cryptocurrency-related academic project housed in the Massachusetts Institute of Technology, described the issue of coordinated vulnerability disclosure as “incredibly challenging” with respect to cryptocurrencies. “Knowing a vulnerability exists means you’re in a position where you can pretty easily and anonymously exploit it,” she said.
Narula, who is working on a paper about the issue, recommended as basic measures that cryptocurrency projects post clear vulnerability reporting guidelines, the email addresses of security contacts, and methods for secure communication, such as public encryption keys.
Dan Guido, CEO of Trail of Bits, a New York-based information security consultancy that does business in the blockchain industry, has personally encountered such roadblocks while coordinating fixes to cryptocurrency vulnerabilities. He has even published a crowdsourced document compiling points of contact at various projects.
But not every vulnerability coordination and disclosure has a simple resolution, as Zcash’s prolonged, behind-the-scenes bug fixing procedure demonstrates.
“It’s not clear yet what the best or right thing to do is,” Narula said. “I think have to work out standards and procedures as a community and that’s still in flux.”
