Cryptocalypse now: Bitcoin’s issue with ‘transaction malleability’
FORTUNE — The past month has been rough for bitcoin, full of arrests, hacks, and anticompetitive action. But the most complex and far-reaching incident was triggered when Mt. Gox, the oldest bitcoin exchange and one of the largest, froze all of its clients’ bitcoin assets as it tried to deal with what it called a “bug” in the Bitcoin protocol. The possibility of such a bug, which would undermine the trust at the core of Bitcoin’s value, sent bitcoin followers and figureheads into a flurry of recrimination and denial. What followed was a turbulent interaction between an overwhelmed business, a complex and evolving body of software code, and an idealistic, impassioned community whose Eden seemed to teeter on the verge of collapse.
It all started on Feb. 10, when Mt. Gox halted its clients’ ability to withdraw bitcoin. Mt. Gox issued a statement laying blame for the halt on so-called transaction malleability, which was referred to as “a bug in the Bitcoin software.”
This characterization was met with a wave of outrage from bitcoin watchers and those within the community, including Bitcoin lead developer Gavin Andresen. Andresen released a statement describing transaction malleability as a “technical detail” rather than a bug, pointing out that it had been known and understood by developers since 2011. Andresen blamed Mt. Gox’s problems, instead, on the exchange’s “highly customized wallet software” and its customer support practices. This war of words was made even more notable by the fact that Mark Karpeles, owner, CEO, and lead programmer of Mt. Gox, sits on the board of the Bitcoin Foundation with Andresen.
The Mt. Gox statement was also significant because it was immediately followed by malicious attacks across the bitcoin network, making use of the newly highlighted “technical detail.” Two major bitcoin exhanges — Bitstamp and BTCe — halted bitcoin withdrawals. Onlookers could have been forgiven for seeing a systemic collapse unfolding.
But what exactly is transaction malleability, and why did it lead to the freeze of hundreds of millions of dollars’ worth of virtual currency? Jeff Garzik is a core developer with the Bitcoin Open Source Project, the group that produces the most widely adopted version of the code that establishes and maintains bitcoin’s transaction records. According to Garzik, transaction malleability refers to users’ ability to change the transaction ID (TXID) that accompanies each bitcoin transaction, during a very short window and under certain limited conditions. “It’s not inherently a bug or flaw,” says Garzik. “Transactions can be mutated for valid reasons — for instance, crowdfunding processes require that multiple users collaborate on transactions, which can result in multiple transaction IDs.”
Because of its mutability, Garzik says that the main purpose of the TXID was never to act as a primary or definitive way for bitcoin wallets and other second-layer protocols to track transactions. Rather, transaction IDs were designed as an easy reference for support services. “[Processors] will issue this transaction ID, and if you have an issue with the transaction, you can call the support desk and tell them if the transaction didn’t arrive.”
The support desk seems to have been where Mt. Gox’s malleability issues became truly toxic. When customers reported that they had not received funds from transactions bearing a mutated TXID, Mt. Gox would have had no alternative way to track the transaction. It appears Mt. Gox’s support staff repeatedly disbursed bitcoin to customers who complained about missing transfers, even when a mutated TXID left them unable to know whether the funds had actually been disbursed or not. This would have left Mt. Gox’s ledgers increasingly out of balance with the public blockchain record. In other words, its reliance on identification information known to be unreliable may have left Mt. Gox without enough bitcoin to cover deposits.
But for those with better practices, transaction malleability has turned out to be a non-issue. Bitstamp, BTCe, and other operators that had halted trading to assess their systems were able to reopen within hours. Others avoided even brief problems. Among these was the bitcoin-based foreign exchange service Kraken, whose staff on Feb. 11 cheerfully tweeted: “Kraken halts nothing! Because planning. Carry on :)” According to Jesse Powell, CEO of Kraken parent company Payward, “We knew that relying on TXID would eventually cause problems, so we built a more robust system of accounting. We never had to make a fix because we planned ahead and didn’t take shortcuts in the beginning.” Kraken and many other services instead used rich information including transaction size, time, and recipient data to track funds received and disbursed.
Exchanges and merchants also found it easy to block the ensuing malicious attacks exploiting transaction malleability. These were, according to Garzik, both simple and relatively innocuous, simply involving one attacker cloning TXIDs, perhaps in hopes of scoring bogus withdrawals from help desk staff. “Any semi-informed programmer could have executed this [attack] with a single computer,” says Garzik. “It was not a ‘massive and concerted attack,’ as some media outlets have described it.” No assets were lost by any exchange other than Mt. Gox.
More importantly, the integrity of the Bitcoin network itself was never under threat. TXIDs are used only for tracking bitcoin transfers by second-layer software, and their malleability has no impact on the actual transfer of bitcoin. This is why the issue has not been a high priority for repair since it came to developers’ attention in 2011, and remains on the back burner even after the panic. “The core software itself, we made zero changes to it, and we plan to make zero changes to it,” says Garzik. While the price of bitcoin staggered immediately following Mt. Gox’s announcement, it rebounded as the limited nature of the problem became clear, and has remained stable since — though still at about 50% of its early-December high.
That same optimism most definitely does not apply to Mt. Gox specifically, or to CEO Karpeles. The transaction malleability problem came after a long string of other destabilizing setbacks for the exchange, most the result of leadership error. In May, the Department of Homeland Security seized some of Mt. Gox’s U.S.-linked accounts due to its failure to meet FinCEN compliance standards. As a result, Mt. Gox lost its relationship with reputable payment processor Dwolla. As it struggled to reestablish a reliable payments relationship, withdrawals of U.S. dollars were often delayed weeks or months, with little adequate explanation. It’s a testament to both the optimism and naivete of the bitcoin community that this elicited only a slow roil of discontent.
There is broad agreement that Mt. Gox’s histrionic public handling of its transaction malleability issues, attempting to lay blame at the feet of the Bitcoin code, were an attempt to distract from those other, still-ongoing problems. Mark Karpeles has earned the probably permanent distrust and ire of bitcoin users and supporters. This has included being targeted by protesters, which may have triggered Mt. Gox’s move to new, more secure offices Thursday.
As of press time, bitcoin withdrawals from Mt. Gox remain frozen. The most likely scenario is that Karpeles and his staff are working to determine how to retroactively confirm their transaction records. But if their system truly relies entirely on TXID information, that may not prove possible. Even if Mt. Gox is able to resume steady operation, it is sure to see huge customer attrition. Its fate, in a relatively unregulated market still highly reliant on trust, seems grim.
Mt. Gox’s issues point to something bigger. Karpeles himself is responsible for implementing and maintaining much of Gox’s code, while also overseeing operations and dealing with mounting regulatory issues as CEO. But over the past year, Bitcoin has transitioned away from its status as a niche, hobbyist market and become subject to much higher pressures and expectations. At its peak, Mt. Gox’s daily trade volume reached over 1 million bitcoins, or nearly a billion dollars. It’s little surprise that Mt. Gox’s one-man show couldn’t keep up with best practices in an exploding ecosystem. Karpeles and Mt. Gox, creatures of bitcoin’s rough-and-ready dawn, were simply left behind.
But by the same token, Mt. Gox’s fall, and the systemic threat it for a time seemed to have exposed, represents a trial by fire from which bitcoin has emerged, if not stronger, then at least more proven. “There were remedies in place which allowed the vast majority of the infrastructure to resume operation quickly,” says Matt Branton, a bitcoin entrepreneur who runs the retail content service Coinlock. “The way the community comes together when bugs are discovered tells you a lot more about the strength of bitcoin than this particular flaw does.” That its oldest servicer’s implosion has not taken the rest of bitcoin with it may be the greatest testament yet to the power of the technology, and the community that backs it.