FORTUNE — In a practice he began when running a hedge fund 50 years ago, Warren Buffett has always positioned his company, Berkshire Hathaway (BRKA), as being in annual competition with the S&P 500 index’s total return — and right now that’s not working too well for him.
Announcing Berkshire’s 2013 results this morning, Buffett said in his annual letter to shareholders that book value per share — Buffett’s standard yardstick in this competition — rose by 18.2% to $134,973. That’s a very solid performance. But the S&P 500 index (SPX), having its best year since 1997, had a huge total return of 32.4%.
Furthermore, though Buffett doesn’t specifically talk about this in his annual letter to shareholders, these results capped a five-year period, year-end 2008 to year-end 2013, in which the S&P 500 beat Berkshire’s gain in book value per share — the first such period in Berkshire’s history. For the five years, the S&P index jumped 128%. Berkshire’s book value per share rose by only 91%.
Buffett has in the past attached importance to the five-year record because he believes that book value is a rough indicator — directionally correct, but greatly understated — of how Berkshire’s intrinsic value is rising. If Berkshire’s intrinsic value doesn’t beat the S&P over the years, Buffett asks, why shouldn’t the company’s shareholders simply defect and buy a passive investment like the S&P index? (Please note how rarely you have heard a CEO ask his shareholders for this kind of close analysis.)
MORE: Buffett’s annual letter: What you can learn from my real estate investments
Obviously there’s nothing magic about a precise five years, and in his new letter Buffett in fact appends a year to the calculation. He points out that over the entire stock market cycle – between year-ends 2007 and 2013 — Berkshire outperformed the S&P. That would be because Berkshire, No. 5 in last year’s Fortune 500, is these days more of an operating company than a stock market investment vehicle while the S&P is, of course, pure stocks. The year 2008 was therefore a disaster for the S&P: Its return was minus 37%. Berkshire’s book value per share fell as well, but by only 9.6%. For the entire six-year period, Berkshire’s book value per share rose 73% to the S&P’s total return gain of 44%.
Buffett uses book value as a yardstick because it includes all capital gains — including those unrealized, which the more popular indicator of earnings does not. Even so, many press accounts of Berkshire’s 2013 record will surely report the earnings figures that the company released today, and these will be excellent. For the company’s A shares, earnings per share were up 32% to $11,850. That’s against yesterday’s stock-price close of $173,700.
Berkshire’s increase in pretax profits from $22.2 billion in 2012 to $28.8 billion in 2013 reflected $3.9 billion in realized investment gains (emanating from both equities and fixed-income securities) and generally favorable operating results throughout this hugely complex company. They showed up in insurance; at major subsidiaries Burlington Northern Santa Fe and MidAmerican Energy, and other industrial units; and in retail and housing. Derivatives, under intense shareholder surveillance for the last few years, contributed $2.6 billion to pre-tax profits in 2013 — up from about $2 billion the year before.
While Berkshire was selling stocks in 2013 to a minor degree, it was concurrently putting a slightly larger amount of net new money, about $4.7 billion, into them. Nearly $1 billion of that went into Buffett’s biggest holding, Wells Fargo (WFC), whose value to Berkshire at the end of the year was a colossal $22 billion. Buffett didn’t touch his share-count in two other monster stakes, Coca-Cola (KO) and American Express (AXP), and bought just a few more shares in a third, IBM (IBM).
MORE: Buffett lost nearly $900 million exiting biggest buyout ever
Buffett also used his letter to introduce another giant stationed in the wings: Bank of America (BAC). Berkshire owns warrants that entitle it to buy 700 million shares of BofA for $5 billion at any time prior to September 2021. And at year-end 2013, those shares were worth $10.9 billion. Bank of America, said Buffett in his shareholder letter, was therefore suiting up to become Berkshire’s fifth-largest equity investment — “and one,” he added, “we value highly.”
Overpowering all of these Berkshire details was the stock market surge of 2013, which carried the company’s mammoth equity holdings with it. Berkshire began the year with $87.7 billion in common stocks, added that $4.7 billion in net new money, and ended 2013 with $117.5 billion in common stocks. And that dollar figure doesn’t include Berkshire’s $12.25 billion investment last year in preferred stock, common stock, and warrants of Heinz. (For more about Buffett’s interest in Heinz, read this story by Fortune’s Stephen Gandel.)
Here’s another way to look at Berkshire’s common stock experience in 2013. At the beginning of the year, Berkshire had about $38 billion of unrealized gains on its common stocks. By Dec. 31, it had $61 billion.
Buffett is quick to give credit for Berkshire’s stock market gains to the two investment lieutenants he hired in the last few years, Todd Combs and Ted Weschler. Both, Buffett says in his letter, handily outperformed the S&P 500 in 2013. And each is now running a portfolio that exceeds $7 billion.
As he did a year ago, Buffett confesses that their investments outdid his own. If such ”humiliating comparisons” continue, he adds, “I’ll have no choice but to cease talking about them.”
Fortune senior editor-at-large Carol Loomis, who wrote this article, is a longtime friend of Warren Buffett’s, the pro bono editor of his annual letter to shareholders, and a Berkshire Hathaway shareholder.
