Economists both inside and out of the Fed have long suspected the U.S. central bank would eventually lose money on the roughly $2.5 trillion in bonds it has bought since the financial crisis in an effort to stimulate the economy. But most expected those losses to be considerably smaller and not materialize for a few more years.

Back in February, former Fed governor Frederic Mishkin and three other economists predicted annual investment losses on the Fed’s bond portfolio to max out at $50 billion. And they didn’t expect those losses to start until 2016.

MORE: The next chairman should run the Fed like a business

An analysis around the same time by investment consultant MSCI put the Fed’s losses under a worst-case economic scenario at nearly $550 billion. If the economy were instead to gradually improve, as it appears to be doing now, MSCI said the red ink in the Fed’s bond portfolio would max out at $216 billion, a number the Fed is in spitting distance of already.

And even the current $151 billion estimate could on the low side. The Fed doesn’t give out a lot of information on its bond holdings other than the size of the portfolio and how much it is invested in Treasuries vs. mortgage bonds. In early May, right before interest rates started rising, the Fed had nearly $1.9 trillion in Treasuries and $1.1 trillion in mortgage bonds.

But in order to figure out how much Bernanke & Co. have lost, what you really need to know is how long it will be until the bonds the Fed has bought will be paid back. The longer that is the more a bond loses when interest rates rise. To figure a bond’s change in price, the general rule of thumb is to multiply its duration by the change in interest rates. And bond prices move in the opposite direction of rates. So a bond that will mature in one-year will lose one percent of its value when interest rates rise one percentage point. A 10-year bond would lose 10%.

The Fed doesn’t tell you the exact duration of its bonds, but what it does do is group its Treasury holdings. One-to-five-year bonds go in one group. Five-to-ten in another. Everything else gets put in the over 10-year category.

To calculate the Fed’s losses, I used an average duration — three years for the one-to-five year group, seven years for the next group, and 15 years for the 10-plus category — and multiplied that by the change in the rates since early May for each group — 0.43 percentage points for the one-to-five year, 0.96 for five-to-ten and 0.90 for the 10-plus. The result: The Fed’s Treasury holdings have lost $127 billion in value in the past seven weeks.

Figuring out losses on the mortgage bonds are a bit trickier. The Fed puts all of its Treasury bonds in the 10-year plus category, because it’s buying 30-year mortgages, and as anyone can tell you, 30 years is indeed more than 10. But as you may know from experience, few people end up sticking with their mortgage for 30 years, or even 10 for that matter. They refinance or sell their house or pay off their home loan in some other way. When mortgage rates are falling or the housing market is hot, loans tend to get paid off quicker. In recent years, banks have generally assumed the average duration of a mortgage is five years. Some banks have gone with as short as three.

MORE: How banks could get blown away by a bond bubble

But when interest rates rise, the assumption is that people will stick with their mortgages for much longer than they used to. So how long will it be until people pay back the ultra-low interest rate mortgages the Fed is now buying? I dunno. Seven years? Twelve? Twenty? Your guess is as good as mine.

So instead of going the duration route, I took the change in the Barclays mortgage-backed securities index, which has lost 2.2% since May 2nd, and applied that to the Fed’s $1.1 trillion in mortgage bonds to get a loss of $24 billion. Add that to the Treasury figure, and that’s how I got to $151 billion. Although you can see that the losses could be considerably bigger if you assume the bonds are longer than average, and given that the Fed has been trying to drive down the long end of the curve, that might be the case.

Even to the Fed, $151 billion is a lot of money. It’s nearly double the $83 billion the U.S. central bank paid to the Treasury last year in excess profits. That money went to lowering the national deficit, which, again for comparison, is expected to total $600 billion this fiscal year and includes all the money the government spent on everything it did, and is still just four times what the Fed lost on its bond portfolio in less than two months.

Does that mean that taxpayers will soon have to pay the piper for the Fed’s stimulus programs? Not quite. The rub is realized losses, of which the $151 billion are not. Unlike a regular bank, the Fed doesn’t have to recognize losses, an accounting move called marking-to-market, in its bond market portfolio until it actual sells. And Bernanke has said he has no plans to do so.

MORE: We’re in the early innings of a sell off

And the Fed has another bit of accounting magic working in its favor as well. Back in 2010, it struck a deal with the Treasury Department that stipulates that even if the Fed has realized losses it can essentially put those losses on layaway and pay them off when it’s profitable again.

What’s more, since the Fed isn’t planning on selling the bonds, any paper losses the Fed has could be offset by interest payments it continues to collect, which are about $20 billion a quarter. And remember the Fed has already booked about $200 billion in interest profits on the portfolio already.

And to be fair, when interest rates were falling and the value of the Fed’s bond portfolio was rising, no one, including me, bothered to point out how much money Bernanke & Co. were making on the bond portfolio as debt prices were rising. When Fortune computed its own estimate of how much money the government made on its bailouts, we only included the cash payments the Fed was making off the portfolio, not the investment gains or losses. Even at the time, Fortune estimated that the government had made a healthy profit on the bailout. And now we’d have to factor in the fact that the bailed out mortgage guarantors Fannie Mae and Freddie Mac are making profits again and turning that money over to the government.

So in the end there are roughly three ways to think about the Fed’s $151 billion in losses:

1. The Fed will use fancy accounting to make this all go away. (Isn’t the math of bailouts nifty.)
2. It’s just the initial bill for the Fed’s stimulus efforts that will end up costing taxpayers hundreds of billions of dollars, and it didn’t really work anyway.
3. Meh. The Fed is going to lose some money on its bond market portfolio, but put in the context of the overall crisis-related bailout efforts, which have made money, it won’t really matter.

My guess is the right way to think about this is N0. 3. But with the bond market still tanking, No. 2 will look like a real possibility to some. That’s the liability for the Fed.