Apple called BS last week to the U.S. Mac sales numbers IDC and Gartner sent to clients two weeks earlier, AppleInsider‘s Daniel Eran Dilger reported Saturday. The two market research firms had sales falling. Apple had them growing by “strong double digits[s],” according to the earnings call transcript [see note below].
It’s not the first time Gartner and IDC have been caught with their methodological pants down. In November 2010 Asymco’s Horace Dediu was aghast to discover a 77 million unit discrepancy in their mobile phone data — data he depends on to draw his famous charts and graphs.
“Our methodologies, which are continually updated and improved, are among our most valuable assets,” according to Gartner, which goes on to describe them in the vaguest possible terms:
“Once an objective is set, analysts gather information through formal and informal surveys of IT users, technology providers and investors, business professionals, academicians and other researchers.” [From Inside Gartner Research]
It’s even squishier than that, according to a former IDC researcher who spent eight years with the company in the late 1990s and early 2000s.
After I posted a story about the 77 million missing phones, he described in some detail how IDC goes about estimating sales.
According to my source, who for legal reasons asked that his name not be used, here’s how it worked:
“Philip, I know something about this sausage-making process, as I used to do it for IDC. In 3Q98, I did a cross-section, an excavation, if you like, of our methodology, putting a crowbar to it and snapping it into fragments. It was the PC count, but the methodology is the same for phones.
“In most quarters, the team starts with OEM guidance and, depending on the country, does some by-country cross-checking. However, for the US team, we just did some systematic adjustments to the vendor guidance and called it a day. For example, we knew that lots of Macs were transshipped from Miami to Latin America. So, we took some percentage of Macs (Apple, of course, never helped; in fact, even objected, saying it wasn’t so) and reallocated them from the US to a smattering of Latin countries, effectively modeling the market but with no low-level data.
“So, in 3Q98, I analyzed the ‘choke points,’ those parts of the supply chain where the channel narrowed enough to get a definitive count. At the time, it was OS, processor, graphics, and hard drive. As I recall, I found 20 million processors with no homes. The market at the time was about 100 million, so this was a 20% discrepancy.
“The process that ensued was a marvel of obfuscation. The leader of the Tracker team figured out a way to rationalize away all the extra units (e.g., multiprocessor servers, inventory, speculation, etc.). It was politically impossible to force the extra units on the regions because it would introduce gross distortions to the historical trends.
“So, the mantra became, preserve the growth rates; to hell with the actual numbers. Even the growth rates are fiction. The fudge is in the “others” category, which is used as a plug to make the numbers work out. In fairness, we did do survey work, calling around, and attending white box conferences and venues to try to get a feel for that market, but in the end, the process was political. I used to tell customers which parts of the data they could trust, essentially the major vendors by form factor and region. The rest was garbage.
“The industry itself was aware of these issues, but agreed to maintain the fiction because it was convenient. Most vendors kept their own numbers, but referred to IDC for public purposes. Thing is, real executives got real compensation based on our numbers. There were other games played, but that’s for another time.
“The question always came down to, are these fiduciary numbers that investment managers charged with keeping money for widows and orphans rely on, or are they just inside baseball for the industry to squabble over and not all that important? That question has never been answered.”
[Note: The language Apple CFO Luca Maestri used in his prepared remarks — “We achieved strong double digit Mac growth across many countries, including the U.S., Canada, Mexico, the UK, Germany, France, Australia, China, India and the Middle-East” — is open to interpretation. Does the growth rate apply to the set of countries listed, or to every element in the set?
Apple spokesman Steve Dowling confirmed Monday that the growth was double digit (i.e. 10% or more) in each of those markets.]
Follow Philip Elmer-DeWitt on Twitter at @philiped. Read his Apple (AAPL) coverage at fortune.com/ped or subscribe via his RSS feed.
