My Journey to the Center of the Internet
Well, one of the them, anyway. On the mind-blowing mundanity of AT&T’s global network operations center.
At the entrance to AT&T’s global network operations center—the telecom giant’s global brain—a sign greets guests. Orange-red dots speckle the wall to form a message. The typeface calls to mind an early computer-era punched card:
It’s a fitting salutation to emblazon the company’s worldwide command center. Before I have time to dwell on the phrase—whoosh—a doorway to my right slips open. So begins my descent into the belly of the Bell.
I stand outside a passageway where the voices of a ghostly choir spill out, chanting from concealed speakers. (The music sounds like it could have been ripped from the original Halo video game soundtrack on Microsoft’s Xbox.) Light fixtures shimmer overhead, arranged like vertebrae along the length of the corridor.
In this center—and it is the center, for AT&T at least—the company monitors the activity and performance of its global infrastructure—from the cellular masts that punctuate the landscape, to the microwave arrays that beam radiation to and fro, to the undersea cables that cling to the ocean’s bottom and anchor distant lands. Here, AT&T’s technicians ensure that the bits and bytes exchanged through its network by people all over the world—the phone calls, family photos, software updates, financial transactions, and so forth—arrive on time and intact.
Without this central overseer, dropped calls, damaged lines, and lost data would be the norm. Digital life would grind to a halt. And so the company’s engineers keep a watchful eye on the information machinery that connects the continents, inspecting the otherwise invisible knitting that ties phone to wire, message to reply, mind to mind, across untold chasms.
Hello world, indeed.
I arrive on the sprawling grounds of AT&T’s former corporate headquarters on a rainy fall morning. The campus is tucked away in Bedminster, a sleepy suburb of New Jersey about 50 miles west of New York City. One of the township’s few claims to fame is that it contains a historic revolutionary war battleground where the nation’s first artillery assembled. (That, and the fact that it’s home to a 600-acre golf course owned by Donald Trump.)
The two-decade-old structure I’ve come to visit is impressive. It is built into the side of a hill, buried halfway inside the Earth—a bunker-style command center. I can hardly think of a better place to marshal the front lines of one of the world’s largest telecom carriers.
When I enter the center Stephen Moser, the hub’s resident tour guide, greets me and leads me down a hall. He points to several screens and tosses off various facts and figures. We stroll past a set of motion-sensing tables bearing light projections that blink and chirp obligingly. He waves his hand over one surface. The gesture sparks a presentation about the explosive number of connected devices soon coming online.
Gartner, the market research firm, forecasts that the number of so-called smart things—that is, Internet-connected objects—will surge to 20.8 billion by 2020, up from 6.4 billion this year. That figure is rapidly approaching and will soon blow past the world’s population of 7.4 billion. Techies refer to this impending onslaught as the “Internet of things.”
Judging by the surroundings, I glean we’re nearing a fully connected future sooner rather than later.
Moser, for his part, has his introductory performance down pat. He has worked at AT&T for 35 years, he says. He joined the company (T) shortly before United States courts ruled the queen of communications a monopoly and ordered it broken up in 1982. “I don’t think anyone saw that coming,” he says with a touch of saudade. The man has seen the company undergo incredible transformation since. Not least: A transition from the point-to-point telephone lines of yore to the open mesh of fiber optic cables, submarine cords, and computers that undergirds today’s Internet. He has witnessed it all.
Get Data Sheet, Fortune’s technology newsletter.
Moser basks in the company’s heritage. On his desk rests a black leather-bound plant engineer’s handbook from the 1930s, an artifact from a bygone industrial era. Beside that squats a fiber cable splicer mounted on a wooden plaque with a gold engraving, a device that helps physically forge new Internet links. Moser says he conducts more than 200 tours per year here at the global network operations center, or GNOC (pronounced gee knock), for short. That’s about one per business day.
Moser invites me to sit down to watch a short film about the company’s digital dominance. The flick describes how AT&T is bringing the world together with its technology. It’s glossy, clever marketing—exactly what one might expect of a company that brought in $146.8 billion in revenue last year, placing it at number 12 on the Fortune 500 list, higher than any other in the telecommuications industry. (Verizon (VZ), next in line, clocks in at No. 15.) At the clip’s close, the projection screen whisks away. Light spills in. A subterranean room is revealed.
Moser and I stand overlooking a chamber where AT&T engineers quietly tinker; from this vantage point in the building’s private theater, we oversee all.
Enter the Matrix
Earlier in the year I had met with Ed Amoroso, AT&T’s security chief. He had told me about the company’s transition to software-defined networks, virtualized infrastructure that affords greater flexibility to its operators. In the new paradigm, code dictates the flow of data more so than circuits; packet-switching becomes more easily programmable; routing traffic gets quicker, more efficient, and more automated. (Fortune spoke to AT&T CEO Randall Stephenson about the conversion last year.) The company had invited me to take a peek behind the scenes to witness where the magic happens: How AT&T maintains the network that so many people and businesses depend on each day. This is what I came to see.
The cavernous space below Moser and I—as well as the fanfare leading up to its reveal—is clearly designed to impress business partners, potential customers, clients, employees, and visiting journalists. “We drew inspiration from theme parks,” Moser admits. He name-checks SeaWorld. Behind the scenes of the main attraction, however, resides a more prosaic allotment of cubicles, tucked away in the recesses of the center.
This subterranean room, on the other hand, is magnificent. Its main wall consists of a patchwork of television screens. The place vaguely smacks of the scene in the 2003 film The Matrix Reloaded, where Neo finally meets “the architect,” the stylishly dressed old man who—spoiler alert—wrought the virtual reality in which all humanity lives. Except the AT&T version has bigger screens. The room accommodates about 70 desks rather than just one chair, and it is less blindingly white.
“There’s a certain military-like precision to this place,” Moser says, as we descend a few staircases to the bottom floor. “Everybody’s got a role. Everybody knows what to do. If ‘X’ goes wrong, they know exactly what to put in place to correct it.” Of course, most responses are automated nowadays, he says. If a spike in network traffic begins to take shape, for example, AT&T’s systems will (ideally) detect and put the kibosh on it. Such a swelling might signal the start of a distributed denial of service attack, or a hacker’s botnet blitz. It’s best to quash such electronic tsunamis immediately, before they do damage.
Cybersecurity is very much a top concern for the team. (The company’s history of close ties to the nation’s security and intelligence apparatus has been well documented.) “We start by looking at the volume of traffic and that leads to a deeper analysis of behavioral patterns,” Moser says. He describes the insights gleaned at the center as those that might be had in a helicopter observing the flow of metropolitan traffic in an urban center. AT&T’s technicians don’t inspect the content of messages, just the aggregate activity, Moser assures me. “Then we can make some determination as to whether there may be some cyber trouble ahead.”
Moser and I amble around the perimeter of the space. The squeak of an ergonomic chair wheel echoes off the 4,200 square feet of screen-laden walls. A bespectacled technician glides between desks, points at a desktop monitor, exchanging murmurs in the glow of the banks of 140 or so television and computer screens. His colleagues clatter away on their keyboards. For a space coursing with so much digital din—an epicenter of electronic communications—the room is mostly silent.
“It never gets any noisier than this,” Moser says, sotto voce.
Signal and Noise
Prominently placed at the room’s center is a world map that depicts whether and where there may be any latency on any segment of AT&T’s network. This is the god’s-eye view of the company’s infrastructural well-being. Off to the left and right are more monitors. One analyzes the sentiment of social media posts that mention AT&T, Moser tells me. “I hate AT&T!” would count as a bad comment, naturally. “If people are cursing us out on social media, we’ll zero in on that,” he says.
As the company moves toward more software defined networking, these ears on the ground become increasingly more important. Hardwired Internet routers are getting replaced with more elastically programmable ones, but issues can arise in the process. Scoping out customer grousing can help spot non-obvious software problems as they arise. AT&T’s research and development labs provide the operations center with tools to help them pinpoint disgruntled customers, and address possible issues. “There have been plenty of examples where all of our network telemetry is telling us we’re green across the board, and the customer is screaming at us and that map has got red all over it,” Moser says. “Basically, we have a heat map of customer happiness.”
If you were to share this story online, in other words, dear reader, AT&T would take note. Posting a negative comment should make the company’s rancor-o-meter tick upwards. Including praise should make its satisfaction bar pop. (Go ahead and give it a try! Share your reactions on Twitter, Facebook, or LinkedIn—why not?)
Moser points out other screens showing the burden on the company’s customer care call centers, the health of its underwater cable systems, the reach of its cellular transmission towers. Some monitors are tuned to news stations, keeping the workers abreast of world events that might impact global communications. AT&T’s engineers track storms too, overlaying weather radar atop maps of the network’s topology. This allows them to forecast tempests’ paths and anticipate points of vulnerability.
To the outsider’s eye, most of these screens look like a Babel of incomprehensible charts. Many are checkered with hieroglyphs, an obscure lexicon known only to network engineers. The workers glance upward every now and then at magenta hourglass icons and traffic light-shaded dotted lines. They decipher them effortlessly, periodically surveying whatever might happen to be affecting the network. As aforementioned, most of the responses to network problems are automated these days. Failsafe measures kick in whenever something goes awry. Anomalies, accidents, and disasters are, generally, taken care of.
A major blizzard hits the coast of the northeastern U.S.? You might see a slight uptick in traffic. On the day of the Super Bowl, activity might decrease during the game, while everyone is glued to their TV sets, and then resurge afterward. During the last World Cup, Brazil, the host country, was an international thoroughfare coursing with off-the-charts Internet traffic. For the most part, however, barring these exceptions, the days of uneven traffic spiking are past, Moser says. Holidays no longer correlate to perceivably increased activity. There is a constant murmur, a perpetual chatter reverberating throughout the network, he says. Humanity tends to its computers, its tablets, its phones and devices 24/7.
Eric Ryan, the AT&T representative who accompanied us on this tour, mentions that when he visited the center several months earlier, he witnessed Chris Christie, governor of New Jersey, announce what would become a short-lived bid for presidency. Ryan says he asked where on the screens—which spurts—might indicate the chatter surrounding that event. Surely there must be a spike, a sputter, somewhere, encoded, he thought. His guide laughed: Not a chance.
“People are just on the network all the time now—there are no big variations anymore,” Moser says, drawing an arc with his arm over a nice curve representing the smooth daily swell of activity. The buzz picks up in the morning and peters off in the evening, reaching a zenith in the afternoon. “There’s got to be some kind of really dramatic event to change the volume,” he says. Otherwise, it’s business as usual.
After the tour I follow Moser to the top of the bunker. As we resurface my phone buzzes repeatedly—a barrage of inbound text messages, emails, and social media alerts. Inside of AT&T’s facility, my cellphone did not have service, ironically. (Granted, I am not an AT&T customer.) While I sort through these notifications, water drips from the sky, smearing the windows of the building in thick droplets. Outside the foyer a sculpture garden imitates rivers on the lawn with winding paths of rocks. Each represents one of the Earth’s oceans, and the boulders paired by their sides represent the continents, Moser says.
I let the significance of that phrase sink in while exiting the center. The words stand as a monument to the immeasurable impact that AT&T has had on the Information Era. A Bell Labs computer scientist had popularized the saying in a widely read programming textbook he authored in the 1970s, during Ma Bell’s heyday. The two words have served as the code writer’s equivalent of an audio technician’s “testing 1…2…3,” an indispensable tutorial for programmers, and a nerdy inside joke for software developers ever since. Now as more and more of the networking world comes to be defined by software, the expression takes on renewed emphasis.
The sheer volume of “hello worlds” swapped daily between members of the human race is a marvel. In a single day, AT&T’s center processes more than 100 million gigabytes of data on average. That’s 100 petabytes (if you prefer the technical term), an unfathomably large number. It is approximately equivalent to streaming 700 years of high definition video, by some estimates. That translates to watching a lot of DirecTV—the satellite broadcaster that AT&T bought for $48.5 billion last year.
Within the GNOC, an unassuming outpost in the heart of New Jersey, humanity’s hopes and dreams and fears are writ in wire, traded in bits, and studied in the abstract. Through cables and corridors and computers, peoples’ sighs and jests and jabs traverse miles instantly, eliminating the distance between them as though they were separated by the short hop of a synapse. Outrages, condolences, congratulations fly through the air, travel undersea, burrow underground, and arrive at their destinations straightaway. Ideas and prayers and threats and good wishes and emojis and selfie snaps and 911 emergency calls transit in a flash.
All these shuttle through the infrastructure maintained by companies like AT&T. The whole gamut of human emotion and connection appears in aggregate on this center’s displays—painted in variegated colors of various inscrutable shapes and sizes. The network itself resembles a living, breathing beast.
It is a brave and new world, to be sure.