• Home
  • Latest
  • Fortune 500
  • Finance
  • Tech
  • Leadership
  • Lifestyle
  • Rankings
  • Multimedia

Trendingnow

1

Analysts expected oil to surge above $200 but China has quietly kept prices half of that—and can’t for much longer

2

Corporate America has been draining the world's water. Matt Damon's new campaign calls on Gap, Starbucks, and Amazon to help give it back

3

Marc Lore’s robots make 500 burrito bowls an hour. A human can make 45

1

Analysts expected oil to surge above $200 but China has quietly kept prices half of that—and can’t for much longer

2

Corporate America has been draining the world's water. Matt Damon's new campaign calls on Gap, Starbucks, and Amazon to help give it back

3

Marc Lore’s robots make 500 burrito bowls an hour. A human can make 45
TechPointCloud

Amazon explains database glitch that impacted big customers

Barb Darrow
By
Barb Darrow
Barb Darrow
Down Arrow Button Icon
Barb Darrow
By
Barb Darrow
Barb Darrow
Down Arrow Button Icon
September 23, 2015, 8:24 AM ET
Photograph by David Ryder — Getty Images

Amazon Web Services has posted an explanation of what happened at its big U.S. East data center that caused customers like Netflix to experience issues on Sunday.

According to the post-mortem of the “service event” (you have to love that term) a brief network disruption at 2:19 a.m. PDT affected a subset of the servers running Amazon’s (AMZN) DynamoDB database service which stores and maintains data tables for customers. Each table is divvied up into partitions, containing a portion of the table data and those partitions, in turn, are parceled out to many servers to provide fast access and to allow data replication.

Per the post, which apparently went up Tuesday night:

The specific assignment of a group of partitions to a given server is called a “membership.” The membership of a set of table/partitions within a server is managed by DynamoDB’s internal metadata service. The metadata service is internally replicated and runs across multiple data centers. Storage servers hold the actual table data within a partition and need to periodically confirm that they have the correct membership. They do this by checking in with the metadata service and asking for their current membership assignment. In response, the metadata service retrieves the list of partitions and all related information from its own store, bundles this up into a message, and transmits back to the requesting storage server.

Emphasis is mine. Read the post for the full blow-by-blow, but in essence, Amazon said the issue Sunday was that because so many customers are using a new DynamoDB feature called Global Secondary Indexes, the affected DynamoDB servers could not query the metadata service within the allotted time and took themselves offline.

As of 5:06 a.m. PDT Amazon thus decided to pause requests to the metadata service to relieve the load. By cutting down all those server retries, it was able to bring up additional capacity and restart the service.

Basically this all boils down to the fact that this Amazon service was operating at near full-capacity but AWS internal monitoring apparently did not pick that up in time to avert an outage, said David Mytton, CEO of Server Density, a London-based company which keeps an eye on web server performance across providers for customers.

“A normal network issue caused enough extra load to push the system over capacity which caused the issue,” he said via email.

Amazon will likely fix the monitoring situation and adjust its processes to improve analysis so this does not happen again, he noted. Amazon is nothing if not reactive.

Events like this one reignite the debate about whether businesses should entrust so much of their critical workload to shared public cloud infrastructure that they themselves do not own or control.

Some even brought up the notion that AWS, the self-proclaimed master of distributed and redundant resources, has become the de facto single point of failure IT people dread.

This morning’s #AWS outage reminds us that we all have a single point of failure now. In some ways, we used to be more resilient than that.

— Michael Jackson (@mjackson) September 20, 2015

Fortune reached out to Amazon for comment, but typically the cloud giant and big customers— including Netflix (NFLX)—note that if applications are designed correctly to take advantage of public cloud resources the benefits outweigh the risks.

The Amazon post ended in an apology and a promise to do better, explaining “… we will do everything we can to learn from the event and to avoid a recurrence in the future.”

Expect to hear more about this issue at AWS Re:invent in Las Vegas next month.

For more on Amazon Web Services, see the video.

Subscribe to Data Sheet, Fortune’s daily newsletter on the business of technology.

About the Author
Barb Darrow
By Barb Darrow
See full bioRight Arrow Button Icon

Latest in Tech

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025

Most Popular

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Fortune Secondary Logo
Rankings
  • 100 Best Companies
  • Fortune 500
  • Global 500
  • Fortune 500 Europe
  • Most Powerful Women
  • World's Most Admired Companies
  • See All Rankings
  • Lists Calendar
Sections
  • Finance
  • Fortune Crypto
  • Features
  • Leadership
  • Health
  • Commentary
  • Success
  • Retail
  • Mpw
  • Tech
  • Lifestyle
  • CEO Initiative
  • Asia
  • Politics
  • Conferences
  • Europe
  • Newsletters
  • Personal Finance
  • Environment
  • Magazine
  • Education
Customer Support
  • Frequently Asked Questions
  • Customer Service Portal
  • Privacy Policy
  • Terms Of Use
  • Single Issues For Purchase
  • International Print
Commercial Services
  • Advertising
  • Fortune Brand Studio
  • Fortune Analytics
  • Fortune Conferences
  • Business Development
  • Group Subscriptions
About Us
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • Facebook icon
  • Twitter icon
  • LinkedIn icon
  • Instagram icon
  • Pinterest icon

Latest in Tech

Why is it so hard to get ROI from AI? Because building from first principles isn’t easy
NewslettersEye on AI
Why is it so hard to get ROI from AI? Because building from first principles isn’t easy
By Jeremy KahnJune 11, 2026
4 hours ago
SpaceX lowballed its bankers on fees. Goldman Sachs has another way to win big
Startups & VentureFinance
SpaceX lowballed its bankers on fees. Goldman Sachs has another way to win big
By Shawn TullyJune 11, 2026
5 hours ago
Three ways that Asia’s enterprises are adopting AI—and where they are falling behind
CommentaryOracle
Three ways that Asia’s enterprises are adopting AI—and where they are falling behind
By Garrett IlgJune 11, 2026
6 hours ago
Dr. Shiv Rao speaks
Startups & VentureHealth
Abridge wants to be the operating system for medicine—and NVIDIA and Eli Lilly are helping build it
By Lily Mae LazarusJune 11, 2026
9 hours ago
Silicon Valley insiders warn U.S. defense supply chain is unprepared for modern warfare
AIBrainstorm Tech
Silicon Valley insiders warn U.S. defense supply chain is unprepared for modern warfare
By Sebastian HerreraJune 11, 2026
9 hours ago
Exclusive: Consumer device giant LG Electronics to launch blockchain to place and sell ads
CryptoBlockchain
Exclusive: Consumer device giant LG Electronics to launch blockchain to place and sell ads
By Jack Kubinec and Ben WeissJune 11, 2026
9 hours ago

Most Popular

Analysts expected oil to surge above $200 but China has quietly kept prices half of that—and can’t for much longer
Energy
Analysts expected oil to surge above $200 but China has quietly kept prices half of that—and can’t for much longer
By Sasha RogelbergJune 10, 2026
1 day ago
Corporate America has been draining the world's water. Matt Damon's new campaign calls on Gap, Starbucks, and Amazon to help give it back
Environment
Corporate America has been draining the world's water. Matt Damon's new campaign calls on Gap, Starbucks, and Amazon to help give it back
By Catherina GioinoJune 9, 2026
3 days ago
Marc Lore’s robots make 500 burrito bowls an hour. A human can make 45
Innovation
Marc Lore’s robots make 500 burrito bowls an hour. A human can make 45
By Amanda GerutJune 9, 2026
2 days ago
Current price of oil as of June 11, 2026
Personal Finance
Current price of oil as of June 11, 2026
By Joseph HostetlerJune 11, 2026
14 hours ago
Pentagon accuses Alibaba, Baidu and BYD, three of China's biggest companies, of supporting the Chinese military
Asia
Pentagon accuses Alibaba, Baidu and BYD, three of China's biggest companies, of supporting the Chinese military
By Kate O'Keeffe and BloombergJune 8, 2026
3 days ago
Costco CEO Ron Vachris rose from forklift driver to the C-suite without a college degree: ‘Don’t chase a title’ is the career advice that got him there
Success
Costco CEO Ron Vachris rose from forklift driver to the C-suite without a college degree: ‘Don’t chase a title’ is the career advice that got him there
By Preston ForeJune 8, 2026
4 days ago

© 2026 Fortune Media IP Limited. All Rights Reserved. Use of this site constitutes acceptance of our Terms of Use and Privacy Policy | CA Notice at Collection and Privacy Notice | Do Not Sell/Share My Personal Information
FORTUNE is a trademark of Fortune Media IP Limited, registered in the U.S. and other countries. FORTUNE may receive compensation for some links to products and services on this website. Offers may be subject to change without notice.