• Home
  • Latest
  • Fortune 500
  • Finance
  • Tech
  • Leadership
  • Lifestyle
  • Rankings
  • Multimedia

Trendingnow

1

OpenAI says its AI models secretly broke out of a secure test environment and hacked into AI company Hugging Face in order to cheat on an evaluation

2

Mark Cuban says he has the solution to growing income inequality, and it's to reward every employee—from CEO to janitor—with company stock

3

Mathematicians grapple with a ‘very rapid and very unsettling change’ as AI cracks yet another century-old problem

1

OpenAI says its AI models secretly broke out of a secure test environment and hacked into AI company Hugging Face in order to cheat on an evaluation

2

Mark Cuban says he has the solution to growing income inequality, and it's to reward every employee—from CEO to janitor—with company stock

3

Mathematicians grapple with a ‘very rapid and very unsettling change’ as AI cracks yet another century-old problem
TechPointCloud

Amazon explains database glitch that impacted big customers

Barb Darrow
By
Barb Darrow
Barb Darrow
Down Arrow Button Icon
Barb Darrow
By
Barb Darrow
Barb Darrow
Down Arrow Button Icon
September 23, 2015, 8:24 AM ET
Photograph by David Ryder — Getty Images
Add Fortune on Google for similar content.

Amazon Web Services has posted an explanation of what happened at its big U.S. East data center that caused customers like Netflix to experience issues on Sunday.

According to the post-mortem of the “service event” (you have to love that term) a brief network disruption at 2:19 a.m. PDT affected a subset of the servers running Amazon’s (AMZN) DynamoDB database service which stores and maintains data tables for customers. Each table is divvied up into partitions, containing a portion of the table data and those partitions, in turn, are parceled out to many servers to provide fast access and to allow data replication.

Per the post, which apparently went up Tuesday night:

The specific assignment of a group of partitions to a given server is called a “membership.” The membership of a set of table/partitions within a server is managed by DynamoDB’s internal metadata service. The metadata service is internally replicated and runs across multiple data centers. Storage servers hold the actual table data within a partition and need to periodically confirm that they have the correct membership. They do this by checking in with the metadata service and asking for their current membership assignment. In response, the metadata service retrieves the list of partitions and all related information from its own store, bundles this up into a message, and transmits back to the requesting storage server.

Emphasis is mine. Read the post for the full blow-by-blow, but in essence, Amazon said the issue Sunday was that because so many customers are using a new DynamoDB feature called Global Secondary Indexes, the affected DynamoDB servers could not query the metadata service within the allotted time and took themselves offline.

As of 5:06 a.m. PDT Amazon thus decided to pause requests to the metadata service to relieve the load. By cutting down all those server retries, it was able to bring up additional capacity and restart the service.

Basically this all boils down to the fact that this Amazon service was operating at near full-capacity but AWS internal monitoring apparently did not pick that up in time to avert an outage, said David Mytton, CEO of Server Density, a London-based company which keeps an eye on web server performance across providers for customers.

“A normal network issue caused enough extra load to push the system over capacity which caused the issue,” he said via email.

Amazon will likely fix the monitoring situation and adjust its processes to improve analysis so this does not happen again, he noted. Amazon is nothing if not reactive.

Events like this one reignite the debate about whether businesses should entrust so much of their critical workload to shared public cloud infrastructure that they themselves do not own or control.

Some even brought up the notion that AWS, the self-proclaimed master of distributed and redundant resources, has become the de facto single point of failure IT people dread.

This morning’s #AWS outage reminds us that we all have a single point of failure now. In some ways, we used to be more resilient than that.

— Michael Jackson (@mjackson) September 20, 2015

Fortune reached out to Amazon for comment, but typically the cloud giant and big customers— including Netflix (NFLX)—note that if applications are designed correctly to take advantage of public cloud resources the benefits outweigh the risks.

The Amazon post ended in an apology and a promise to do better, explaining “… we will do everything we can to learn from the event and to avoid a recurrence in the future.”

Expect to hear more about this issue at AWS Re:invent in Las Vegas next month.

For more on Amazon Web Services, see the video.

Subscribe to Data Sheet, Fortune’s daily newsletter on the business of technology.

About the Author
Barb Darrow
By Barb Darrow
See full bioRight Arrow Button Icon
Add Fortune on Google for similar content.

Latest in Tech

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025

Most Popular

Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Finance
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam
By Fortune Editors
October 20, 2025
Fortune Secondary Logo
Rankings
  • 100 Best Companies
  • Fortune 500
  • Global 500
  • Fortune 500 Europe
  • Most Powerful Women
  • World's Most Admired Companies
  • See All Rankings
  • Lists Calendar
Sections
  • Finance
  • Fortune Crypto
  • Features
  • Leadership
  • Health
  • Commentary
  • Success
  • Retail
  • Mpw
  • Tech
  • Lifestyle
  • CEO Initiative
  • Asia
  • Politics
  • Conferences
  • Europe
  • Newsletters
  • Personal Finance
  • Environment
  • Magazine
  • Education
Customer Support
  • Frequently Asked Questions
  • Customer Service Portal
  • Privacy Policy
  • Terms Of Use
  • Single Issues For Purchase
  • International Print
Commercial Services
  • Advertising
  • Fortune Brand Studio
  • Fortune Analytics
  • Fortune Conferences
  • Business Development
  • Group Subscriptions
About Us
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • About Us
  • Press Center
  • Work At Fortune
  • Terms And Conditions
  • Site Map
  • Facebook icon
  • Twitter icon
  • LinkedIn icon
  • Instagram icon
  • TikTok icon
  • YouTube icon

Latest in Tech

‘It’s called StreetEasy, not StreetHard’: Mamdani cracks down on scourge of ‘housefishing’ and ‘real estate slop’
Real EstateHousing
‘It’s called StreetEasy, not StreetHard’: Mamdani cracks down on scourge of ‘housefishing’ and ‘real estate slop’
By Tatiana SatauaJuly 22, 2026
3 hours ago
As Washington panics about Chinese AI, Jensen Huang says open-source models like Kimi are ‘excellent’ and should be embraced, not banned
AIChina
As Washington panics about Chinese AI, Jensen Huang says open-source models like Kimi are ‘excellent’ and should be embraced, not banned
By Marco Quiroz-GutierrezJuly 22, 2026
3 hours ago
OpenAI CEO Sam Altman sits next to U.S. President Donald Trump.
AITech regulation
OpenAI’s rogue hacking incident was a warning shot. Will it be a wake-up call to finally create AI safety regulation?
By Jeremy Kahn and Emily ForliniJuly 22, 2026
3 hours ago
Man holding his hands open
InvestingStock
A new group of stocks is transforming global equities markets. Meet ‘Memi,’ the $3 trillion sector fueled by AI’s insatiable hunger for memory chips
By Amanda GerutJuly 22, 2026
4 hours ago
Sam Altman getting into a car
CybersecurityOpenAI
OpenAI’s models went rogue and hacked Hugging Face. It’s a wake-up call, experts say, but more concerning behavior may be next
By Beatrice NolanJuly 22, 2026
4 hours ago
Northrop Grumman launches private robot mission to strap jetpacks on aging satellites
InnovationSpaceX
Northrop Grumman launches private robot mission to strap jetpacks on aging satellites
By Marcia DunnJuly 22, 2026
7 hours ago

Most Popular

OpenAI says its AI models secretly broke out of a secure test environment and hacked into AI company Hugging Face in order to cheat on an evaluation
Cybersecurity
OpenAI says its AI models secretly broke out of a secure test environment and hacked into AI company Hugging Face in order to cheat on an evaluation
By Jeremy Kahn and Emily ForliniJuly 21, 2026
1 day ago
Mark Cuban says he has the solution to growing income inequality, and it's to reward every employee—from CEO to janitor—with company stock
Success
Mark Cuban says he has the solution to growing income inequality, and it's to reward every employee—from CEO to janitor—with company stock
By Sasha RogelbergJuly 20, 2026
2 days ago
Mathematicians grapple with a ‘very rapid and very unsettling change’ as AI cracks yet another century-old problem
AI
Mathematicians grapple with a ‘very rapid and very unsettling change’ as AI cracks yet another century-old problem
By Eva RoytburgJuly 21, 2026
1 day ago
The millennial generation has split, new Fed research shows: Those over 35 are edging toward boomer-style wealth, while everyone else falls behind
Real Estate
The millennial generation has split, new Fed research shows: Those over 35 are edging toward boomer-style wealth, while everyone else falls behind
By Nick LichtenbergJuly 22, 2026
7 hours ago
Despite a $156 million contract, Knicks star Jalen Brunson still calls his parents for financial advice any time he makes a big purchase
Success
Despite a $156 million contract, Knicks star Jalen Brunson still calls his parents for financial advice any time he makes a big purchase
By Emma BurleighJuly 21, 2026
1 day ago
‘I want to die broke’: Billionaire philanthropist Denny Sanford dies after giving away $4 billion
Success
‘I want to die broke’: Billionaire philanthropist Denny Sanford dies after giving away $4 billion
By Sydney LakeJuly 20, 2026
2 days ago

© 2026 Fortune Media IP Limited. All Rights Reserved. Use of this site constitutes acceptance of our Terms of Use and Privacy Policy | CA Notice at Collection and Privacy Notice | Do Not Sell/Share My Personal Information
FORTUNE is a trademark of Fortune Media IP Limited, registered in the U.S. and other countries. FORTUNE may receive compensation for some links to products and services on this website. Offers may be subject to change without notice.