Just about everybody who was online yesterday will have noticed something was wrong with ye olde internet. A major issue at network service provider Cloudflare brought down everything from and X and ChatGPT to order screens at McDonald’s restaurants. Now Cloudflare has posted a full explanation of the outage and it turns out the problem was entirely internal and self inflicted.
Cloudflare CEO Matthew Prince’s mea culpa was remarkably unambiguous. Up front and literally in bold, he kicks off with the following statement:
“The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind.”
He follows that up with an entirely caveat-free apology. “We are sorry for the impact to our customers and to the Internet in general. Given Cloudflare’s importance in the Internet ecosystem any outage of any of our systems is unacceptable. That there was a period of time where our network was not able to route traffic is deeply painful to every member of our team. We know we let you down today.”
So, what, exactly, happened? Cloudflare did initially suspect foul play, most likely a massive DDoS or distributed denial of service attack. However, further investigation revealed that, “it was triggered by a change to one of our database systems’ permissions which caused the database to output multiple entries into a ‘feature file’ used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.”
Unfortunately, Cloudflare’s bot management software had a hard coded file size limit, which that newly doubled feature file exceed. Boom, that software failed.
Image 1 of 1
Cloudflare’s 5xx error HTTP status codes began to spike around 11:20 UTC yesterday. (Image credit: Cloudflare)
The problem began at 11:20 UTC and Cloudflare says it had correctly identified the issue, stopped the propagation of the larger-than-expected feature file and got core traffic “largely flowing as normal” by 14:30. By 17:06, “all systems at Cloudflare were functioning as normal.”
Cloudflare says the event was the company’s worse outage since 2019. Several mitigations are being put in place to prevent a repeat, including more global kill switches for features and eliminating the ability for core dumps or other error reports to overwhelm system resources.
Wrapping up his blog post, Prince concludes, “an outage like today is unacceptable. We’ve architected our systems to be highly resilient to failure to ensure traffic will always continue to flow. When we’ve had outages in the past it’s always led to us building new, more resilient systems. On behalf of the entire team at Cloudflare, I would like to apologize for the pain we caused the Internet today.”
If you’re interested in the finer details, Prince’s blog post is absolutely forensic about every aspect of the problem and Cloudflare’s relevant systems and practices. It’s an intriguing insight into how a very important part of the internet works and the impact of having large parts of the modern information world reliant on a single company.
&
Best graphics card 2025
All our current recommendations
Pick a category Best overallCore ComponentsChairs & DesksSystems & LaptopsScreensPeripheralsCases, Cooling & Storage Choose a list Best CPU for gaming 2025Best graphics card 2025Best gaming motherboard 2025Best Mini-ITX motherboardBest RAM for gaming 2025 
BEST OVERALL 1. AMD Radeon RX 9070 Check Price 
BEST VALUE 2. AMD Radeon RX 9060 XT 16 GB Check Price 
BEST BUDGET 3. Intel Arc B570 Check Price 
BEST MID-RANGE 4. Nvidia GeForce RTX 5070 Ti Check Price 
BEST HIGH-END 5. Nvidia GeForce RTX 5090 Check Price Read the full guide: Best graphics card 2025POWERED BY
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
From PCGamer latest via this RSS feed