Facebook’s (FB) week went from bad to worse Monday when the social networking giant’s apps went dark worldwide for roughly six hours. In a post written Monday afternoon, Facebook VP of infrastructure Santosh Janardhan blamed the outage on a configuration error on the “backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication.”
That error, Janardhan writes, had a cascading effect on Facebook’s entire network, both inside and outside of the company, knocking out its services around the world. Because of that, employees couldn't fix the problem remotely, forcing the company to dispatch engineers to one of its data centers to perform a manual reset, according to a memo obtained by The New York Times.
While the company doesn’t go into specifics about what exactly went wrong, journalist Brian Krebs, citing sources within Facebook, pointed to the company’s Border Gateway Protocol (BGP) as the culprit. BGP is essentially how internet traffic is routed across the web. It automatically finds the best paths for your requests for websites to get to a site’s servers and back. Without it, there’s no internet.
When Facebook’s BGP routes went down, there was no way for it to talk to the broader internet, dropping the company’s services from the web, Cloudflare’s engineering director Celso Martinho wrote in a blog post.
Cloudflare works with the internet’s domain name system or DNS, which is what turns website names like www.yahoofinance.com into the actual numerical address that computers read to whisk you away to the right site. Because of that, Cloudflare got a look at the BGP problems Facebook was dealing with in real time.
The outage, which began at roughly 12 p.m. ET, shut down the main Facebook app, Instagram, WhatsApp, Messenger, and the company’s Oculus virtual reality apps. The collapse was so widespread that Facebook employees couldn’t communicate with each other using the company’s internal chat app, while others were unable to open doors on the Facebook campus.
CEO Mark Zuckerberg posted a two sentence apology on Monday evening saying that the company’s apps were coming back online and that he knows “how much you rely on our services to stay connected with the people you care about.”
The company’s stock, meanwhile, took a nasty hit during the outage, dropping nearly 5% on an already rough day for the rest of the tech sector with competitors including Apple, Amazon, and Google shares falling between 1% and 2%.
The outage came a day after former Facebook employee and whistleblower Frances Haugen appeared on “60 Minutes” to discuss a trove of internal documents she leaked to the news program and to The Wall Street Journal.
Haugen is set to testify before the Senate Commerce Committee’s Subcommittee on Consumer Protection on Tuesday, where she’ll discuss the leaks. Last week, the subcommittee heard testimony from Antigone Davis, Facebook’s global head of safety, who was grilled over accusations that Facebook has taken a page from Big Tobacco in hiding research that its products are harmful.
More from Dan
Microsoft Windows 11 review: Big upgrades for workers and gamers alike
Apple is relying on the iPhone 13's new cameras to power record sales
Follow Yahoo Finance on Twitter, Facebook, Instagram, Flipboard, LinkedIn, YouTube, and reddit
Got a tip? Email Daniel Howley at [email protected] over via encrypted mail at [email protected], and follow him on Twitter at @DanielHowley.