#AceNewsReport – Oct.05: While we can speculate all day as to exactly what issue occurred on the 4th of October in Facebook’s infrastructure, it is however fun to arm-chair investigate using open source information, and send #hugops to the Facebook’s network operations team!
#AceDailyNews says according to media news Facebook, Whatsapp and Instagram back after outage that lasted 6-hours with it being blamed on technical issue but Brian Krebs security analyst says it was their DNS records show no data present see below ……
Cybersecurity Expert Brian Krebs Explains What Is Known About Catastrophic Facebook Outage according to Breitbart By Alana Mastrangelo 04.Oct: 2021
Cybersecurity expert Brian Krebs says Facebook, as well as its Instagram and WhatsApp platforms, are all suffering from ongoing global outages due to someone from inside Facebook updating the company’s Border Gateway Protocol (BGP) records, which took away the map telling the world’s computers how to find its online properties. According to a New York Times reporter, employees cannot even open doors with their security cards due to the catastrophic outage.
While it remains unclear why this happened, Krebs says how it happened is clear: On Monday morning, something inside Facebook caused the company to rescind key digital records that tell computers and other devices how to find the destinations online.
A Tale of DNS & BGP: The Facebook Outage, October 2021
This is a short tale. One of despair and intrigue, as we realise the fragility of the global internet that we all so love and adore. On Monday 4th October 2021, people stopped scrolling through Facebook, they gave up posting selfies to Instagram, they ceased texting on WhatsApp, and Facebook employees abandoned doing any work. For all Facebook-owned websites were down, thanks to a couple of three letter acronyms: DNS and BGP.
While Facebook have not released any key details nor a postmortem, the internet is built on a series of open source standards and protocols that allow us to remotely inspect some of the fallout.
When you type
riskledger.com or any other name into your web browser of choice, an invisible background process begins. This starts with the Domain Name System, or DNS for short. It is often said to be the phone book of the internet. It is responsible for converting long series of arcane numbers, useful only to machines, to something memorable.
The DNS system is contacted by your web browser asking simply, “what is the internet address for facebook.com?”. Now if one device was answering every DNS question, it would quickly become overwhelmed, so over the years a distributed hierarchy formed. This starts with a DNS implementation built right into your device’s operating system, to devices called “recursers” operated by your ISP and sometimes large organizations such as Google or CloudFlare, and even scaling up to central “root” devices that know the whereabouts of every .com, .co.uk and more.
Facebook in this case, operates a set of intermediary DNS servers that are responsible for everything between your ISP’s recursers and the roots. These are responsible for facebook.com, instagram.com, whatsapp.com and everything else they operate. These servers are not responding. This is what we see above from our web browser when it hints to us the error is
DNS_PROBE_FINISHED_NXDOMAIN. Our web browser tried it’s best to work out the internet address for Facebook, but it didn’t get a reply.
While DNS may allow our computers to rapidly translate from names useful to humans to numbers useful for computers, how does your device, with it’s own internet address, traverse the global internet to reach Facebook?
No two devices on the internet are directly connected. At home, you will have your residential ISP. In a datacentre, there will be multiple commercial ISPs. Given two internet addresses, how do the two communicate? This is where routing comes in.
Routing is the system by which a path between two devices is calculated and established, potentially traversing dozens of ISP networks in the process. Given there must be tens-of-thousands of ISPs, how do we ensure they all can speak to each other?
We establish a standard! In 1989 no less, the Border Gateway Protocol, or BGP for short, was accepted by the internet community as RFC 1105. The document laid out a protocol by which routers operated by different companies and ISPs could exchange routing information which each other.
So what does this have to do with Facebook? We discussed earlier how our web browsers were not receiving a response to their DNS questions, this was however not a result of DNS itself, it was a result of Facebook’s routers ceasing to speak BGP with the rest of the internet, and ultimately all of our ISP’s routers stopped knowing where to send our DNS requests, or any traffic to Facebook for that matter.
While we can speculate all day as to exactly what issue occurred on the 4th of October in Facebook’s infrastructure, it is however fun to arm-chair investigate using open source information, and send #hugops to the Facebook’s network operations team!
It blamed an internal technical issue, which not only affected Facebook’s services, but reportedly also employees’ work passes and email.
The services were down from about 16:00 GMT until around 22:00 on Monday.
But the company said there was “no evidence that user data was compromised”.
Sheera Frenkel, the New York Times’ technology reporter, told the Today programme part of the reason it took so long to fix was because “the people trying to figure out what this problem was couldn’t even physically get into the building” to work out what had gone wrong.
In a statement, Facebook said that the faulty configuration change affected the company’s internal tools and systems which complicated attempts to resolve the problem.
Downdetector, which tracks outages, said some 10.6 million problem reports around the world. However, the real number of people affected is much higher: more than 3.5 billion people use Facebook, Messenger, Instagram and Whatsapp.
According to the business website Fortune, it also cost Facebook founder Mark Zuckerberg an estimated $6bn (£4.4bn) at one point as shares plummeted.
Mr Zuckerberg has apologised to those affected by the outage.
Some people also reported problems using Facebook’s virtual reality headset platform, Oculus, and apps which require Facebook logins were affected, including Pokémon Go.
An outage of this scale for such a long time is rare. A disruption in 2019 left Facebook and its other apps mostly inaccessible across the world for more than 14 hours.
Several other tech companies, including Reddit and Twitter, poked fun at the social media giant’s predicament – prompting responses from the affected apps.
Hi and happy Monday 😵💫— Instagram (@instagram) October 4, 2021
The BBC is not responsible for the content of external sites.View original tweet on Twitter
The disruption comes the day after an interview with a former Facebook employee who leaked documents about the company.
Frances Haugen told CBS news on Sunday that the company had prioritised “growth over safety”.
On Tuesday she will testify before a Senate subcommittee in a hearing titled “Protecting Kids Online”, about the company’s research into Instagram’s effect on the mental health of young users.
Many outages get resolved fairly quickly. They are often localised too, with some people unable to open a website that can be viewed in another country.
This outage, however, was global, and affected all of Facebook’s many spin-offs.
The length of time it was off grid is also unusual. There were reports of “mayhem” in Facebook headquarters, as technicians scrambled to fix the problem.
Interesting too that the outage hampered Facebook’s ability to tackle the crash – bringing down internal tools needed to remedy the problem.
It should also be said that Facebook’s statement is carefully written. It doesn’t rule out foul play.
The week had already started off badly – after the whistleblower in the “Facebook Files” revelations revealed herself on Sunday.
But a bad week has become a terrible one for the social network.
BBC Security News/ Breitbart/
#AceNewsDesk report …………Published: Oct.05: 2021:
Editor says …Sterling Publishing & Media Service Agency is not responsible for the content of external site or from any reports, posts or links, and can also be found here on Telegram: https://t.me/acenewsdaily all of our posts fromTwitter can be found here: https://acetwitternews.wordpress.com/ and all wordpress and live posts and links here: https://acenewsroom.wordpress.com/and thanks for following as always appreciate every like, reblog or retweet and free help and guidance tips on your PC software or need help & guidance from our experts AcePCHelp.WordPress.Com