Personal View site logo
Fun: Facebook outage happened due to stupidity
  • Independent experts and the Facebook team both stated that the reason for this massive failure was a change in the configuration of the central routers. They are responsible for coordinating traffic between the data centers of the social network. Inexperienced Employee Company representatives performed an unsuccessful reconfiguration of their routers, resulting in traffic disruption and other problems. The equipment in question is responsible for BGP sessions and their announcements, plus the company's internal network. After that, the FB NS servers were no longer available, and the DNS records disappeared as well. Naturally, under such conditions, there can be no question of any normal operation of the social network and its services.

    The failure was cumulative, which can be compared to the effect of falling dominoes. One problem caused another, one caused the next, and so on. There are many rumors about the culprit or the culprits of the global network outage, but the company has not made an official statement on this matter.

    The failure was not promptly eliminated precisely because of the fall of not only the external, but also the internal network of the company. So the team of emergency responders simply did not manage to quickly get to the data centers. All the passes did not work for the employees.

    Everything had been planned by "geniuses", so if one router is misconfigured everything falls, including doors.

  • 1 Reply sorted by
  • OVH - one of the biggest hosting providers (that PV uses) fully repeated Facebook thing.

    Also router reconfiguration and around 40 minute total blackout of all datacenters.

    Details
    Start time: 13/10/2021 07:20 UTC
    Impact : Since 07:20 UTC this morning the entire OVH network is unavailable. We are experiencing a network incident located in the United States. All the technical teams are working to resolve the incident
    Comment : Since 08:22 UTC all services are gradually returning following the isolation of network equipment in the US.