An Incident Summary of the most catastrophic BCMC Website Nuclear disaster in BCMC History by the culprit, the BCMC Webmaster.

Categorization of disaster on International Nuclear Event Scale: 7 - Major Accident

Time Line of Incident:

January 9th, 2016

11:09 PM BCMC Webmaster (me) logs into BCMC Server to perform seemingly routine task requested of him by web development team
11:12 PM BCMC Webmaster creates subdomain and redirect to existing .php page on website, or at least so he thought.
11:13 PM BCMC Webmaster immediately notices main BCMC Website is now offline.
11:14 PM BCMC Webmaster deletes subdomain with redirect assuming this will return website to original state
11:15 PM BCMC Webmaster notices that the main BCMC Website is still offline despite the elimination of the subdomain
11:17 PM BCMC Webmaster takes a peek into the reactor core (file structure) to see if anything has changed.
11:18 PM BCMC Webmaster takes note that the reactor core has been damaged and countless files and folders are missing.
11:20 PM BCMC Webmaster sounds the alarm and summons all members of the Web Development Team to assist.
11:30 PM Other members of the Web development team confirm that the BCMC Website reactor core has been compromised.
11:45 PM (or thereabouts), the Web Development team decides that a full restoration of the previous day's backup file is in order.
11:46 PM BCMC Webmaster initiates backup restoration of BCMC Website through server's restoration backup application.

Crisis and Meltdown

January 10th, 2016

12:03 AM backup restoration fails, unexpected server full error indicated
12:03 AM Server interface is destroyed/non-functional, access to server is lost, reactor core is in full meltdown. The crisis is now underway.
12:13 AM BCMC Webmaster contacts India based emergency responders to aid in the crisis and pleads for assistance.  
1:00 AM BCMC Webmaster is still unable to gain access to the core, now in meltdown.
3:00 AM BCMC Webmaster and web team packs it in for the night, unable to make any progress.
10:00 AM BCMC Webmaster and members of Web Team with minor assistance from the server company attempts to gain access to the now red-hot core via direct SSH interface to bypass the now destroyed server interface.  These early attempts were unsuccessful.
11:00 AM BCMC Webmaster and Web Team discover that the root folder has been maxed out at 100% with temporary files and is likely on a feedback loop process.

5:57 PM Emergency Response Team (ERT) from India Arrives on scene and is horrified.
6:10 PM BCMC Webmaster provides access and passwords to ERT
6:55 PM The full extent of the meltdown and the horror becomes apparent.  All backup files appear to now have been lost.
7:30 PM BCMC Social event begins; its participants completely unaware of the magnitude of the ongoing crisis so close by.  Most of the Web Development team is nowhere to be seen at the Club Social.
7:40 PM BCMC Webmaster and ERT are uncertain as to best course of action at this point - full rebuild, salvage bits and pieces, update to a newer core, etc.
7:50 PM Paul Olynek arrives on scene after a long day of work and takes charge of the situation
9:00 PM BCMC Webmaster, now exhausted packs it in for the night, leaving Paul and ERT to pull an all-nighter to clean up the webmaster's (my) mess.

January 11th, 2016

8:06 AM After an all-nighter herculean effort from Paul Olynek and ERT, Paul informs BCMC Webmaster that he is about to power up the reactor again (website).
8:10 AM - 9:00 BCMC Website core gradually chokes and sputters back to life.
10:00 Miraculously, Paul Olynek and ERT were able to restore the website with almost no loss of data.

1:00 PM Web development team is not sure whether to laugh or to cry.
2:00 PM Work continues in the background by the Web Development Team in the aftermath of the crisis, but the website is back in business.

Going Forward:

Once the Web Development team has recovered from the trauma and radiation exposure, it will meet to debrief and put procedures into place to prevent future reactor meltdowns from ever occurring.   

Chris Ludwig

BCMC Webmaster