Day 22. Naturally it would have to happen that Gateway would crash while I’m out of town. I’ve asked my friend Tory to reboot it, and hopefully he will.<\/p>\n
Yesterday was a awful day on all fronts. Woke up at 10:00a, went down to the clients site, wrote a disaster recovery plan, and pulled a first-year-sysadmin stupidity stunt.<\/p>\n
I was using text mode on a linux console and switching back and forth between about four consoles. I forgot that one of the consoles had been sshed to another host, and when a process hung because of some incorrect settings, I did a killall – on the wrong machine. Unfortunately, the process was a dependancy for the production services run on that machine, and so they went down, leaving 300 people without their database. All of whom were in the same building I was.<\/p>\n
We restarted the database (took about 10 minutes) and ostensibly no data was lost. But I still feel awful. THe boss didn’t even yell at me – just said ‘you know what you did..’<\/p>\n
Then more awful things happened..<\/p>\n","protected":false},"excerpt":{"rendered":"
Day 22. Naturally it would have to happen that Gateway would crash while I’m out of town. I’ve asked my friend Tory to reboot it, and hopefully he will. Yesterday was a awful day on all fronts. Woke up at 10:00a, went down to the clients site, wrote a disaster recovery plan, and pulled a […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/www.sheer.us\/weblogs\/wp-json\/wp\/v2\/posts\/1702"}],"collection":[{"href":"https:\/\/www.sheer.us\/weblogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sheer.us\/weblogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sheer.us\/weblogs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sheer.us\/weblogs\/wp-json\/wp\/v2\/comments?post=1702"}],"version-history":[{"count":0,"href":"https:\/\/www.sheer.us\/weblogs\/wp-json\/wp\/v2\/posts\/1702\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.sheer.us\/weblogs\/wp-json\/wp\/v2\/media?parent=1702"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sheer.us\/weblogs\/wp-json\/wp\/v2\/categories?post=1702"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sheer.us\/weblogs\/wp-json\/wp\/v2\/tags?post=1702"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}