An explanation for Mortal Kombat Online’s downtime…

D'Arque Bishop

Well, Mortal Kombat Online is facing an extended downtime. Apparently we are suffering from such a catastrophic hardware failure, so to speak, that it’s going to take a few days and a LOT of work to get everything back up and running the way it was. I’ve left the standard disclaimer on the site itself, but I figured I would use this post on my own site to give an in-depth explanation as to what happened and allow others to give feedback if necessary.


In any event, here’s what’s happening.

Back in Christmas of 2005, the hard drive on the main MK Online server suffered a crash. We were forced to get our hosting provider to replace the drive; when we had them replace it, we demanded that they put in mirrored drives like we had when we first got our dedicated server with them. (At the time, we were actually on a new dedicated server.) They did so, but instead of using two identical drives they used a single SCSI (heavy-duty) drive and a much larger IDE (consumer-level) drive. We had meant to get with our hosting provider to fix that, as such a configuration is much less than ideal. However, we never got around to it as bigger issues kept coming up.

Well, recently we’d been noticing system crashes, caused by memory errors. We soon realized that it was because of virtual memory (which is on the IDE drive), and it appears that the IDE drive is close to failure. The problem has gotten much worse over the past week, to the point where the filesystems are being corrupted. The last straw came an hour ago, when the forum database table that held the actual posts was lost. Obviously, we can’t continue like this.

So what are we doing to correct this? I’m working on restoring a backup copy of the database from this morning. With luck, it will be in a decent state but there is the very real possibility of data loss. In the meantime, our hosting provider is building a newer, faster server for us. Once it’s ready, we’ll rush to copy the data from the old server onto the new one and try and get us back online as quickly as possible. Still, we expect to be offline until the weekend at least.

We apologize for the delays and inconvenience this has caused. Trust me, I’m as frustrated as anyone else about this, if not more so. (After all, I’m the one trying to pick up the pieces here….) Hopefully once we have everything corrected it’ll be smooth sailing. :-)

EDIT – 7:00 PM CDT – It looks like the latest backup of the database is intact. Everything up to this morning at around 9 AM EDT appears to have been saved. What a relief… heh. Now we just need to wait for the new server to come online, and we’ll work from there. I’ll provide more information as it becomes available.

EDIT – 7:50 PM CDT – By the way, the Java chat is still available for those of you who come onto #mortalkombat (aka, the #MortalKombat Online IRC Network). You can reach it by clicking here.

EDIT – 7:20 AM CST 10 May – Thanks for the input, everyone. I just wanted to point out a few things based on comments I am seeing…

  • For those asking, when the site comes back up it will come back up with no changes in layout, software, etc. CCShadow hasn’t had time to work on version 8 of CDS (the backend of the site), and to expect us to change software packages at a drop of the hat is unrealistic at best.
  • Jonin01 is right in that this will take time. I’ve done everything I can do on this end; now it’s just up to our hosting provider to get a machine ready for us to migrate to. Once I hear the new machine is up and running, I can copy all of our old data to it and bring it online. Until then, it’s simply a matter of waiting.
  • Finally, for those offering assistance, it is much appreciated. Truth be told, we have everything under control; I suppose it helps a great deal when I do this stuff (system administration and support) as my paying job as well as for MK Online. It’s really not as catastrophic as it sounds; even if we were to lose the main server filesystem today I have a site backup elsewhere that can be used to get MK Online back up and running. We’re in good shape, all things considered; it’s more annoyances than anything else right now.

As before, once I get more information I’ll be sure to post it here. Thanks for your support and patience while we work to get these problems taken care of. :-)

EDIT – 2:20 PM CDT 11 May – The new server is ready for us to start working on it. However, being out of town and without a direct path to the internet, I can’t access the server properly to begin working on it. I should be home by 10 PM CDT, at which point I’ll begin working on it. If all goes well, we should be back up and running tomorrow evening. :-)

EDIT – 10:50 PM CDT 11 May – There’s a problem with the operating system’s configuration that’s preventing us from using it. Rather than going through hours of trying to fix it manually, I’ve had CCShadow contact our hosting provider with precise instructions for an OS reinstall. Hopefully sometime tomorrow it’ll be ready for the actual site reimplementation.

EDIT – 9:40 PM CDT 12 May – No, our hosting provider hasn’t rebuilt the server yet. I’m going to get with CCShadow as soon as he gets online about contacting them directly and getting them to hurry it up. I know everyone’s getting annoyed; I’m probably more annoyed than anyone at this point. Thanks for your patience.

EDIT – 3:35 PM CDT 13 May – Our hosting provider went ahead and made the needed changes without requiring a reinstallation of the operating system. I’m in the process of configuring the server and restoring the data. This will take a while, so please be patient.

EDIT – 7:50 PM CDT 13 May – All of the data is restored. We’re working now to get the new server onto the old server’s IP address and do some final testing. Once that’s done we’ll reopen the site.

EDIT – 8:50 PM CDT 13 May – It’s going to be a while, as their automated phone system is backed up and I can’t get a call through. CCShadow will call them when he gets home and get it online. I don’t know if we’ll be open tonight, but we’ll do our very best. Otherwise, we’re hoping tomorrow at the very latest.

EDIT – 8:25 AM CDT 14 May – The good news is that the new server is on the necessary IP address. The bad news is that I’m encountering some serious issues getting the site to work properly on the new server. I’ll keep working on it throughout the day to see if I can get the problems fixed.

EDIT – 11:30 AM CDT 14 May – CCShadow and I have taken a preliminary look at the server. Right now we think there’s some incompatibility or software conflict that’s causing CDS not to work properly. I’ll continue looking at it when I have time (which isn’t often as there are upgrade issues here at work that need resolving; as that’s my paying job it has to take priority), and CCShadow will do heavy-duty debugging when he gets home tonight.

EDIT – 3:10 PM CDT 14 May – If the testing we’ve been doing is anything to go by, we’ve knocked out all but one minor annoying bug. Right now we’re evaluating whether it’s safe to reopen the site now or not. If CCShadow gives the go-ahead, we’ll go ahead and reopen.

EDIT – 3:55 PM CDT 14 May – Never mind. It turned out that one bug is affecting more site functions than we thought. Reopening will have to wait until CCShadow’s had a chance to stamp out the bug. Sorry, folks… this is frustrating us as much as it is you. Hopefully we can be back up and running tonight…

EDIT – 10:35 PM CDT 14 May – We’re back up!

286 thoughts on “An explanation for Mortal Kombat Online’s downtime…”

  1. man, i wish it would’ve been done by lat night; hopefully its done before Monday.

  2. I wouldn’t use CentOS5 for a while. Not at least till 5.1 — Centos 4.5 should be out soon since we just released RHEL4.5

  4. Oh this is the worst news iI’ve ever heard
    I wish good luck for MK group in order to fix the site
    Please when fixing the site, inform me at my E-mail:akwa_libnani_junior@hotmail.com
    Sincerely Yours
    From Lebanon Mohammad Issa Jamal A.K.A Madz

  8. owww…i miss MKO!
    ok i have a life,but MKO is the most useful ssite on MK universe…
    I’ll wait until everything will come back to normal without disturbing.

  9. I miss it so much!!!, i can’t wait; i went to starbucks, got a coffee, and went home! I miss it so much… :(

  10. Oh man i just made myself an avatar as well and wanted to show everyone ( i just got photoshop)

  12. IM LOOSING MY FREAKING MIND!!! I CANT WAIT TILL ITS BACK ON!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  13. Aww, dammit! I was going to the website to find the Finish Him’s/Finish Her’s for MK3. Sucks to know the site is down. Hope y’all get back up real soon.

  14. i do have a life, but i definitely want to get on sometime either today/tommorow.

  15. But you can get on the IRC in the mean time, so it really isnt all that bad. Talking to other fans in real-time is just as fun.

  20. YOU SEE WHAT HAPPEND WHEN YOU BANNED ME MKO LOL YOU GET VIRUS ON YOUR SYSTEM LOL DAMN IM A GOOD
    H A C K E R!!!!!!!!! FUCKK MKO MODS

  21. YOU SEE WHAT HAPPEND WHEN YOU BANNED ME MKO LOL YOU GET VIRUS ON YOUR SYSTEM LOL DAMN IM A GOOD
    H A C K E R!!!!!!!!! FUCKK MKO MODS

    The amusing thing is, this idiot didn’t even read up to understand that the problem was hardware and not software in nature. Ah, well… obviously he was banned from the site (and from here now) for good cause (namely, being a useless troll). :-)

  22. I wish to be impervious….but this whole problem has caused me to lose the benifit of the doubt…..

  26. Shouldn’t be much longer now, i agree with you SF-64, man i’m so bored! I’m about to break down if the site doesn’t get fixed by at least 12:00 tonight… :(

  30. Why don’t you guys go listen to music or something, I’m sick of all these complaining. Yeah it sucks MKO is still down, but it’s not gonna help if you just sit here and complain.

  31. i do listen to music and etc. superawesomeness, its just that MKO is one of my favorite things to do when i get home.

  33. OMFG OMFG WOOT HOW LONG WILL IT TAKE FOR THE SITE TO GET BACK UP NOW, MAYBE 1 HOUR, IDK MAYBE 20 MINS HOLY CRAP THIS IS SO COOL! Good working D’Arque.

  36. God, i’m so happy ive been, so bored; I’m glad that it’s getting back up and running. Sorry about that post, i was so happy… I love this website. Well i’m glad it’s gonna get back up running, Good work D’Arque, and all the other people that helped. I wish you goodluck for the rest of it. :) Sincerly – Redman

  38. oh well, i’m going to watch tv and go to sleep anyway, hopefully i’m able to get on tommorow. Goodnight everyone.

  40. damn it, now I gotta use the crappy MKAC site to continue being an MKA nerd. BTW MKO>MKAC by a mile.

  43. Freak! Damn it! Freak! Damn it!
    Our MKO is still down. My fingers have been crossed for so long now I think they’ve merged together to become one giant finger. I’ll check back in 6hours!
    Big shout out to to all MK freaks across the globe! See you all soon on MKO and May the force be with you all!

  44. dang, still ain’t on yet! Hopefully we’ll be able to get on after i get back from college, or this will get annoying.

  47. Wow this is the longest it has been down in a long time.

    oh and to what KINGMOB said,Bishop dont worry,This guy is a complete moron and he thinks he is the greatest person in the world. The guy cant spell and he is the worst player on MKA. He is Rank 1 to,Sad.

    Sorry about all of that. I just really cant stand that guy and im glad he is banned from here as well as MKO. lol

    Well hopefully things are getting back to normal soon and i hope to see the site up and running sometime soon. If you have to,Take as much time as you need to make sure that it works properly once again. ; )

  49. I cried myself to sleep last night. Oh well, good things are worth waiting for.

    P.S. 28 Weeks Later was cool. Pretty much non-stop bloody violence. :)

  50. Well guys, from what it sounds like, today’s the day. I hope this works out!
    BTW, isn’t it great that network sites such as the Kombat Pavilion and MK Warehouse are still up and running?

Comments are closed.