An explanation for Mortal Kombat Online’s downtime…

D'Arque Bishop

Well, Mortal Kombat Online is facing an extended downtime. Apparently we are suffering from such a catastrophic hardware failure, so to speak, that it’s going to take a few days and a LOT of work to get everything back up and running the way it was. I’ve left the standard disclaimer on the site itself, but I figured I would use this post on my own site to give an in-depth explanation as to what happened and allow others to give feedback if necessary.


In any event, here’s what’s happening.

Back in Christmas of 2005, the hard drive on the main MK Online server suffered a crash. We were forced to get our hosting provider to replace the drive; when we had them replace it, we demanded that they put in mirrored drives like we had when we first got our dedicated server with them. (At the time, we were actually on a new dedicated server.) They did so, but instead of using two identical drives they used a single SCSI (heavy-duty) drive and a much larger IDE (consumer-level) drive. We had meant to get with our hosting provider to fix that, as such a configuration is much less than ideal. However, we never got around to it as bigger issues kept coming up.

Well, recently we’d been noticing system crashes, caused by memory errors. We soon realized that it was because of virtual memory (which is on the IDE drive), and it appears that the IDE drive is close to failure. The problem has gotten much worse over the past week, to the point where the filesystems are being corrupted. The last straw came an hour ago, when the forum database table that held the actual posts was lost. Obviously, we can’t continue like this.

So what are we doing to correct this? I’m working on restoring a backup copy of the database from this morning. With luck, it will be in a decent state but there is the very real possibility of data loss. In the meantime, our hosting provider is building a newer, faster server for us. Once it’s ready, we’ll rush to copy the data from the old server onto the new one and try and get us back online as quickly as possible. Still, we expect to be offline until the weekend at least.

We apologize for the delays and inconvenience this has caused. Trust me, I’m as frustrated as anyone else about this, if not more so. (After all, I’m the one trying to pick up the pieces here….) Hopefully once we have everything corrected it’ll be smooth sailing. :-)

EDIT – 7:00 PM CDT – It looks like the latest backup of the database is intact. Everything up to this morning at around 9 AM EDT appears to have been saved. What a relief… heh. Now we just need to wait for the new server to come online, and we’ll work from there. I’ll provide more information as it becomes available.

EDIT – 7:50 PM CDT – By the way, the Java chat is still available for those of you who come onto #mortalkombat (aka, the #MortalKombat Online IRC Network). You can reach it by clicking here.

EDIT – 7:20 AM CST 10 May – Thanks for the input, everyone. I just wanted to point out a few things based on comments I am seeing…

  • For those asking, when the site comes back up it will come back up with no changes in layout, software, etc. CCShadow hasn’t had time to work on version 8 of CDS (the backend of the site), and to expect us to change software packages at a drop of the hat is unrealistic at best.
  • Jonin01 is right in that this will take time. I’ve done everything I can do on this end; now it’s just up to our hosting provider to get a machine ready for us to migrate to. Once I hear the new machine is up and running, I can copy all of our old data to it and bring it online. Until then, it’s simply a matter of waiting.
  • Finally, for those offering assistance, it is much appreciated. Truth be told, we have everything under control; I suppose it helps a great deal when I do this stuff (system administration and support) as my paying job as well as for MK Online. It’s really not as catastrophic as it sounds; even if we were to lose the main server filesystem today I have a site backup elsewhere that can be used to get MK Online back up and running. We’re in good shape, all things considered; it’s more annoyances than anything else right now.

As before, once I get more information I’ll be sure to post it here. Thanks for your support and patience while we work to get these problems taken care of. :-)

EDIT – 2:20 PM CDT 11 May – The new server is ready for us to start working on it. However, being out of town and without a direct path to the internet, I can’t access the server properly to begin working on it. I should be home by 10 PM CDT, at which point I’ll begin working on it. If all goes well, we should be back up and running tomorrow evening. :-)

EDIT – 10:50 PM CDT 11 May – There’s a problem with the operating system’s configuration that’s preventing us from using it. Rather than going through hours of trying to fix it manually, I’ve had CCShadow contact our hosting provider with precise instructions for an OS reinstall. Hopefully sometime tomorrow it’ll be ready for the actual site reimplementation.

EDIT – 9:40 PM CDT 12 May – No, our hosting provider hasn’t rebuilt the server yet. I’m going to get with CCShadow as soon as he gets online about contacting them directly and getting them to hurry it up. I know everyone’s getting annoyed; I’m probably more annoyed than anyone at this point. Thanks for your patience.

EDIT – 3:35 PM CDT 13 May – Our hosting provider went ahead and made the needed changes without requiring a reinstallation of the operating system. I’m in the process of configuring the server and restoring the data. This will take a while, so please be patient.

EDIT – 7:50 PM CDT 13 May – All of the data is restored. We’re working now to get the new server onto the old server’s IP address and do some final testing. Once that’s done we’ll reopen the site.

EDIT – 8:50 PM CDT 13 May – It’s going to be a while, as their automated phone system is backed up and I can’t get a call through. CCShadow will call them when he gets home and get it online. I don’t know if we’ll be open tonight, but we’ll do our very best. Otherwise, we’re hoping tomorrow at the very latest.

EDIT – 8:25 AM CDT 14 May – The good news is that the new server is on the necessary IP address. The bad news is that I’m encountering some serious issues getting the site to work properly on the new server. I’ll keep working on it throughout the day to see if I can get the problems fixed.

EDIT – 11:30 AM CDT 14 May – CCShadow and I have taken a preliminary look at the server. Right now we think there’s some incompatibility or software conflict that’s causing CDS not to work properly. I’ll continue looking at it when I have time (which isn’t often as there are upgrade issues here at work that need resolving; as that’s my paying job it has to take priority), and CCShadow will do heavy-duty debugging when he gets home tonight.

EDIT – 3:10 PM CDT 14 May – If the testing we’ve been doing is anything to go by, we’ve knocked out all but one minor annoying bug. Right now we’re evaluating whether it’s safe to reopen the site now or not. If CCShadow gives the go-ahead, we’ll go ahead and reopen.

EDIT – 3:55 PM CDT 14 May – Never mind. It turned out that one bug is affecting more site functions than we thought. Reopening will have to wait until CCShadow’s had a chance to stamp out the bug. Sorry, folks… this is frustrating us as much as it is you. Hopefully we can be back up and running tonight…

EDIT – 10:35 PM CDT 14 May – We’re back up!

286 thoughts on “An explanation for Mortal Kombat Online’s downtime…”

  1. God Damn it! This sucks. Computers suck!

    Well, I wish all the luck to you upon fixing the problem. Good luck.

  2. That sucks… Hope its fixed when I get back sunday… Let me know if I can help or something…

  3. could be worse, they could have lost EVERYTHING! and we all would have to start from scratch and blah blah at least we know it will be back soon

  7. Weekend is almost upon us. Hopefully MKO should be ready by Monday. Let’s hope it works out!

    BTW, GGS to all the PUNKS I played last night on PS2 PAL. KAK whores still got whacked!

  8. Damn, I don’t think I’ve been MKOless for this long… like ever.
    I’m going into withdrawal.

    P.S. Ninjas still rule the earth.

  14. Damnit, I need this fuckin site, I can’t live without it!!! I can’t believe how long my awsomeness has been un-noticed

  15. You know what, Fuck this shit and fuck me in the ass hole. I really do have no life -_-

  16. Fuck you DARQUE! Ya’ll ain’t doin SHIT!! It does not take this long to repair this broken piece of shit site.

  17. excuse my language, I’m just a little bothered right now, there is nothing to do!

  19. I dont give a shit what they say, take your time & fix this problem, good luck

  21. Fuck you DARQUE! Ya’ll ain’t doin SHIT!! It does not take this long to repair this broken piece of shit site.

    I can feel the love. I can tell it runs deep.

    And normally you would be right. It normally does not take this long to fix the site… provided it is a software issue. This is a hardware issue, and there is nothing I can personally do until our hosting provider replaces the server. I’m stuck cooling my heels until I get word that the new server is ready. Once it IS ready (and I am hoping it will be before the end of the weekend), it should hopefully take a few hours to get the site back up and running, provided there are no work interruptions. (As stated in another site post, I’m on a business trip now and there are big upgrades at work happening this weekend.)

    The site will be back up as soon as we can get it. Until then, please realize that I’m as impatient as the rest of you and want to get everything fixed ASAP. My hands are just tied at this point by things out of my control.

  22. meh worse comes to worse wanna borrow my mallet for the redneck way of fixing a computer :p

  23. Oh man it sux that MKO aint working I can’t live without this site lol Good luck in getting it back up D’Arque I have faith in you man.

  24. Okey, for the idiot that posted using my name, I would really really appreciate if you dont do that. I never called Shyriu a faggot, some fool thought it would be funny if they posted using my name

  25. Okey, for the idiot that posted using my name, I would really really appreciate if you dont do that. I never called Shyriu a faggot, some fool thought it would be funny if they posted using my name

    That was Shiryu. I’ve edited his comment to reflect that.

    For future reference: if I catch someone spoofing others on here again, they’re being banned from the site once it’s back up. No exceptions.

  26. sayin hi its NeoScorpion by the way long time poster first or maybe 5th tie complainer :)

  28. I hope all goes well. I hadn’t visited the site in a few days and am wondering if I missed anything important ^^

  29. Grlrdf de MK, fgrrgxi id Bishop – grxxiglrf dergg grf Xill alo Armageddon u Wii, frrxilx ialx MKA Profiles alo grixllxif lergrrf. ]{0mbat Pavilion grxlfxi u Xill lgfilcx? Nahx ligr u de grxxff.

    Skaxxia u MKO

  30. Just hoping that MKO will run up within the next few days. Goodluck to the server next time around. Hopefully, something this severe doesn’t happen again. See ya till MKO is up and running. =]

  32. Personally, now that I think about it MKO could use the break from the flame wars lol.Good luck guys, see you there.

  34. I beleive that I speak for a lot of people when I say that we understand this is a technical issues and while we are eager to get back to our flame wars, useless conversation about innate subjects, arguing over who rules at m.k, and our general haberdashery, we in no way blame Darq for this. I suspect that alot of the more mature memebers are simply not bothering to comment because, well…..they actualy have lives outside of the net. I miss M.K.O. like everyone else and I will be here when it get’s back, regardless of how long it takes. Just take your time and do it right :-)

  37. this sucks, oh well it’ll give me some more time to study for my final college exams. Hope everything goes well and we’re able to get on again as soon as possible.

  40. Good to see things turning up. Good work D’arque.

    As much some most of like to return to MKO ASAP, I rather not have the situation fixed in a rush. Just think about it, do we want something like this to happen again? Hopefully, most of us don’t. Don’t ask when MKO will be up. Be patient.

  41. I know DAB has already linked it in his main post on here, but why don’t you people who enjoy the site so much stop by the chatroom or something to tide you over until the site is up and running? A lot of us from the site are there and it can be interesting chatting realtime to people you know on the site. There are exceptions, but hey it’ll give you something to do. lol

  48. Great progress Bishop! Let’s hope the ISP are able to follow your instruction on OS installation and give MKO back to us. Good luck!

  50. I agree with 5animals. Most of the mature members *excluding Jerrod and such* have either chosen not to post at the sheer stupidity of some of the other posters, or they’re patient enough to know that your hands are tied. I, although not a full computer geek, knows what it’s like to have hardware fouling up, not software. I am also aware at how there’s nothing you can do until your new server is ready. So I just say good luck once the server is ready to get MKO back up, and to mostly everybody else, just be patient and wait. There’s no problem posting here to give your support, but after that, go outside, have fun, don’t sit here and flame other people who are trying to flame Bishop. You don’t see him flaming back do you? Either way, I hope all goes well with the rebooting process.

Comments are closed.