An explanation for Mortal Kombat Online’s downtime…

D'Arque Bishop

Well, Mortal Kombat Online is facing an extended downtime. Apparently we are suffering from such a catastrophic hardware failure, so to speak, that it’s going to take a few days and a LOT of work to get everything back up and running the way it was. I’ve left the standard disclaimer on the site itself, but I figured I would use this post on my own site to give an in-depth explanation as to what happened and allow others to give feedback if necessary.


In any event, here’s what’s happening.

Back in Christmas of 2005, the hard drive on the main MK Online server suffered a crash. We were forced to get our hosting provider to replace the drive; when we had them replace it, we demanded that they put in mirrored drives like we had when we first got our dedicated server with them. (At the time, we were actually on a new dedicated server.) They did so, but instead of using two identical drives they used a single SCSI (heavy-duty) drive and a much larger IDE (consumer-level) drive. We had meant to get with our hosting provider to fix that, as such a configuration is much less than ideal. However, we never got around to it as bigger issues kept coming up.

Well, recently we’d been noticing system crashes, caused by memory errors. We soon realized that it was because of virtual memory (which is on the IDE drive), and it appears that the IDE drive is close to failure. The problem has gotten much worse over the past week, to the point where the filesystems are being corrupted. The last straw came an hour ago, when the forum database table that held the actual posts was lost. Obviously, we can’t continue like this.

So what are we doing to correct this? I’m working on restoring a backup copy of the database from this morning. With luck, it will be in a decent state but there is the very real possibility of data loss. In the meantime, our hosting provider is building a newer, faster server for us. Once it’s ready, we’ll rush to copy the data from the old server onto the new one and try and get us back online as quickly as possible. Still, we expect to be offline until the weekend at least.

We apologize for the delays and inconvenience this has caused. Trust me, I’m as frustrated as anyone else about this, if not more so. (After all, I’m the one trying to pick up the pieces here….) Hopefully once we have everything corrected it’ll be smooth sailing. :-)

EDIT – 7:00 PM CDT – It looks like the latest backup of the database is intact. Everything up to this morning at around 9 AM EDT appears to have been saved. What a relief… heh. Now we just need to wait for the new server to come online, and we’ll work from there. I’ll provide more information as it becomes available.

EDIT – 7:50 PM CDT – By the way, the Java chat is still available for those of you who come onto #mortalkombat (aka, the #MortalKombat Online IRC Network). You can reach it by clicking here.

EDIT – 7:20 AM CST 10 May – Thanks for the input, everyone. I just wanted to point out a few things based on comments I am seeing…

  • For those asking, when the site comes back up it will come back up with no changes in layout, software, etc. CCShadow hasn’t had time to work on version 8 of CDS (the backend of the site), and to expect us to change software packages at a drop of the hat is unrealistic at best.
  • Jonin01 is right in that this will take time. I’ve done everything I can do on this end; now it’s just up to our hosting provider to get a machine ready for us to migrate to. Once I hear the new machine is up and running, I can copy all of our old data to it and bring it online. Until then, it’s simply a matter of waiting.
  • Finally, for those offering assistance, it is much appreciated. Truth be told, we have everything under control; I suppose it helps a great deal when I do this stuff (system administration and support) as my paying job as well as for MK Online. It’s really not as catastrophic as it sounds; even if we were to lose the main server filesystem today I have a site backup elsewhere that can be used to get MK Online back up and running. We’re in good shape, all things considered; it’s more annoyances than anything else right now.

As before, once I get more information I’ll be sure to post it here. Thanks for your support and patience while we work to get these problems taken care of. :-)

EDIT – 2:20 PM CDT 11 May – The new server is ready for us to start working on it. However, being out of town and without a direct path to the internet, I can’t access the server properly to begin working on it. I should be home by 10 PM CDT, at which point I’ll begin working on it. If all goes well, we should be back up and running tomorrow evening. :-)

EDIT – 10:50 PM CDT 11 May – There’s a problem with the operating system’s configuration that’s preventing us from using it. Rather than going through hours of trying to fix it manually, I’ve had CCShadow contact our hosting provider with precise instructions for an OS reinstall. Hopefully sometime tomorrow it’ll be ready for the actual site reimplementation.

EDIT – 9:40 PM CDT 12 May – No, our hosting provider hasn’t rebuilt the server yet. I’m going to get with CCShadow as soon as he gets online about contacting them directly and getting them to hurry it up. I know everyone’s getting annoyed; I’m probably more annoyed than anyone at this point. Thanks for your patience.

EDIT – 3:35 PM CDT 13 May – Our hosting provider went ahead and made the needed changes without requiring a reinstallation of the operating system. I’m in the process of configuring the server and restoring the data. This will take a while, so please be patient.

EDIT – 7:50 PM CDT 13 May – All of the data is restored. We’re working now to get the new server onto the old server’s IP address and do some final testing. Once that’s done we’ll reopen the site.

EDIT – 8:50 PM CDT 13 May – It’s going to be a while, as their automated phone system is backed up and I can’t get a call through. CCShadow will call them when he gets home and get it online. I don’t know if we’ll be open tonight, but we’ll do our very best. Otherwise, we’re hoping tomorrow at the very latest.

EDIT – 8:25 AM CDT 14 May – The good news is that the new server is on the necessary IP address. The bad news is that I’m encountering some serious issues getting the site to work properly on the new server. I’ll keep working on it throughout the day to see if I can get the problems fixed.

EDIT – 11:30 AM CDT 14 May – CCShadow and I have taken a preliminary look at the server. Right now we think there’s some incompatibility or software conflict that’s causing CDS not to work properly. I’ll continue looking at it when I have time (which isn’t often as there are upgrade issues here at work that need resolving; as that’s my paying job it has to take priority), and CCShadow will do heavy-duty debugging when he gets home tonight.

EDIT – 3:10 PM CDT 14 May – If the testing we’ve been doing is anything to go by, we’ve knocked out all but one minor annoying bug. Right now we’re evaluating whether it’s safe to reopen the site now or not. If CCShadow gives the go-ahead, we’ll go ahead and reopen.

EDIT – 3:55 PM CDT 14 May – Never mind. It turned out that one bug is affecting more site functions than we thought. Reopening will have to wait until CCShadow’s had a chance to stamp out the bug. Sorry, folks… this is frustrating us as much as it is you. Hopefully we can be back up and running tonight…

EDIT – 10:35 PM CDT 14 May – We’re back up!

286 thoughts on “An explanation for Mortal Kombat Online’s downtime…”

  2. this fukkin website suuuuuuukx , ya better shutdown this shit!!!!!!!!!!!!!!

  3. Not cool.

    Anywayz, hope all gets better from here. Take all the time that you need, and good luck! =)

  4. Well, it sounds like things are looking up at the moment, judging by your updates. I would like everyone who entered SOTW to know that voting for this week and entries for next week will still happen, so do not fret. If I’m not mistaken the theme for next week is No Renders, so if you want to get a head start on that feel free.

  5. A few of you people posting have to realize that the site can’t be magically fixed in a matter of hours. There is a lot of work behind the scenes that needs addressed. It’s out of DAB’s hands until the webhost gets the server up and running and tests are made to put the backups on again and to ensure it works smoothly again. If they did a slam bang job and put things back online too soon, it might end up crashing again and they could lose more work and time than if they had waited. Just be patient it should be up and running by the weekend as they have said barring difficulties. I enjoy reading/posting on the forums too, but you have to be patient and wait these things out.

  6. Find something else to do with your time than wasting it on some Mortal Kombat site. A hobby perhaps?

  7. I don’t actually understand why people are acting like this is the end of the world. It’s only going to be out for few days. Even xbox.com suffers outages. How about that?
    Shit happens and it’s good to know we have people taking care of it. I’m sure you guys have the technical capability to deal with this minor glitch but if you need a hand, please feel free to look me up!

  9. Awwwww man! I havent been on MKO in like 4 days now and i finally get home and this happens. I hope everything works out for you. Best of luck!

  10. Well if you think about it it’s not all bad because we could all use a break from mko so…

  11. One more thing.

    Will there be any site changes at all, after all of this has been fixed? I hope so. : )

  12. Hey.Ummm im a huge fan of mortal combat.Unfortunately i cant find any games on PS2 in the shops.Its really annoying.Do i need any software or to download anything for this site?Sorry im a reall noob at using computers hahaha.So if anyone out there willing to spend afew minuets replying to this on the website about my question,it would be greatly appreceated.thanks,Rob

  13. Hey.Ummm im a huge fan of mortal combat.Unfortunately i cant find any games on PS2 in the shops.Its really annoying.Do i need any software or to download anything for this site?Sorry im a reall noob at using computers hahaha.

    MK Online isn’t a site on which you can play MK games. We’re a news and community site. If you want to play MK on your PC, I suggest picking up Midway Arcade Treasures: Deluxe Edition, as it has the first three MK games on it. Otherwise, try Amazon or GameStop online for PS2 versions.

  14. i’m sure that the problem will be fixed as soon as possible.
    i’ll wait!

  15. Thanks for posting the Java chat link. Hopefully the site will be back up soon!

  16. This is very displeasing to say the least. I hope you guys can get the site back soon. Well in any case good luck guys.

  17. I hope things go relatively smooth for you guys.

    P.S. Ninjas rule the earth.

  21. Man on my site it’s equal MK Warehouse is there and newgrounds
    and Tabmok99 too, what are you talking about that Mk Online is important?
    MK Online is a f**kin site. There’s nothing interesting.

  24. Too bad about the site. Good luck to you guys and I can’t wait to get back onto mkonline and start posting

  27. *crosses fingers* I’ll keep checking once a day to see what is up, good luck

  29. Wow, Ladymage finally decides to return when MKO is down. Where have you been? We’ve missed you.

  30. MINION… that picture had already been posted in the forum…
    Now, this is the new drive by!

  32. Shyriu, oh well. I didn’t see it. Maybe it was. But im not gonna scroll through everyone post on this black board to find out. If it’s old new’s yay. If not yay x2.

  33. that sux i just got this for my birthday today hope you fix it soon later

  35. show de bola,muito bom mesmo,mas eu quero me cadastrar no site de voces.

  37. DAMN IT DAMN IT DAMN IT!”!!!!!!!!!!!!!!!!!!!!!!!!!!!!! FUCK YOU!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!damn it, damn it, damn it, damn it, fuck you!!!!!!!!!!!!!!!!!! YOU BITCHES HURRY UP DAMN IT, FFFFUUUUCCCKKK!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! YOU ASSHOLES!!!!!!!!!!! YOU ASSHOLES!!!!!!!!!!!!!!!!!!!!!!!!! YOU ASSHOLES!!!!!!!!!!!!!!!!!!!!!!!!!! YOU ASSHOLES!!!!!!!!!!!!!!!!!!!!!!!!!!! YOU ASSHOLES!!!!!!!!!!!

  39. hehe I should explain myself, wedding and honey moon, I came back a few days ago kind of (naturally the site dies my fault I swear it didn’t like me XD)

  40. Hey guys, thanks for the update this sucks for sure lol.But I’m sure things will be fine, some people just seriously need to be patient.Do what you do best D’Arque Bishop I’m sure you don’t need anyone to tell you what to do.I mean come on(to the people thinking they know what’s wrong) these guys have run MKO for years I think they know what they’re doing. ; )

    I’ll be here like the rest of you until MKO is back up.Good luck guys!

  42. Good luck Guys. This must be a bitch to fix. But you guys can do it. Good luck :)

  45. I love how some of these comments don’t have shit to do with the actual post, and are there just to bash the site and the hard work that D’Arque and others are doing to fix the problem. Morons.

  46. Lol, flameshang Proffecer Ahngaka, hasent been on and didnt update it. It would take a while.

  47. sup…just thought i’d put my two cents in, good luck with getting everything back up and can’t wait for the site to come up again, peace

Comments are closed.