Downtime & How To Make A Repair Job On Mysql Go Faster

Screw Driver

The site crashed yesterday morning at around 9am, although it’s hardly noticable until a couple hours later. I didn’t notice it until I started picking stories to feature on the homepage. Whenever i tried to feature an article, it would stop the site from loading up.

From that, I knew that the MySQL database must be checking for corrupt tables. And if it’s doing that, a crash must have occurred. Checking out stats at my ISP confirmed that the server was restarted at 9am.

When a table’s being checked, it still could be read, so people can still access the site. It’s just when you I tried to update it, it would hold up everything else from running.

Eventually, one of the tables was found to be corrupted and needed to be repaired. When a table’s being repaired, you can’t read or write to it, so it’s down time for the site until it’s fully repaired. That took about three hours.

One of the worst things you can do while a table’s being repaired is to cancel it or restart the server. The entire corrupted table could be deleted, and there’s no way to undo that. I found that out the hard way with a previous site outage. You just have to wait it out, unless you have a backup that you can readily deploy.

Still, three hours is a long time to wait. When I checked a list of running processes, I spotted corrupted table with “repair with keycache.” Apparently, that’s a really slow way to repair a table.

The way to avoid that is to add or modify a couple settings in MySQL: “myisam_max_sort_file_size” and “myisam_max_extra_sort_file_size.”

Set those values to be higher than the size of the largest index in your database.

The index for the corrupt table that was being repaired was about 1GB, and that took three hours. My largest index is 15GB. That would take a couple days to complete if that table gets corrupted. That wouldn’t be fun.

I upgraded my storage space and set the values for the two config settings to 20GB. Should that 15GB table get corrupted, a repair job should only take hours, not days.

Sections:  note from the editor   
Topics:  database   mysql   site outage   site updates   system admin   webmaster   wopular   
  • "Storm The Gates" Mobile Game
    My friends just launched a mobile game called "Storm the Gates"! It's available on both Android and iOS. I'm not much of a gamer, so I didn't know what to do when I started it. But my 9-year-old son turned it on and knew exactly what to do. More
  • "Ip Man 3" Has New Release Date in China and Posters to Prove It
    Over at, I’ve written two articles regarding “Ip Man 3,” the film that’s holding its own in Asian territories against “Star Wars: The Force Awakens.” If you’re not familiar with Asian cinema,”Ip Man” one of the most popular franchises in that area of the world. More
  • Live-Blogged The Golden Globes at MWB
    Yesterday, I live-blogged The Golden Globes -- for various reasons. First, it gets traffic. Second, I might as well do something useful while watching TV. Lastly, I was looking forward to seeing Ricky Gervais host the event again. Nothing against Tina Fey and Amy Poehler, but he’s my favorite. A google search revealed that the last time I did this for this award show was More
  • Kings vs. Mavs: Ominous Stats, News for Sacramento Before Game
    The preview on and the injury report from don’t look good for the Sacramento Kings going into tonight’s game against the playoff-bound Dallas Mavericks. More
  • What?! Kings Beat Thunder, 116-104
    (Image from the Sacramento Bee. Click here to view more images from the game.) What a win. It was a wild one, especially in the first half. We [Sacramento Kings] were down by 17 in the first quarter. In frustration, I was about to turn off the TV, but the teaml called a timeout. More
  • Another Network Outage ... Argh ...
    I spent the whole day dealing with servers. The facility where Wopular and MoviesWithButter are hosted were having some network issues. I was able to get MWB up and running within an hour or so. Wopular was a whole different story. More
  • Moved SimplePie File Caching to Database
    A while back, My ISP told me my backups took half a day to complete, which is too long. If anything goes wrong during that period, the process would have to start all over again, which happened quite often. The customer rep told me that my server had too many files, and even though they didn’t take up much disk space when added up, the process of copying each one of them takes a lot of time. More
  • Relieving High Server Load by Blocking Search Bots
    Over the years, whenever the site was slow to load up, I always went through the same routine. I checked traffic logs to see if it was due to a sudden increase in traffic. If not, I would look at the mysql slow query log to see if there were slow queries bogging up the database. More
  • At Cannes Covering the Film Festival
    You had probably noticed that the site was down for a couple hours on Monday and hadn’t been updated until this morning. I was on my way to the airport for Cannes and, of course, the site had to go down. More
  • AddThis Removes "via @addthis" Suffix in Tweets
    I just noticed in the last couple of days that the sharing platform AddThis has removed “via @addthis” at the end of tweets using their widget. I use AddThis throughout the site. It’s a handy little widget that allows users to share articles through any social bookmarking/sharing site, like Facebook, Twitter, StumbleUpon and Pinterest. As a website owner, I use it to tweet articles to my Twitter account. More


Comment On This Story