The Importance of Backups

[Justin]

Howdy, guys. I wanted to make a quick thread to educate everyone on the importance of having regular, steady backups available for your server.

So let's say you've got a wiki on a server, and everything's going great. You've got a decent userbase and you've built up a nice amount of content. Then something happens. Maybe the database crashes for some reason. Whatever the reason for it, you've now lost your database and can't recover it.
This is where the prepared webmaster will clear the bugged database and recreate it from a backup.
This is where the unprepared webmaster will spend hours in vain trying to recover data until finally giving up and having to admit to his users that the data is gone.

Not sure about you guys, but I like to be the former. It's very important to maintain a system of backups, and I'll be going over some tips about creating a system for making regular backups. Note that you need shell access to your server for this. If you don't have it, you can usually get it enabled by your hosting company if you open a support request.

1. Create a script to backup your database regularly
Your first step should be to write a script which can run as a cron job to back up the DB. If you're too lazy to do this, send me an email. I've created a modular script which you can set up with your site's configuration that will provide you with a stable backup system.

Things to consider in your script:
A) Your script should not be in your web directory. To run automatically, it needs to have the password for your mysql user in it, and you don't want to publicize that information.

B) Consider keeping multiple backups, in case an underlying issue occurs but remains undetected for a little while. My recommendation is as follows:
Low-traffic ( < 20 edits per day ): Backup at least twice a week, keeping two weeks' worth of backups.
Mid-traffic ( 20-100 edits per day ): Backup 3-4 times a week, and keep at least 5 backups at one time.
High-traffic ( > 100 edits per day ): Backup every day, and keep a full week's worth of backups.
Naturally, the more edits you have per day, the more often you want to backup the DB. You want to make sure you stay up to date on the data in your backups.

C) Be sure to lock your database before backing it up. If you open LocalSettings.php, you can put the database into lock mode automatically by setting $wgReadOnly. You should set it with a message to the users explaining that you are performing maintenance and that the wiki is temporarily read-only. Here's a sample message:
$wgReadOnly = "The database is currently locked from editing while we perform routine maintenance. Check back in about fifteen minutes and it should be unlocked again. We apologize for the inconvenience.";

D) Plan a solid location for your backups. If possible, try to store your backups on a remote server, or on a separate drive from your database. It's very counterintuitive to keep your DB and backups in the same location on the same drive. If there's a filesystem failure, you lose the backups, too! I'M WORKING WITH SOMEONE WHO HAS THIS PROBLEM; DO NOT REPEAT HIS MISTAKE!

2. Test your script.
Once you've made the script, test it on a development server or a local install. Don't test it on your production server, as you run the risk of a small bug destroying everything.

3. Automate the script.
Once you've made the script, you need to automate it. If you're running a Microsoft IIS server, you'll need to research how to do that for yourself. (Sorry, but I don't advocate or support Windows.)
Linux distros usually allow each use to set their own crontab file via the crontab command. You can type crontab -e to edit the file with your default text editor. If you don't understand how a crontab works, this article goes in-depth about how to format a crontab file. Choose a time when your editing traffic is the lowest. Usually, the optimal time will be 7-10 AM UTC, as that is 2-5 AM EST in America. If the bulk of your traffic doesn't come from the US, identify your heaviest area of traffic, and tailor your database locking to non-peak traffic times for that region. If you have root access, you can view crontab files in /var/spool/crontab/, but it is not recommended to edit them directly. Use crontab -e instead. Do not place it in a system cron (/etc/crontab), because it will not run as the user who owns the files. It will instead run as root, and if the script manipulates any files, root will gain ownership of them.

4. Create your initial backup.
Run the script yourself! But again, don't do it during peak hours. You'll inconvenience your users, and could even cause some to think your site is broken. MediaWiki's readonly message is very unnoticeable at the top of edit forms, and it provides no indication other than a notice. Users can still edit the form and even click the save button, but the DB will refuse the connection and will not save the edit.
Once you have your initial backup, test its integrity and even try to import it into a local installation to ensure that it's working properly.

5. Monitor your backups regularly.
Pay attention to your script. It's a good idea to check your mail every time you login, just to be sure that there's no errors. And just to clarify, I'm talking about the account's mail on the server. Again, I can't help IIS users, but Linux distros store their mail in a file located in /var/mail/(username). I always read mine with:
tail -n 100 /var/mail/(name) | less
That will pull the last 100 lines from the mail file, and display it in a dynamic window in which you can scroll up or down with the arrow keys. By default, cron will automatically email the user any errors produced by scripts in their cron, which can be used for debugging.
This is good practice in general. After any major updates or package upgrades, test your script to make sure that it's still functional.

6. Relax.
Now you have nothing to fear. A database corruption can't stop you!


Some of you may be wondering why I took the time to write this out. Well, it's because I've noticed a disturbing trend of webmasters not backing their sites up, which can WILL lead to trouble in the end. It doesn't matter how long your site runs reliably; it will fail one day. Computers are not perfect. The disaster with WiKirby losing its database should serve as enough of a cautionary tale. In fact, Zelda Wiki lost some data, too, although we only lost a week's worth (and it was my own fault, anyway, because I failed to take a necessary step after an upgrade) because I was prepared.

So remember, people. Be prepared, and don't get caught with your pants down. Never dismiss the importance of backups.

[FlyingRagnar]

An excellent write up Justin.  Can you also share your thoughts on backing up the file system?  Since this can be many, many GBs of data, I'm curious how often you do a tar of the files and then store them remotely somewhere.  Weekly? 

[Greenpickle]

^Wouldn't you want to use rsync or equivalent instead for that?

[FlyingRagnar]

rsync would certainly work.  That would be more of a mirror than an archive/backup I suppose.  When I have done it for our wiki it has been to just create a tar and then upload somewhere else.  This is what is described on the mediawiki site (http://www.mediawiki.org/wiki/Manual:Backing_up_a_wiki#File_system).  However, that can be very time consuming to do all at once, which is why I thought there might be progressive backup methods that use crons, etc. that some wikis might already have implemented.

[Greenpickle]

Hmm, that's an interesting point: what if you want to end up with a single file, but still benefit from rsync's deltas.  Perhaps mount the archive and mirror to the mount point?  (A quick search suggests 'archivemount' could do it.)

[Justin]

An excellent write up Justin.  Can you also share your thoughts on backing up the file system?  Since this can be many, many GBs of data, I'm curious how often you do a tar of the files and then store them remotely somewhere.  Weekly?

Backing up filesystems is a nasty piece of work, since they can grow to astronomical sizes. What I'd recommend is that you rsync the wiki directory to your backup server and then tar and gzip it.
To accomplish this, you'll either have to set the username and password for the login on the other server, or you can set up SSH keys. I'd recommend creating a separate user whose sole function is to perform the rysnc, for security purposes. You could run the rsync on either the webserver or the backup server, but I recommend the backup. I don't recommend piping the rsync through tar, though, unless you're the only one with access to the files. I say this for two reasons:

1. An initial rsync takes awhile, but anything after that is trivial. It checks for differences in files, and this is usually quick and efficient.
2. A user changing something mid-rsync could cause things to go FUBAR, and could even create a broken backup. For example:

Say someone is testing a new extension, and discovers that it doesn't do what he or she thought. The user then removes the
require_once() in Localsettings and removes the extension. However, the rsync has already synced the LocalSettings, and it ends up creating a backup with that line still there, but without the extension. You now have a non-functional backup, and for a production server, error_reporting is usually turned off by default, which means that:
1. You have no clue why it's broken
2. The only two ways to figure out are extensive testing or exposing your errors by temporarily enabling error_reporting.

The first situation may take a long time, which equals lots of downtime. This is BAD. The second effectively tells the whole world about your stack trace, and can give potential attackers something to work with.
I know in this situation, a stack trace isn't really going to hurt you, but this is just one scenario. Imagine if (god forbid) you used eval() somewhere. Hold your stones, people, I'm not advocating the use of eval(). But let's say it's used by something, and a potential attacker sees that exposed. They now effectively know where they can insert malicious PHP code to do what they want.

Just because I mentioned eval(), I feel it's my duty to let you know that eval() is dangerous and there is almost never any call to use it. It evaluates a passed string as PHP code, and the potential for its abuse is very real. If you ever see ANYTHING that uses eval(), steer clear. Steer VERY clear. It's more trouble than it's worth, I promise.

Now, ZW rsyncs the full wiki directory to a secure location and then runs a script to create a zipped archive at a scheduled time. It also uses a logrotate style method which deletes the backups older than seven days. You may not be able to do this if your server doesn't have enough room, but I definitely recommend it.

Sorry for the late reply. I'm up to my ears in work between Zelda Wiki and a couple other sites.

[Greenpickle]

Hmm, files changing as you back up; I hadn't thought of that.  There are probably solutions out there that cope with that, but if you can reasonably expect files not to change very often at all (which they won't here, I don't think: images will be added, but that won't cause any breakage), it shouldn't be much work to write a script to handle it.

Just store modification timestamps (or checksums, if you want) before backing up, and then again afterwards.  If any files that existed before changed, undo the backup (really easy if you use a snapshot-capable filesystem like btrfs on the destination), then try again (probably with exponential backoff or something, if you can be bothered).

[Justin]

I take it into account because there are other staff members with FTP access. While I'm the only one who has the privileged account for high level work, we also have an account which allows the rest of our staff to access and modify the files in the event that an extension needs to be reconfigured or something.

Although I will say that those are some pretty good ideas, Steve. Of course, I'd like to point out that you should always set $wgReadOnly before backing up anything, which will also prevent file uploads. That will narrow your main point of contention to others changing the files as you back up. But, as you've suggested, storing modification times or doing a checksum could greatly assist with ensuring the quality of your backup.

As far as btrfs goes, does it support journaling? I would never recommend a filesystem that doesn't, regardless of what cool features it has. Things break eventually, and journaling has proven invaluable in figuring out why on more than one occasion.

[Greenpickle]

Although I will say that those are some pretty good ideas, Steve.

...Not that I mind, but why are you calling me Steve?

Of course, I'd like to point out that you should always set $wgReadOnly before backing up anything, which will also prevent file uploads. That will narrow your main point of contention to others changing the files as you back up.

Ah, yes, of course.  I'm not too familiar with MediaWiki on the backend, so even after you mentioned it in your opening post it slipped my mind.  That would certainly make things easier.

As far as btrfs goes, does it support journaling? I would never recommend a filesystem that doesn't, regardless of what cool features it has. Things break eventually, and journaling has proven invaluable in figuring out why on more than one occasion.

Yes, btrfs is quite a new filesystem, so it's pretty much a given.  Last I checked, it wasn't quite ready for use in production environments, but I think some bleeding-edge Linux distributions have made it the default.

[Justin]

Not sure. I don't even know how I knew that was your name.

But anyways, I'm not a fan of bleeding edge stuff because it's all great "in theory." In practice, things can get ugly. I tried using the bleeding edge Ubuntu release and within an hour had deleted it in favor of the 12.04 LTS.

[Greenpickle]

Heh, it's not my name, that's why I brought it up.

Yeah, I know; by that I was really saying that it's not ready yet, but it's starting to get used, so it's getting there.

[KidIcarus]

Is there a concern that someone could get ahold of the back-up copies?