Automatic Amazon EC2 snapshot backups

Amazon Web Services EC2 platform has a great snapshot feature that can be used as a backup for your server and files. Here’s how I’ve automated the process so there is a backup around in case something bad happens.

This process is written and tested for Ubuntu instances. I’m working on tweaking it for RHEL as well.
(Note: I’m the first to admit that I’m a terrible programmer. Hence, I’ve adapted this process from other people’s work around the web. Credit goes out to John O’Connor for getting me most of the way there.)

It would benefit you, especially if you have a MySQL or other type of database, to have it on a volume that has a filesystem that supports snapshots. XFS is one example.
To set up XFS support:
apt-get install -y xfsprogs
You’ll now have the option to use XFS on new volumes you attach and format.

Next we’ll setup the repository Alestic PPA (personal package archive) that’s on Launchpad.net:
1. add-apt-repository ppa:alestic
2. apt-add-repository ppa:awstools-dev/awstools
3. apt-get update
4. apt-get install -y ec2-api-tools ec2-consistent-snapshot

We’ll also need to make sure we have a JDK installed correctly:
apt-get install openjdk-6-jdk

Now come the script files themselves:
vol-snapshot.pl

#!/usr/bin/perl
#
# File: vol-snapshot.pl
#!/usr/bin/perl -w

##############################
# Global Variables
##############################

my $EC2_HOME = ‘/usr/’;
my $EC2_BIN = “$EC2_HOME/bin”;
my $EC2_PRIVATE_KEY = ‘/home/pk-*.pem’; # AWS Private Key file
my $EC2_CERT = ‘/home/cert-*.pem’; # AWS Certificate file
my $EC2_Volume_ID = ‘vol-*’; # Name of the Volume we’re backing up
my $SiteFilePath = ‘/home/user’;
my @oldsnapshots = ();
#my @volumes = ();
my @snapshots = ();
my $SNAPSIZE_MAX = 3; # This is the number of snapshots to save
my $DEBUG = 1; # Set this to 0 to get less output.  I recommend leaving this alone.

##############################
# Subroutine Prototypes
##############################
sub getSnapshots();

##############################
# Main
##############################

print “$_\n” if $DEBUG;
`ec2-consistent-snapshot—mysql—mysql-defaults-file /root/.mysqldefaults—xfs-filesystem $SiteFilePath—description “Site Files & DB backup” $EC2_Volume_ID`;

@snapshots = getSnapshots();
my $snapsize = @snapshots;
print “$snapsize number of snapshots\n” if $DEBUG;
if ($snapsize > $SNAPSIZE_MAX) # check for and delete the oldest snapshot
  {
my $oldsnapshot = `$EC2_BIN/ec2-describe-snapshots -K $EC2_PRIVATE_KEY -C $EC2_CERT | grep $EC2_Volume_ID | grep backup | sort -k 5 | head -1 | awk '{print \$2}’`;
      print “$oldsnapshot\n”;

      `$EC2_BIN/ec2-delete-snapshot $oldsnapshot`;

  }


##############################
# Subroutines
##############################

sub getSnapshots()
  {
my $snapstring = `$EC2_BIN/ec2-describe-snapshots | grep $EC2_Volume_ID | grep backup`;
my @snapshots = split(/\n/, $snapstring);
my @snap = ();
my $snapcount = 0;
foreach my $snapshot (@snapshots)
{
if ($snapshot =~ /SNAPSHOT\s+(snap-(?:\S+))/)
{
($snap[$snapcount]) = $snapshot =~ /SNAPSHOT\s+(snap-(?:\S+))/;
$snapcount++;
}
    }
  return @snap
  }

vol-snapshot.ksh

#!/bin/bash
#
# File: vol-snapshot.ksh
# Author: John O’Connor
# Copyright (C) 2009, John O’Connor
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTIBILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

export EC2_HOME=’/usr/’
export EC2_BIN=$EC2_HOME/bin
export EC2_PRIVATE_KEY=/home/pk-*.pem # AWS Private Key file
export EC2_CERT=/home/cert-*.pem # AWS Certificate file
export PATH=$PATH:$EC2_BIN
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/

touch /root/logs/snapshot_`date +%Y%m%d_%H%M%S`.log
./vol-snapshot.pl > /root/logs/snapshot_`date +%Y%m%d_%H%M%S`.log

.mysqldefaults

[client]
user = username
password = password
host = localhost


Now onto tweaking the server instance with these files on it.
1. Make the vol-* files executable
2. Get a x.509 Private key file and Cert file and put them somewhere safe on the server. Enter the path into both vol-snapshot files
3. Add the correct volume name in the vol-snapshot files
4. You can add the java export path to the .bashrc file just for the sake of being redundant
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
5. Edit the .mysqldefaults file to grant access to doing database freezes
6. Create log file directory - default path in the file is /root/logs

Next, test the creation by running the vol-snapshot.ksh file. Once it’s successfully creating the snapshots in the AWS console, set up a cron job running the vol-snapshot.ksh file as often as you’re wanting a backup taken.

 

Comments

Returning visitor? Please login or register.

If you don’t want to set this up yourself, you can use Skeddly to create automatic EC2 snapshots.

http://www.skeddly.com

Matt Posted on: May 16, 2012 at 02:04 PM

Leave A Comment

Please help us stop spam by answering the question below: