Update: the complementary article to this one, on Rotating EBS snapshots has now been posted.
After the recent failure of EBS volumes on Amazon’s cloud, backup is quite likely something to be considering. If, during the aforementioned incident, recent snapshots of the affected EBS volumes were available, it might have been possible to relaunch instances and recreate new volumes in unaffected zones and minimize downtime.
The ec2-consistent-snapshot script is designed to minimize data loss by freezing the file system and flushing MySQL tables. However, it is released for Ubuntu, and some pre-requisites are necessary in order for it to run on Amazon’s Linux distribution (CentOS/RHEL derived).
A simplified version of the steps below worked well on the latest version of Amazon’s Linux (2011.09), since it includes the EPEL repository by default. (Note: priorities are enabled by default on Amazon’s Linux – which means most of the perl modules below will come from the amzn repository).
yum --enablerepo=epel install perl-Net-Amazon-EC2 perl-File-Slurp perl-DBI perl-DBD-MySQL perl-Net-SSLeay perl-IO-Socket-SSL perl-Time-HiRes perl-Params-Validate ca-certificates wget -O /opt/aws/bin/ec2-consistent-snapshot http://bazaar.launchpad.net/~alestic/ec2-consistent-snapshot/trunk/download/head:/ec2consistentsnapsho-20090928015038-9m9x0fc4yoy54g4j-1/ec2-consistent-snapshot chmod 0775 /opt/aws/bin/ec2-consistent-snapshot
The procedure below is obsoleted by that above, but retained for interest sake.
The following steps were able to get ec2-consistent-snapshot operational, with a minimum of hassle (adapted from: http://pojo-mojo.blogspot.com/2011/03/ec2-consistent-snapshot-for-centos.html).
yum install perl-Params-Validate PERL_MM_USE_DEFAULT=1 cpan Net::Amazon::EC2 yum install perl-File-Slurp perl-DBI perl-DBD-MySQL perl-Net-SSLeay perl-IO-Socket-SSL perl-Time-HiRes cpan Mozilla::CA cd /opt/aws/bin wget http://bazaar.launchpad.net/~alestic/ec2-consistent-snapshot/trunk/download/head:/ec2consistentsnapsho-20090928015038-9m9x0fc4yoy54g4j-1/ec2-consistent-snapshot chmod 0775 ec2-consistent-snapshot
A few notes about the above items:
- Params::Validate is needed by Net::Amazon:EC2 (or one of its dependencies), however, CPAN failed to install it for me. Yum did the job without issue.
- The CPAN installation of Net::Amazon::EC2 takes a long time – about 30 min – and has numerous dependencies. The ‘PERL_MM_USE_DEFAULT=1’ setting will let CPAN choose the defaults (essentially saying yes to all dependencies). This probably won’t work if you haven’t setup CPAN previously.
- There is a good chance many servers already have some of the packages yum is asked to install – however, yum will simply ignore those.
- I have opted to place the script in the same directory as the AWS utilities (/opt/aws/bin).
You will need to specify the mount point for ec2-consistent-snapshot, and might want to know the type of each file system – both can be found by running:
If you are going to create a backup of MySQL data, and wish to take advantage of ec2-consistent-snapshot’s features, you will need to create a user with
REPLICATION CLIENT, and
SUPER privileges, similar to the following:
GRANT RELOAD, REPLICATION CLIENT, SUPER ON *.* TO 'snapshotter'@'localhost' IDENTIFIED BY 'PASSWORD'
Since the ec2-consistent-snapshot command requires your access key and secret key, and will copy the same into the snapshot (which is stored in S3), it may be preferable, from a security standpoint, to create a user (with limited privileges that can create snapshots) via AWS Identity and Access Management (IAM) specially for this task.
As an example of a non-database volume:
ec2-consistent-snapshot --aws-access-key-id MYACCESSKEY --aws-secret-access-key MYSECRETKEY --region us-east-1 --freeze-filesystem /var/www --description "Website Files" vol-1234abcd
As an example of a MySQL database volume:
ec2-consistent-snapshot --aws-access-key-id MYACCESSKEY --aws-secret-access-key MYSECRETKEY --region us-east-1 --freeze-filesystem /data --mysql --mysql-host localhost --mysql-socket /data/var/lib/mysql/mysql.sock --mysql-username snapshotter --mysql-password PASSWORD --description "MySQL Databases" vol-2345bcde
This even works for the root volume (in my case an ext3, EBS file system) – and appears to have no adverse effects on the system (despite passing the –freeze-filesystem parameter).
To set this up as a cron script:
Create a bash script (e.g. /root/backup.sh) with the desired commands (don’t forget to use the full paths), make it executable, and run crontab.
#!/bin/sh /opt/aws/bin/ec2-consistent-snapshot --aws-access-key-id MYACCESSKEY --aws-secret-access-key MYSECRETKEY --region us-east-1 --freeze-filesystem / --description "Root volume: $(date +%c)" vol-3456cdef
Add a line to run the script daily:
37 2 * * * /root/backup.sh > /dev/null 2>&1
In this case, I have added the date to the description (not really necessary, but I felt like including it). The cron job is set to run at 2:37am – I like choosing odd times when nothing else is running and server load is low – and will discard output (in this case the snapshot IDs).
The next step, of course, would be to setup some form of rotation to delete the old snapshots when they accumulate. The setup of such a script is detailed in the article Rotating EBS snapshots: ec2-prune-snapshots