Update: the complementary article to this one, on Rotating EBS snapshots has now been posted.
After the recent failure of EBS volumes on Amazon’s cloud, backup is quite likely something to be considering. If, during the aforementioned incident, recent snapshots of the affected EBS volumes were available, it might have been possible to relaunch instances and recreate new volumes in unaffected zones and minimize downtime.
The ec2-consistent-snapshot script is designed to minimize data loss by freezing the file system and flushing MySQL tables. However, it is released for Ubuntu, and some pre-requisites are necessary in order for it to run on Amazon’s Linux distribution (CentOS/RHEL derived).
A simplified version of the steps below worked well on the latest version of Amazon’s Linux (2011.09), since it includes the EPEL repository by default. (Note: priorities are enabled by default on Amazon’s Linux – which means most of the perl modules below will come from the amzn repository).
yum --enablerepo=epel install perl-Net-Amazon-EC2 perl-File-Slurp perl-DBI perl-DBD-MySQL perl-Net-SSLeay perl-IO-Socket-SSL perl-Time-HiRes perl-Params-Validate ca-certificates wget -O /opt/aws/bin/ec2-consistent-snapshot http://bazaar.launchpad.net/~alestic/ec2-consistent-snapshot/trunk/download/head:/ec2consistentsnapsho-20090928015038-9m9x0fc4yoy54g4j-1/ec2-consistent-snapshot chmod 0775 /opt/aws/bin/ec2-consistent-snapshot
The procedure below is obsoleted by that above, but retained for interest sake.
The following steps were able to get ec2-consistent-snapshot operational, with a minimum of hassle (adapted from: http://pojo-mojo.blogspot.com/2011/03/ec2-consistent-snapshot-for-centos.html).
yum install perl-Params-Validate PERL_MM_USE_DEFAULT=1 cpan Net::Amazon::EC2 yum install perl-File-Slurp perl-DBI perl-DBD-MySQL perl-Net-SSLeay perl-IO-Socket-SSL perl-Time-HiRes cpan Mozilla::CA cd /opt/aws/bin wget http://bazaar.launchpad.net/~alestic/ec2-consistent-snapshot/trunk/download/head:/ec2consistentsnapsho-20090928015038-9m9x0fc4yoy54g4j-1/ec2-consistent-snapshot chmod 0775 ec2-consistent-snapshot
A few notes about the above items:
- Params::Validate is needed by Net::Amazon:EC2 (or one of its dependencies), however, CPAN failed to install it for me. Yum did the job without issue.
- The CPAN installation of Net::Amazon::EC2 takes a long time – about 30 min – and has numerous dependencies. The ‘PERL_MM_USE_DEFAULT=1’ setting will let CPAN choose the defaults (essentially saying yes to all dependencies). This probably won’t work if you haven’t setup CPAN previously.
- There is a good chance many servers already have some of the packages yum is asked to install – however, yum will simply ignore those.
- I have opted to place the script in the same directory as the AWS utilities (/opt/aws/bin).
You will need to specify the mount point for ec2-consistent-snapshot, and might want to know the type of each file system – both can be found by running:
If you are going to create a backup of MySQL data, and wish to take advantage of ec2-consistent-snapshot’s features, you will need to create a user with
REPLICATION CLIENT, and
SUPER privileges, similar to the following:
GRANT RELOAD, REPLICATION CLIENT, SUPER ON *.* TO 'snapshotter'@'localhost' IDENTIFIED BY 'PASSWORD'
Since the ec2-consistent-snapshot command requires your access key and secret key, and will copy the same into the snapshot (which is stored in S3), it may be preferable, from a security standpoint, to create a user (with limited privileges that can create snapshots) via AWS Identity and Access Management (IAM) specially for this task.
As an example of a non-database volume:
ec2-consistent-snapshot --aws-access-key-id MYACCESSKEY --aws-secret-access-key MYSECRETKEY --region us-east-1 --freeze-filesystem /var/www --description "Website Files" vol-1234abcd
As an example of a MySQL database volume:
ec2-consistent-snapshot --aws-access-key-id MYACCESSKEY --aws-secret-access-key MYSECRETKEY --region us-east-1 --freeze-filesystem /data --mysql --mysql-host localhost --mysql-socket /data/var/lib/mysql/mysql.sock --mysql-username snapshotter --mysql-password PASSWORD --description "MySQL Databases" vol-2345bcde
This even works for the root volume (in my case an ext3, EBS file system) – and appears to have no adverse effects on the system (despite passing the –freeze-filesystem parameter).
To set this up as a cron script:
Create a bash script (e.g. /root/backup.sh) with the desired commands (don’t forget to use the full paths), make it executable, and run crontab.
#!/bin/sh /opt/aws/bin/ec2-consistent-snapshot --aws-access-key-id MYACCESSKEY --aws-secret-access-key MYSECRETKEY --region us-east-1 --freeze-filesystem / --description "Root volume: $(date +%c)" vol-3456cdef
Add a line to run the script daily:
37 2 * * * /root/backup.sh > /dev/null 2>&1
In this case, I have added the date to the description (not really necessary, but I felt like including it). The cron job is set to run at 2:37am – I like choosing odd times when nothing else is running and server load is low – and will discard output (in this case the snapshot IDs).
The next step, of course, would be to setup some form of rotation to delete the old snapshots when they accumulate. The setup of such a script is detailed in the article Rotating EBS snapshots: ec2-prune-snapshots
Thanks for all the great information in this post, I’ve used it to get daily backups going on our EC2 web and MySQL servers.
Two other important things I learned in the process:
1) The MySQL user for the MySQL server snapshot has to have Reload, Super, and Replication Client privileges. I learned this by first running the backup script with a user that had no server privileges; the error messages called out which privileges were required. I use a MySQL user that has only these privileges, for the snapshot process.
2) I added the following line at the top of my crontab:
This is because fsfreeze is not in the default path of the cron process (which is /usr/bin:/bin). Because of this, the fsfreeze command was failing, but the script continues on, even without a frozen filesystem. I only found this out by setting the MAILTO= variable in the crontab, and not using the > /dev/null redirect, so it sent me any output and errors.
Thanks for the comment (and good to know the information was useful to someone). With regards to point #1: the required MySQL privileges are actually noted in the post (under ‘Useful Info’) – I didn’t realize that MySQL would display the specific privileges required, so that is definitely good to know for future reference. For point #2 – thanks for the pointer, its handy to know (It didn’t happen on my setup – I use xfs_freeze, but that is also in /sbin). I have more than one EBS volume that I backup, so the commands are all in a script that cron executes. Since the script initializes a bash environment, the additional PATH variables do not need to be explicitly specified.
Ah, of course, I had actually used your ‘GRANT RELOAD …’ statement to create a MySQL user, but later deleted the user for some reason. When the script wouldn’t run, it was actually the ec2-consistent-snapshot error messages that, in several different messages, told me what privileges were needed.
Thanks again for all the great information.