The Little S3 Backup Script That Could
After having an external enclosure fail (grrr LaCie!) and realizing that all of my backup data was spread over about 6 hard drives, I decided to check out Amazon’s Simple Storage Service (S3) to back up all of my important files to one central location. A friend recommended Jungle Disk because it was “free”, but since it wasn’t, I decided to play with it on my own. Given my recent love for Bash scripting, I decided to start that way.
My first task was to make a back up of all my pictures, which was a set of about 100 folders following the format “YYYY-mm-dd name of event”. Since S3 just gives you a table interface, I decided to archive each folder in a tar and then upload the tar. It seemed unlikely that I’d want only one picture from a set, so this also made sense. The caveats were that I needed to change spaces to underscores in the .tar filename and that I only wanted to keep the .tar around long enough to upload it. I found Google’s s3-bash to be the most straightforward utility for copying files to S3 from my shell.
#!/bin/sh
ls -d /media/gadget/pictures/* | while read -r FILE; do
# transpose spaces to underscores
tarfile=`echo $FILE | tr ' ' '_' `.tar
tar cvf $tarfile "$FILE"
/opt/s3-bash/s3-put -k #MY_KEY# -s /opt/s3-bash/.secret \
-T $tarfile /derwiki-media/`basename $tarfile`
rm $tarfile
done
