General Support - new full featured rdiff-backup script, rd-h2.sh

h2 - 04.10.2006, 01:51 Uhr
Titel: new full featured rdiff-backup script, rd-h2.sh
After months of figuring out how to make my own backup script useable for most people, I think it's now ready to go.

I do not plan on updating this script at all unless someone finds a bug in it, so this one should not require any updates.

The script home page is located here

Or you can just download it here [bzipped file containing one directory which holds 4 files]

The script is a tar/bzipped directory called 'bu', which contains 4 files;

  1. rd-h2.sh, the main script
  2. root-excludes.txt, the excludes file for your root directory. You shouldn' need to change this unless you have more stuff mounted that you want to exclude, like for example /data. The default is to not backup your debian apt archived debs, so if you want to change that, just delete that item.
  3. home-excludes, this is a blank file, but it needs to be there or the script /home backup component will give errors. Add anything to that exclude list you don't want backed up that is located in /home
  4. a full readme-rd-h2.txt readme file.
    This is a complete howto text, including directions on what user changeable variables you need to set before you run the script. Please do not ask me questions that the readme answers, that's why I wrote it!

To download directly, say into /home/username/bin [creating a user bin directory is a good place to put stuff like this], do, as regular, not root, user:

cd ~/bin
wget techpatterns.com/bu
tar xjvf bu.tar.bz2

Then edit the primary user variables at the top of the script. The readme covers all that, and also explains what the script does. But to make it short, it selects one of two primary backup directories based on the current month of the year, then either deletes it, if you pick that option, or just runs the backup regularly.

The way rdiff-backup works is that first it creates a baseline backup, then every time you run it again to that primary backup directory, it just backs up filies that have changed since the last backup. And you can always restore to older versions of the file, unlike with for example cp -a or partimage type backups, for example.

Ok, this one is now done, and hopefully won't require too many changes.

Script features 3 options, 2 of which, -b and -d, will either do automatic backup, non-interactive, or -d, delete old then non-interactive backup.

-s runs the script with a little spinning wheel activity display, do not try to kill the script when you use the -s option since that won't kill the background backup or delete processes.

The script defaults to just backing up your /home and / as separate backups, but supports running anything from only 1 to 4 unique backups. I like backing up /home and / separately, as well as various other data that I don't like mixing up with the other backups, so I use 4 on my main box, 2 [ /home and /] on secondardy boxes

that's it, hope it's useful to some of you. The only thing most users should need to change is a few variable paths and names in the top user variable section, then when you run it, it will just ask you if you want to do a normal backup or delete plus backup.
kenyee - 04.10.2006, 02:25 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
Out of curiosity, why do you have two backup archives?
Have you seen a case where one archive croaks? Auf den Arm nehmen
h2 - 04.10.2006, 02:35 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
Good question, lol, I figured someone would ask or wonder that.

Because when I delete one to start over again, that leaves me with the other one. Since I actually have to use backups for work purposes etc, this is very convenient for me. Doing it automatically based on current month is something I came up with because I kept chosing the wrong one, so I figured I'd better remove the possibility for user error.

Imagine this not all that unlikely scenario, for example: you delete the old backup, you begin the new one. your system hard drive, which was about to fail, fails because of the heavy disk access activity. Now you have no backup.

Since I sometimes create backup solutions for clients, I've learned that having double or triple layers of backup protection is never a bad idea. That was confirmed to me once when a mirrored raid array died, one of our backups for that failed, and the third one worked. After that I never questioned the need for the maximum backup protections I could provide.

In terms of how this script runs, it will alternate every 3 months, which means that essentially you'll have after using it for a while at least a minimum of 3 months of backups, upto 6 months, or more if you chose, to roll back to.

Using only one creates increasingly massive backups, which at some point you have to delete. From my experience, I've never needed a file older than 6 months.

As to the why, really it's just because it seemed like a good idea to me, and since I had to develop this script based on my own needs, then expand it to make it more user friendly and flexible, I had to have that feature built in.

For users who don't want that feature, they can simply assign the value 1 to every month grouping, then you'll only have 1 backup primary directory. Other users might want 4, it all depends.

What I see happening for at least myself is that every time the script rolls over to a new primary directory, my previous 3 months sit untouched, just in case, then when it rolls over again, I will probably delete the old stuff, and start fresh. Starting fresh with rdiff-backup takes A LOT longer than running weekly backups, by the way. Weekly backups can take only a few minutes after you set the initial one.

I've been running various encarnations of this script for my own uses for about 6 months, and this is what I finally ended up with as being the most flexible and error free setup I could get while still keeping it reasonably simple.

Ideally I'll also start using it commercially, so I built it to be far more powerful than many users might need, but that's just a bonus I'd say. That's for example why it supports the non-interactive mode options of -b and -d, for cron job run backups, or for single command backup, start and do something else. Commercially, I can easily see running a total of 4, which will give me 12 months of full backups. I'll see where that goes though in the future.

Speaking of backup safety, keep in mind this: if you backup to the same hard disk you want to restore to and your disk dies, that isn't going to do you much good. So at least backup to a 2nd internal, but ideally to a removable external. Or a removable hard disk tray, which is the fastest you can do, SATA ideally for hot swapping.
2radical - 04.10.2006, 02:59 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
h2: Thank you very much!
h2 - 04.10.2006, 03:04 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
2radical, this was done with you in mind by the way, your last problems is what finally prompted me to get the public release of this finished, well, that and the fact that the 3 month cycle just rolled over so I needed to get that stuff running for testing by 9-30, then I waited to clean up more issues and fix some stuff before releasing what I really hope is the only version of this, lol...

You'll find that this script is very easy to run and setup.
gs - 04.10.2006, 11:31 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
@h2: just tried your rdiff-backup-script, using the option -s.
There were a lot of messages in Konsole such as

/media/hdb14/bu-1/home/gskan2006hdb10/rdiff-backup.tmp.4 does not match source
SpecialFileError gskan2006hdb10/.DCOPserver_GuentersBox_:1 [Errno 1] Operation not permitted

What does this mean and does it matter? As a matter of fact a file bu-1 was createt in the partition I choose. It had nearly (but no totally) as much GB as the files to be backed up. I could look at /home within bu-1 and it seems to contain all folders of my original home, but with some strange spelling.....
And while I am at it: How would I go about restoring the backup?
Thanks for useful scripts which make KANOTIX even more user-friendly
h2 - 04.10.2006, 19:34 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
I believe this rdiff-backup wiki item covers your question:

http://rdiff-backup.solutionsfirst.com. ... teErrorOne

translation, some file in your /home directory changed between the time rdiff-backup looked at it and the time rdiff-backup tried to back it up.

As always, don't work on your machine while backup runs, at least not while your user changeable data backups run. Common culprits: email clients automatically downloading emails. Shutdown email clients, and any other application that might create new or changed files in /home while rdiff-backup runs. I suspect also streaming media might create this problem.
h2 - 04.10.2006, 21:33 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
<updated>I added a timer function to the script, so now it will output the total time for the backup after it's completed. Version 1.3.0 has this feature.

Modified some stuff with the -s spinning wheel indicator function to improve display output.

This should be about it unless a bug comes up.

If you got an earlier version, I recommend you get the latest one, it should work slightly better.

Hopefully this will be about it for changes, but you never know.
kenyee - 04.10.2006, 23:01 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
Another good reason to use LVM...it supports snapshots so you don't do rdiff-backup on a changing system Smilie
FYI, here's a script that handles the snapshot:
gs - 04.10.2006, 23:12 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
@h2: culprit found - was listening to music via DVB-S while your script was running.....
h2 - 04.10.2006, 23:28 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
That's what I thought it was.

If you have a version less than 1.3.4, I recommend you get the latest version, and I'll try to not do many more changes unless absolutely necessary.

I've cleaned up mostly output stuff, progress wheel [ using -s option that is ] works more reliably now, outputs completed messages too, etc, that's about it for changes.

After today hopefully I won't need to do many more changes to the script.
h2 - 06.10.2006, 05:19 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
If you downloaded this already, you need to add one line to your: roots-excludes.txt file

+ /var/cache/apt/archives/partial

Just add that line to the end of the file and save it, that will prevent an apt error on full root restore.
h2 - 08.10.2006, 07:03 Uhr
Titel: RE: new full featured rdiff-backup script, rd-h2.sh
<small fix>
Since I just had the opportunity to test rdiff-backup restore, I found one more small glitch with the root-excludes.txt file:

please change it to this, plus of course anything you've changed to it:

+ /var/cache/apt/archives/partial

the order of the last three are important, I had them backwards, the include + item has to come before the exclude lower parent item.

All this does is save the empty 'partial' directory, which prevents an apt error when you run apt-get after doing a full restore.

This is not a big deal since you can fix the error by just creating the /var/cache/apt/archives/partial directory, but it is a glitch, so might as well have it right in the first place.

No further changes to the main script I'm happy to say.
Alle Zeiten sind GMT + 1 Stunde
PNphpBB2 © 2003-2007