Beginner to Beginner: rsync exclude-from
Oh, I am so about to make a fool of myself in public…
I now have a D-Link DNS-323 plugged into my home network. It’s a network storage device that I want to use as a centralized backup for my family’s various computers because some of us don’t always plug our Macs into our USB external hard drive to let the Mac Time Machine work its backup magic. Unfortunately, the hack I found on the Net to get Time Machine to recognize the DNS-323 doesn’t work for me: Time Machine lets me say I want the backup to be housed on the DNS-323, but the software craps out when it actually tries to back up to it. If there’s an easy way around that, I’d love to hear about it.
In the interim, I’ve been playing with rsync, a command-line utility included in Leopard that does backups. I’ve had no luck with rsyncX, which is a Mac specific version, but rsync is working. It took some doing to get it running on the DNS-323, including installing fun plug (the DNS-323 is a linux box) and writing a config file that specifies which machines rsync recognizes. My Linux hacker nephew Greg did that part of it for me. (Thanks, Greg.)
There’s a script that enables rsync to mimic Time Machine. It’s been working pretty well — my hourly backups go far slower than they should, so I’m undoubtedly doing something wrong — but I had a heck of a time telling it which directories I want it to back up. You gain control over the backup set by specifying a file of inclusions and exclusions. You do this in the rsync command line by saying “–exclude-from filename” where you replace “filename” with the name of the file that has the list.
After a bunch of Internet research and way too much trial and error, I now have a list that does what I want, although I’m sure it’s laughably kludgy, and possibly fatally wrong. Nevertheless, here’s how I think it works…
The file can list both includes and excludes. You indicate which is which by prefacing each item with a + or a -. The list assumes that the root directory is whichever one you specified in the rsync command line. So, if your command line said that you want to back up “/Users/me/”, then you would tell it to exclude “/Users/me/junk” by putting the following line in your exclude-from file:
- junk/
Likewise, to include /Users/me/importantstuff/ you’d put in the line:
+ importantstuff/
But, at least in my experiments, that line will not include any subdirectories of importantstuff. After failing to understand the instructions I found on the Net, and after a lot of trial and error, I’ve found that it works if I also include the line:
+ importantstuff/**
The double stars tell it to backup all the subdirectories and all their subdirectories, ad infinitum. I’ve found I have to put in both the line without the stars and then the line with the stars. You’d think the line with the stars would be enough, but in my tries and my errors, it wasn’t.
The list of inclusions and exclusions is sensitive to the order of the list. If you have particular subdirectories you want to exclude (e.g., importantstuff/junk/), put them first:
- importantstuff/junk/**
If you want rsync to backup only designated directories, list your excludes first, then your includes, and end with
- *
which tells it to exclude anything you didn’t already tell it to include. I have the feeling that that may be an ugly hack with unintended consequences. Remember, I don’t know what I’m doing.
So, my exclude-from file looks roughly like this:
- *Azureus*/
- *Azureus*/**
- Documents/TiVo*
- Documents/Aptana*
+ Sites/
+ Sites/**
+ Pictures/
+ Pictures/**
+ Music/
+ Music/**
+ Documents
+ Documents/**
- *
Two important notes: 1. The -n parameter on the command line will run rsync in “what if” mode, showing you what it would do without actually doing it. 2. As I’ve likely made some embarrassing and awful mistakes, please read the comments in hopes that some knowledgeable and kind soul will correct me.
Categories: tech
oh my god, none of your readers know how to use rsync
including me
And I think “rsync” is where we wash our hands.
jeux casino en ligne gratuites…
A casino online gratis casinos pagina web jugar gratis pagina web poker torneos gratis juego poker erotico…
“…”rsync” is where we wash our hands….”
Thanks for that. It’s the first time rsync has given me something to smile about.
I’m struggling with the “exclude-from” option myself.
Joho,
This helped me.
Thanks.
Agreed, rsync’s exclude-from file is very confusing. I hoped it would do regular expressions but no…
This helped me too. I’m a Software Developer :)
Thanks
Thanks a lot! I can’t believe that the man page doesn’t include anything on how to format exclude files
The manual page does include information about exclude files under “FILTER RULES”. Man pages shouldn’t be read in haste. If you don’t read the ENTIRE man page, you will miss numerous references to some very clearly stated information.
[snip src="rsync(1)" ]
–exclude=PATTERN
This option is a simplified form of the –filter option that defaults to an exclude rule and does not allow the full rule-parsing syntax of normal filter rules.See the FILTER RULES section for detailed information on this option.
…..
FILTER RULES
The filter rules allow for flexible selection of which files to trans-
fer (include) and which files to skip (exclude). The rules either
directly specify include/exclude patterns or they specify a way to
acquire more include/exclude patterns (e.g. to read them from a file).
As the list of files/directories to transfer is built, rsync checks
each name to be transferred against the list of include/exclude pat-
terns in turn, and the first matching pattern is acted on: if it is an
exclude pattern, then that file is skipped; if it is an include pattern
then that filename is not skipped; if no matching pattern is found,
then the filename is not skipped.
Rsync builds an ordered list of filter rules as specified on the com-
mand-line. Filter rules have the following syntax:
RULE [PATTERN_OR_FILENAME]
RULE,MODIFIERS [PATTERN_OR_FILENAME]
You have your choice of using either short or long RULE names, as
described below. If you use a short-named rule, the ‘,’ separating the
RULE from the MODIFIERS is optional. The PATTERN or FILENAME that fol-
lows (when present) must come after either a single space or an under-
score (_). Here are the available rule prefixes:
exclude, – specifies an exclude pattern.
include, + specifies an include pattern.
merge, . specifies a merge-file to read for more rules.
dir-merge, : specifies a per-directory merge-file.
hide, H specifies a pattern for hiding files from the transfer.
show, S files that match the pattern are not hidden.
protect, P specifies a pattern for protecting files from dele-
tion.
risk, R files that match the pattern are not protected.
clear, ! clears the current include/exclude list (takes no arg)
When rules are being read from a file, empty lines are ignored, as are
comment lines that start with a “#”.
Note that the –include/–exclude command-line options do not allow the
full range of rule parsing as described above — they only allow the
specification of include/exclude patterns plus a “!” token to clear the
list (and the normal comment parsing when rules are read from a file).
If a pattern does not begin with “- ” (dash, space) or “+ ” (plus,
space), then the rule will be interpreted as if “+ ” (for an include
option) or “- ” (for an exclude option) were prefixed to the string. A
–filter option, on the other hand, must always contain either a short
or long rule name at the start of the rule.
Note also that the –filter, –include, and –exclude options take one
rule/pattern each. To add multiple ones, you can repeat the options on
the command-line, use the merge-file syntax of the –filter option, or
the –include-from/–exclude-from options.
[/snip]