Which files should be excluded in an rsync backup (with macOS Big Sur / version 11.5.1)?

Question

Context:

I've been backing up my MacBook Pro with rysnc (now version 3.2.3, protocol version 31) to external hard drives formatted with NTFS. I'd like to continue with this procedure if possible.

Usually, for a given macOS version, I would use the StdExclusions file located at /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions.plist
as a starting point to decide what files/folders to exclude.

In Big Sur, this StdExclusions file doesn't seem to exist (at least not in its usual location).

Sub-questions:

In Big Sur, is there a new StdExclusions file or equivalent in another location?
Given the new file system (APFS with firmlinks/wormholes and things that are still mysterious to me) should I exclude /System/Volumes/Data/Users and copy /Users or the other way around, or do I need to copy both?
More generally, (and more importantly and to the point than sub-question 2) how should one approach the /System/Volumes/Data directory in context with the whole system when backing up?

Context detail:

Initially when I was using rsync version 2.6.9 protocol version 29, I didn't need to include the --ignore-errors flag but with rsync 3 I found I had to add that flag, so my fundamental rsync command takes this form:

copyall() {
  rsync -aAXPvh --modify-window=1 --partial-dir=../rsync_TMP_PARTIAL --delete-after --ignore-errors "$@"
}
BDL="--backup-dir=../DeletedFiles_$(date +\%Y-\%m-\%d)"
EXC="--exclude-from=/Location/of/my/modified/custom/exclusions/file"
copyall "${EXC}" --backup "${BDL}" "${SRC_BASE}/" "${DST_BASE}"

@Gilby, for recovery, I'd be okay with having only a copy of my unique material (and config files saved that make re-installations easy); if trouble happens, I could re-install the OS if necessary and then copy over material from my backup drive and do any remaining re-installs. I believe I chose NTFS for flexibility in being able to access my backup drives from a variety of operating systems. A few years ago I bought software (Paragon NTFS for Mac 15) that allows me to use NTFS. On their website, looks like it supports Big Sur. The question about /System/Volumes/Data/Users is important to me. — zeroparallax, Aug 06 '21 at 23:20
One of my mistakes: looks like the -E option changed to -AX (ACL & extended attributes) between rsync version 2 and 3, so rsync -aEPvh should instead be rsync -aAXPvh. And it looks like Paragon NTFS does support extended attributes. Does this cover all APFS metadata? I don't know. — zeroparallax, Aug 07 '21 at 01:58
You only need to backup /System/Volumes/Data, the OS itself can‘t be restored anyway and is easily reinstalled from Recovery. — nohillside, Aug 07 '21 at 05:23
@nohillside, thanks. Could you explain how you came to this knowledge? And is there any official documentation that would confirm your knowledge? If I take your advice, it seems that the old StdExclusions file could be applied to this directory; so /.Spotlight-V100 would become /System/Volumes/Data/.Spotlight-V100 and similar for /.fseventsd, etc. I wonder though if there are new files I should be excluding particular to Big Sur. That is, what should be the standard exclusions for Big Sur? — zeroparallax, Aug 07 '21 at 05:42
You can check if a file or directory is excluded by Time Machine by using tmutil isexcluded so, for example tmutil isexcluded ~/Library/Caches/ yields [Excluded] /System/Volumes/Data/Users/michaelr/Library/Caches — MichaelR, Aug 07 '21 at 06:12
@MichaelRourke, thanks. You helped me discover what I think might be a precise answer to my sub-question 1, which was answered in this other apple stack-exchange post. What confuses me is that, when I perform that meta-data query, I get a list of files that only has three files outside of my /Users/<username> folder, two of those being outside the /Users folder. I expected to more of the files listed in the old StdExclusions file. I don't (yet) see a single same filename. — zeroparallax, Aug 07 '21 at 07:45
According to various sources (including this one, Time Machine will add a .exclusions.plist file where a Time Machine backup is performed. It would be nice if someone could post an example of that. It seems that a full answer should refer to that as well as an mdfind meta-data query (as mentioned and linked above) and a reference to the contents of /Library/Preferences/com.apple.TimeMachine.plist, as described here. Still may not be exhaustive. — zeroparallax, Aug 10 '21 at 23:14

score 1 · Answer 1 · answered Aug 07 '21 at 00:02

1

Using rsync to copy files to an NTFS drive will not provide a seamless recovery.

File and Folder metadata will be lost along with aliases, symlinks, etc.
Recovering applications, settings and state information will be largely lost and would require careful copying of files stored in /Library and ~/Library.
Any use of iCloud would need to be reestablished.
I am doubtful that you would able to recover the libraries used by the Photos and Music libraries to the state they were before disaster.

The best you will be able to do is to recover files in folders such as ~/Documents.

The standard exclusions used by Time Machine as that would leave you rsync-ing lots of files that would not provide recovery to the original state of the Mac.

Rather that looking for exclusions, I suggest that you need to first look at inclusions and determine what, potentially, can be recovered.

If using rsync you should limit your expectations to recovery of most data files, but without much of the state information stored in file metadata and /Library and ~/Library.

Regarding your second sub-question: /System/Volumes/Data/Users and /Users are the same thing.

Strong recommendation:

For a seamless disaster recovery, you need to use a backup program which is aware of macOS data structures. That would be Time Machine or third-party products like Carbon Copy Cloner or Chronosync.

answered Aug 07 '21 at 00:02

Gilby

10,852

I appreciate your pointers and recommendations; in the long-run I'll consider options you point to. In the short-run, since I don't require seemless recovery, I'll continue with rsync. If I include the options I mentioned in another comment on the main post (in particular -aAX, which includes extended attributes), does that get all the essential APFS metadata? I think symlinks are copying fine, and I don't think I need aliases. I like your emphasis on inclusions rather than exclusions, because I'll want to minimize what I'm copying. I guess I'll exclude /System/Volumes/Data/Users. – zeroparallax Aug 07 '21 at 02:08
@zeroparallax I am sure you know more, form the practical point of view, about rsync than I do. But certainly, including extended attributes will get the essential metadata at the file level. You may want to consider whether that will maintain file creation and modification dates (if that is important to you). But it really is about testing again and again until you are confident of good enough recovery. – Gilby Aug 07 '21 at 02:57
I appreciate your answer and help, but I'd prefer to mark as "accepted" an answer that is more definitive about solving the "exclusions" issue for a general backup, in particular with reference to what the StdExclusions.plist equivalent might be for Big Sur, and also how one should approach the /System/Volumes/Data/ directory. Although I didn't highlight that in my question, I'm still confused about that. (Regarding your helpful focus on "inclusions", perhaps that could be another question about what is necessary in a minimalist backup.) – zeroparallax Aug 07 '21 at 03:46
I have no problem with you waiting for an answer more focused on your precise need. /System/Volumes/Data is the read-write part of the macOS boot disk. So '/System/Volumes/Data/Usersis the same as/Userssince all of/Usersis on the read-write part of boot disk. But/System/Volumes/Applicationsjust has apps not part of macOS and so is different to/Applications` which includes both read-only apps (like Mail) as well as added apps. – Gilby Aug 07 '21 at 05:56
I disagree. *Modern* versions of rsync are perfectly capable of "lossless" backup and restoration. This is easy enough to demonstrate; see this for example. – Seamus Aug 08 '22 at 07:05

nohillside · Answer 2 · 2021-08-07T11:46:35.593

The file system structure got rather elaborated with Big Sur (Big Sur boot volume layout gives a good overview). The positive part of it is that the content worth backuping up is all part of /System/Volumes/Data.

$ cd /System/Volumes/Data
$ ll
total 2
drwxrwxr-x  170 root  admin  5440 Aug  6 19:42 Applications/
drwxrwxr-x    5 root  admin   160 Dec 15  2020 Incompatible Software/
drwxr-xr-x   84 root  wheel  2688 Jul 26 20:30 Library/
drwxr-xr-x@   3 root  wheel    96 Jan  1  2020 System/
drwxr-xr-x   12 root  admin   384 Jan  1  2020 Users/
drwxr-xr-x   10 root  wheel   320 Aug  7 08:06 Volumes/
drwxr-xr-x    2 root  wheel    64 Oct 25  2020 cores/
dr-xr-xr-x    2 root  wheel     1 Aug  7 08:11 home/
drwxr-xr-x    2 root  wheel    64 Oct 25  2020 mnt/
drwxr-xr-x    5 root  wheel   160 Dec  3  2020 opt/
drwxr-xr-x    6 root  wheel   192 Jan  1  2020 private/
drwxr-xr-x    2 root  wheel    64 Oct 25  2020 sw/
drwxr-xr-x@   5 root  wheel   160 Jan  1  2020 usr/

(you may not have ./sw)

Looking at that list it seems save to skip

Incompatible Software/
Library/Caches/
Volumes/
cores/
home/
mnt/

in general.

On user level, the things which can be skipped are ~/Library/Caches and any VMs you have laying around. Whether on wants to skip ~/Downloads is more a matter of personal taste (TimeMachine skips as far as I remember).

PS: In general I would recommend against backup up macOS data on an NTFS formatted volume, there are too many things which can go wrong and you have to rely on 3rd party software to recover data afterwards). If NTFS is all you have, creating a sparsebundle DMG formatted with APFS and using this as a backup target might be better.

TM does not skip ~/Downloads. Will the OP be able to correctly (usably) recover much of ~/Library from an NTFS backup - I am thinking of things like ~/Library/Containers? — Gilby, Aug 07 '21 at 09:57

Which files should be excluded in an rsync backup (with macOS Big Sur / version 11.5.1)?

2 Answers2