11

There are several folders on my system (Monterey 12.2.1) on which I can successfully run ls, but if I run ls -l on the same folder I get "Operation not permitted".

I am running these commands as root, and Terminal has full disk access.

I found this because timemachine can't back up these files, and I have no explanation as to why this would happen.

Here's an example:

 # ls /private/var/folders/44/26mm0l190mn7sf9xj5gmn3wh0000gp/0/com.apple.routined
dv
# ls -l /private/var/folders/44/26mm0l190mn7sf9xj5gmn3wh0000gp/0/com.apple.routined
ls: dv: Operation not permitted
total 0

Permissions and extended attributes on the parent directory seem normal:

# ls -ld@ /private/var/folders/44/26mm0l190mn7sf9xj5gmn3wh0000gp/0/com.apple.routined
drwxr-xr-x  3 aaroni  staff  96 Jun 24  2020 /private/var/folders/44/26mm0l190mn7sf9xj5gmn3wh0000gp/0/com.apple.routined

Any ideas?

AaronI
  • 221
  • 4
    You don't need to back up those folders. /var is transitory data. – Marc Wilson Feb 26 '22 at 19:00
  • 1
    @MarcWilson, /var is nominally transitory. In practice, a lot of the data is pretty long-lived, and some of it (such as log files) can be useful to back up. – Mark Feb 27 '22 at 21:45

2 Answers2

12

I can successfully run ls, but if I run ls -l on the same folder I get "Operation not permitted".

Difference between ls and ls -l

ls displays the name of the files/directories contained in a directory, while ls -l displays a lot of more information (excerpt from man ls):

If the -l option is given, the following information is displayed for each
file: file mode, number of links, owner name, group name, number of bytes
in the file, abbreviated month, day-of-month file was last modified, hour
file last modified, minute file last modified, and the pathname.

Usually, read access to the parent directory should be enough to get this information, but in this particular case (SIP enabled and datavault flag set, see below), it's not.

For example, to get the number of links, you need to know the inode number. You can get that information with ls -i, and you will see it fails for a number of files and directories (and consequently ls -l will fail for those files/directories, too):

ls -i  /private/var/folders/51/y66nwk4x1b700xgy1fs5d_9w0000gq/0/

ls: com.apple.LaunchServices.dv: Operation not permitted
ls: com.apple.SharedWebCredentials: Operation not permitted
ls: com.apple.lockoutagent: Operation not permitted
ls: com.apple.nsurlsessiond: Operation not permitted
ls: dmd: Operation not permitted
18693497 com.apple.ScreenTimeAgent  18701711 com.apple.corespeechd      18688786 com.apple.icloud.searchpartyd  18687284 com.apple.progressd
18693950 com.apple.Spotlight        18687828 com.apple.dmd          18687147 com.apple.notificationcenter   18691540 com.apple.routined
18688254 com.apple.bird         18702008 com.apple.dock.launchpad   11150084 com.apple.pluginkit

On the other hand, ls simply displays filenames, and that information can be retrieved for all files.

Why doesn't sudo ls -l work?

sudo let's you run commands as root:

$ sudo whoami
root

but root is no longer an almighty user: since OS X 10.11 "El Capitan", its powers are restricted by System Integrity Protection (SIP):

System Integrity Protection restricts the root user account and limits the actions that the root user can perform on protected parts of the Mac operating system.

Before System Integrity Protection, the root user had no permission restrictions, so it could access any system folder or app on your Mac. Software obtained root-level access when you entered your administrator name and password to install the software. That allowed the software to modify or overwrite any system file or app.

System Integrity Protection includes protection for these parts of the system:

  • (...)
  • /var

so /var is one of the protected folders that root has no longer unrestricted access to.

But there's more going on: disabling SIP makes these files accessible to root and to the owner. All files that couldn't be read have the extended attribute com.apple.rootless set, which marks them as being protected by SIP, but they also have the datavault flag enabled, for example:

ls -ledD@ com.apple.LaunchServices.dv

drwx------@ 4 jaume  staff  datavault 128 Jan 17 10:35 com.apple.LaunchServices.dv
    com.apple.rootless   27

which makes them unreadable except for apps with the corresponding entitlement (when SIP is disabled, access is again governed by mode and ownership, as in a traditional UNIX-like file system).

I found this because timemachine can't back up these files

/private/var/folders stores transient data and its contents are not backed up by Time Machine.

jaume
  • 15,010
  • 3
    Fair enough but this answer does not explain why root cannot access these. Normal UNIX permissions convention is that if you have r and x on a directory, you can get all dentry data. So how come OP can get some, but not others? And how come root is permission restricted anyway? – abligh Feb 27 '22 at 06:52
  • @abligh Good point, there's a lot going on behind the scenes, with SIP and the datavault flag restricting access, even for root. When SIP is disabled, access works according to file mode and ownership. As for why the OP can get information on some files, but not others: from checking the source code of ls (https://github.com/apple-oss-distributions/file_cmds/tree/file_cmds-321.100.11), it seemed that fts_read couldn't get the inodes of some files, but I can't give a conclusive answer (dtruss didn't help much). I've expanded my answer with more information on root access and SIP. – jaume Feb 27 '22 at 18:48
  • 1
    "To access some of this information, you need read access to that file or directory." None of that information requires read access. You just need to be able to call stat() on the file. – Barmar Feb 28 '22 at 03:56
  • @Barmar I agree, stat() doesn't require read access to the file, and yet, when the datavault flag is set and SIP is enabled, stat() (and for that matter stat) fail, while fts_read() seems to be able to retrieve the filename. I've rephrased the answer to make it clear that I refer to this particular case, thanks for pointing it out. – jaume Feb 28 '22 at 06:45
  • 1
    Thank you for this incredibly detailed answer! I'm not sure why timemachine is attempting to back up /private/var (and various other folders containing unreadable files), but this gives me a good starting point. – AaronI Feb 28 '22 at 08:40
6

The "dv" you're seeing here is, as I recall, "datavault". Apple has in general been moving away from standard Unix conventions to a more "rootless" system based on entitlements instead of user permissions. My guess is that the files you're looking at belong to routined (a daemon which tries to figure out things you do often) and you would need an entitlement in order to access them. There are a number of other directories protected in this way, for example everything in /System as well as e.g. iMessage database files.

computergorl
  • 61