Errors from whatis command. Unable to rebuild database with makewhatis?

Question

How can I update the whatis database?

$ sudo /usr/libexec/makewhatis
Password:
makewhatis: /usr/share/man/whatis.tmp: Read-only file system

I believe being able to update this database will solve some other issue I'm having. My path to discovery as follows...

I recently started noticing that fish shell completions were annoyingly slow on my machine, possibly shortly after upgrading to Catalina.

I did a little profiling with fish -d5 and noticed that the majority of the time was spent on the apropos command. I did some reading and learned that the tools apropos, whatis and makewhatis are all related. They index man pages and make them searchable. Fish shell is (correctly) using them to offer helpful completions.

When I run whatis or apropos standalone, I get the following output:

$ whatis man
hugo-gen-man(1)          - Generate man pages for the Hugo CLI
groff_man(7)             - groff `man' macros to support generation of man pages
groffer(1)               - display groff files and man~pages on X and tty
man(1)                   - format and display the on-line manual pages
man.conf(5)              - configuration data for man
zshall(1)                - the Z shell meta-man page
xml2man(1)               - MPGL to mdoc (man page) translator
makewhatis: /usr/lib/./libgutenprint.2.dylib: No such file or directory
makewhatis: /usr/lib/libsasl2.2.0.1.dylib: Not a directory
makewhatis: /usr/lib/libldap.dylib: Not a directory
makewhatis: /usr/lib/libsqlite3.0.dylib: Not a directory
makewhatis: /usr/lib/libcom_err.dylib: Not a directory
...

Followed by at least 100 more lines of the "Not a directory" messages. I believe it is all these useless lines that are slowing things down.

So I thought maybe I just need to rebuild the whatis database (perhaps after the Catalina upgrade?). However, it doesn't seem to work:

$ sudo /usr/libexec/makewhatis
Password:
makewhatis: /usr/share/man/whatis.tmp: Read-only file system

So this part is a little disturbing. How can I rebuild the whatis database? I have a hunch this will solve my issues if I can figure it out.

Answer 1

The following can be used as a workaround for the macOS 10.15.1 version of the apropos command, wherein it spews out complaints of the form makewhatis: /usr/lib/lib … .dylib: Not a directory.

First create the workaround script:

$ mkdir -p ~/workarounds
$ sed -e 66s@/usr/lib@@ /usr/bin/apropos > ~/workarounds/apropos.macos_10.15.1 
$ diff /usr/bin/apropos ~/workarounds/apropos.macos_10.15.1 
66c66
<     for d in /var/cache/man $manpath /usr/lib
---
>     for d in /var/cache/man $manpath 
$ chmod +x ~/workarounds/apropos.macos_10.15.1

Next add an alias to your shell to tell it to use the workaround script, until a newer version of the canonical script becomes available.

For Zsh you can use the following command:

$ /bin/cat <<END >> ~/.zshrc
# Workaround for broken apropos command.
alias apropos=~/workarounds/apropos.macos_10.15.1
END

For other shells such as ksh or bash, use ~/.profile or ~/.bash_profile, as appropriate.

What does the workaround do?

Apropos requests (and man -k requests) are handled by the /usr/bin/apropos script. That script searches for “whatis” database files in all the directories of the man path (see man —path), plus /var/cache/man and /usr/lib. The checks for /var/cache/man/whatis and /usr/lib/whatis appear to be there for historical reasons, however those files are not actively generated in Mojave or Catalina. A lot of different people have contributed to the various flavours of Unix over the years, and many of them had different good ideas about where to put different types of files. At some point in time, somebody decided that /usr/lib would be a good place to put a whatis file, and at some other point somebody else figured that /var/cache/man would be a good place. Others thought that the appropriate place would be the respective man page directories. Different solutions that seemed appropriate at the time. The apropos script has traditionally checked those locations in case a whatis file was present.

With the move to making the system directories on Catalina read-only (a good move), whatis database files can not be written to directories such as /usr/share/man. There are different ways that Apple could handle that, but for this release somebody decided to alter the apropos script by making it generate results on the fly by calling /usr/libexec/makewhatis.local for any man page directory that does not contain a whatis file.

That new apropos code works fine for actual man page directories, and for /var/cache/man (since it doesn’t exist), but it fails for /usr/lib. The workaround detailed above just eliminates /usr/lib from the list of directories searched.

As a final step, set yourself a reminder for a month or two from now to check whether Apple has fixed the apropos script. If so remove your workarounds, by removing the alias and the workaround script.

Answer 2

I just ran into this and googled my way here...

Looks like "whatis" will either grep through generated whatis files, or generate them on the fly to the stdout. What we're seeing is the output of "makewhatis" being run on /usr/lib.

You'll get the same errors from:

/usr/libexec/makewhatis -o /dev/fd/1 /usr/lib

/usr/lib isn't in the manpath (output of "man --path") - it's added explicitly by "whatis", though for what reason I have no idea. There are no man pages there, and makewhatis is clearly expecting everything in a man folder to be a subdirectory.

If we could edit the "whatis" script, we could fix it. But we can't, because /usr/bin is read-only.

If we could generate an empty /usr/lib/whatis, the complaints would stop. But we can't because /usr/lib is read-only.

It might be possible to fix /usr/libexec/makewhatis.local to stop this nonsense, but it's read-only.

I need to do some research to see if there's a way to get the OS volume mounted read-write for a bit.

On a related note: Even if we did get a "makewhatis" to run successfully, it won't generate /usr/lib/whatis, because /usr/lib not in the man path... so it won't fix this. Creating an empty /usr/lib/whatis is probably the easiest and safest option, if we can figure out how.

Answer 3

Regarding a Solution that would generate the missing whatis files:

The solution to updating the "whatis" database in /usr/share/man requires a fix from Apple. They need to either add /usr/share/man to their list of firm links (similar to the implementation for /usr/share/snmp), or add a static copy of the whatis file to the system volume.

Firm links are a new feature of the APFS; designed to support merging read-write volumes with read-only system volumes. As of the Catalina release the core operating system files are held on a read-only volume, which is then merged with a read-write data volume via the use of firm links. Under macOS version 10.15.1, /usr/share/man is only present on the read-only system volume. You can add an entry for /usr/share/man to the data volume by creating the directory /System/Volumes/Data/usr/share/man, as demonstrated by klanomath’s answer, but it will not be mapped onto the system directory (/usr/share/man) until a corresponding firm link is created.

A list of the current firmlinks can be found in /usr/share/firmlinks. The documentation that I have been able to dig up thus far is not clear as to whether firmlinks is a reference file, or a configuration file, but it looks to me like a configuration file that is read and utilized as part of the boot procedures. Assuming that it is a configuration file, then you could theoretically correct the issue by adding an entry for /usr/share/man to the file.

Unfortunately, since /usr/share/firmlinks is housed in the read-only system volume, you can not edit it as a user, or even as the super-user. Even in single-user mode, mounting the system volume group in read-write mode is prevented (ie: /sbin/mount -uw /does not work). It may be possible to r/w mount the the system volume as a subsidiary drive on a secondary system, and then make the edits; but that is more experimentation time than I was willing to put in.

So in short, the improved security of Catalina prevents updating that directory until Apple fixes the issue.

The above notes are relative to Catalina (macOS v 10.15.1). As it is a simple fix, I expect the issue will be corrected soon.

Answer 4

I just found this issue today and created the following aliases in ~/.zshrc to clean up the command's output:

alias apropos="apropos 2>/dev/null"
alias whatis="whatis 2>/dev/null"

The aliases remove the errors from the output by using redirection. The shell has two file descriptors for output. The standard output is file descriptor 1 and standard error output is file descriptor 2. The errors being generated by the makewhatis.local script are being sent to the standard error output.

Redirecting the standard error output is done by using the stderr file descriptor “2”, the output redirection operator “>”, and the destination file “/dev/null”. The /dev/null file is a special filesystem object that discards everything written into it. With the errors redirected only the desired results are displayed.

Answer 5

Try this one ():

1. sudo mkdir /System/Volumes/Data/usr/share/man
2. sudo /usr/libexec/makewhatis -o /System/Volumes/Data/usr/share/man/whatis

3. user@host ~ % cd /System/Volumes/Data/usr/share/man/
   user@host man % lsl
   total 384
   drwxr-xr-x  3 root  wheel  -          96 Nov  3 01:33 .
   drwxr-xr-x  4 root  wheel  sunlnk    128 Nov  3 01:32 ..
   -rw-r--r--  1 root  wheel  -      160236 Nov  3 01:33 whatis
   

   lsl is an alias for ls -laOe@ on my system

Fun facts:

• I don't know where this file is (except that the file is there) – the file can't be found with sudo find / -name "whatis" in the file system
• The file survives a reboot
• I have no clue whether this file is used at all by whatis/apropos/fish|bash|zsh shell completion (and solves your problems)

Answer 6

My own workaround to this problem consisted of creating a symlink at /usr/share/man/whatis. Since I use Emacs, I often rely on the M-x man command, which in turn uses apropos for auto-completion entries. With a broken whatis file, I can't get proper auto-completion results, which is very inconvenient.

I've also hitherto used a root cronjob script to keep my whatis file up-to-date, as follows:

*/5 * * * * /usr/libexec/makewhatis -o /usr/share/man/whatis /usr/share/man /Applications/Xcode.app/Contents/Developer/usr/share/man [etc. ...]

However, this no longer worked since /usr/share/man/whatis is no longer directly writable. For me, it was also not even present in the filesystem path any longer.

So, to solve this problem, after much investigation and consideration, I decided to create a symlink at /usr/share/man/whatis instead of an actual file, pointing to a writable file under my home folder: /Users/user/.emacs.d/whatis

I created this symlink by using this utility: https://github.com/fxgst/writeable_root

Specifically, I ran:

$ ./writeable_root
$ cd usr/share/man
$ ln -s /Users/user/.emacs.d/whatis whatis

And then the system rebooted, with my new symlink in place.

I could not use synthetic.conf to create this symlink, as synthetic.conf only creates symlinks directly under /. It will not create them in any lower directories in the filesystem hierarchy.

That way, I can have this file editable and I simply changed the cronjob to be run by my own user instead of as root, with the simple change that it modifies the file in my home folder instead of at /usr/share/man/whatis:

*/5 * * * * /usr/libexec/makewhatis -o ~/.emacs.d/whatis /usr/share/man /Applications/Xcode.app/Contents/Developer/usr/share/man [etc. ...]

Answer 7

Catalina broke the man command.

To ignore the error messages under /bin/bash, create an alias for man:

alias man='/usr/bin/man 2>/dev/null'

Answer 8

I believe this is fixed in MacOs 10.15.4. Thanks to @minopret for pointing this out in the comments on the original question. Running the unmodified whatis or apropos commands do not result in errors.