7

For late visitors to this question - the highest-voted answer is completely wrong. I offered the bounty but did not award it. The bounty was auto-awarded by the system. I cannot revoke it.

The workaround I had found has since failed again. If anyone manages to find a true working solution to this, a 500 pt bounty is all yours.

TL:DR
I've honed this issue down to one simple 'fact'.
The Time Machine drive will error with "The volume is the wrong format for a backup." on any attempt, by any method, to copy any file from within Backups.backupdb to any APFS volume. It works to any HFS+ volume.

I can copy a file to the TM volume root [outside the Backups.backupdb folder] & then copy out to APFS, but anything inside the backups folder will error.

I've trimmed most of the early experimental structure from this & left just the pertinent latest information

macOS Mojave, Mac Pro 5,1
Boot drive APFS [SSD], Time Machine drive HFS+ [HD]. All drives internal.

I have tested using two clones of my boot drive, as one SSD is mounted on a PCI card & the Mac considers it 'external'. This seems to make no difference. Boot drives are SSDs, GUID/APFS, case-insensitive.

Time Machine has been tested to three different drives, starting with the original year old backup & then with brand new backups & even brand new hard drives. All formatted GUID/HFS+ case-insensitive, as standard.

All drives have been tested with both TechTool Pro 14 & DiskWarrior [HFS only for DW] & all have a clean bill of heath.

All drives were formatted in Disk Utility using standard methods. Time Machine was set up & run as standard.
Files tested for recovery have been checked & opened to see they are still correct & not corrupted in any way - from original source & also directly from within Time Machine. Images & text files have been tested. No corruption detected at all.
Recoveries tested from Safe Mode [from where I deleted all local snapshots, including some sticky ones that wouldn't go away], no change.
Recoveries also tested from a new admin account. No change.

Additional, possibly pertinent [though possibly not] information.

Backups to the same Time Machine from HFS+ volumes will restore to the original HFS volume. Admin permission is requested.

Any file restored this way will then request admin perms even to delete [& even if 'ignore perms' is set on the drive.]. This is whether it was restored in the normal [working] manner back to HFS or manually dragged from an APFS backup to HFS.
Perms show the same as any other file on the volume, eg -rw-r--r--@ 1 glee staff which is exactly the same as any other file in the same folder that wasn't brought back from TM.
Removing all attributes xattr -c & setting perms to 'everyone' does not change this request for admin perm to delete.
Removing ACLs, chmod -N does stop this admin request, but I can't see what the offending ACL is 0: group:everyone deny write,delete,append,writeattr,writeextattr,chown unless it's that the owner seems to be 0 not 501.

Realised SIP was disabled. Re-enabled, ran fix perms on boot drive 

diskutil resetUserPermissions / id -u
trying yet another brand new Time Machine drive…
No change.

Any suggestions, next steps, tests to perform?

At the moment I am not willing to destroy the entire boot drive to clean install. I shall be able to try this as soon as Apple ship the new M1 iMacs, as I will then have a spare, almost identical Mac Pro I can play with.

Results of diskutil list & below diskutil apfs list
The boot volume is disk6 & disk7 which is PCI-mounted, though the OS thinks it's external.
If I boot instead from the other 'internal' SSD, 'Clone' I get exactly the same results.

/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                  Apple_HFS Downloads               1.0 TB     disk0s2
   3:       Microsoft Basic Data MacWin7                 202.1 GB   disk0s3
   4:                  Apple_HFS Empty                   1.8 TB     disk0s4
   5:                 Apple_Boot Recovery HD             650.0 MB   disk0s5

/dev/disk1 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk1
   1:                        EFI EFI                     209.7 MB   disk1s1
   2:                  Apple_HFS OhDaSpace               4.0 TB     disk1s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk1s3


/dev/disk2 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *3.0 TB     disk2
   1:                        EFI EFI                     209.7 MB   disk2s1
   2:                  Apple_HFS Tim O'Sheen             3.0 TB     disk2s2


/dev/disk3 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk3
   1:                        EFI EFI                     209.7 MB   disk3s1
   2:                 Apple_APFS Container disk4         1000.0 GB  disk3s2


/dev/disk4 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +1000.0 GB  disk4
                                 Physical Store disk3s2
   1:                APFS Volume SSD2                    26.3 GB    disk4s1
   2:                APFS Volume Clone                   478.2 GB   disk4s2
   3:                APFS Volume Preboot                 21.2 MB    disk4s3
   4:                APFS Volume Recovery                515.6 MB   disk4s4
   5:                APFS Volume VM                      20.5 KB    disk4s5


/dev/disk5 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk5
   1:                        EFI EFI                     209.7 MB   disk5s1
   2:                  Apple_HFS 4Space                  3.0 TB     disk5s2
   3:       Microsoft Basic Data MacWin7-safety          202.1 GB   disk5s3
   4:       Microsoft Basic Data Acronis                 829.8 GB   disk5s4


/dev/disk6 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *1.0 TB     disk6
   1:                        EFI EFI                     209.7 MB   disk6s1
   2:                 Apple_APFS Container disk7         1.0 TB     disk6s2


/dev/disk7 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +1.0 TB     disk7
                                 Physical Store disk6s2
   1:                APFS Volume KickMeHard              397.8 GB   disk7s1
   2:                APFS Volume Preboot                 21.2 MB    disk7s2
   3:                APFS Volume VM                      2.1 GB     disk7s3
   4:                APFS Volume Recovery                507.6 MB   disk7s4

+-- Container disk7 829CC676-CE99-4AF1-8119-2F1647C8E1EE
    ====================================================
    APFS Container Reference:     disk7
    Size (Capacity Ceiling):      1023999787008 B (1.0 TB)
    Capacity In Use By Volumes:   400643194880 B (400.6 GB) (39.1% used)
    Capacity Not Allocated:       623356592128 B (623.4 GB) (60.9% free)
    |
    +-< Physical Store disk6s2 88456C48-AA1A-4A18-81AC-6FC0218583B5
    |   -----------------------------------------------------------
    |   APFS Physical Store Disk:   disk6s2
    |   Size:                       1023999787008 B (1.0 TB)
    |
    +-> Volume disk7s1 7828165C-2088-4328-9B9C-FE2CF9942E7F
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk7s1 (No specific role)
    |   Name:                      KickMeHard (Case-insensitive)
    |   Mount Point:               /
    |   Capacity Consumed:         397756780544 B (397.8 GB)
    |   FileVault:                 No
    |
    +-> Volume disk7s2 E34F4E10-8186-476C-AB92-1E8B989C9FE1
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk7s2 (Preboot)
    |   Name:                      Preboot (Case-insensitive)
    |   Mount Point:               Not Mounted
    |   Capacity Consumed:         21213184 B (21.2 MB)
    |   FileVault:                 No
    |
    +-> Volume disk7s3 04FB46F5-CA22-4B53-8510-53D438E80EDA
    |   ---------------------------------------------------
    |   APFS Volume Disk (Role):   disk7s3 (VM)
    |   Name:                      VM (Case-insensitive)
    |   Mount Point:               /private/var/vm
    |   Capacity Consumed:         2147504128 B (2.1 GB)
    |   FileVault:                 No
    |
    +-> Volume disk7s4 4F395B93-B320-4046-AC59-4420A7C10C5F
        ---------------------------------------------------
        APFS Volume Disk (Role):   disk7s4 (Recovery)
        Name:                      Recovery (Case-insensitive)
        Mount Point:               Not Mounted
        Capacity Consumed:         507604992 B (507.6 MB)
        FileVault:                 No

Results from the other SSD, which contains 'Clone' just for reference

+-- Container disk4 3AF85C3F-0ABD-4F06-BE2D-4DE42BD4C286
|   ====================================================
|   APFS Container Reference:     disk4
|   Size (Capacity Ceiling):      999995129856 B (1000.0 GB)
|   Capacity In Use By Volumes:   505167609856 B (505.2 GB) (50.5% used)
|   Capacity Not Allocated:       494827520000 B (494.8 GB) (49.5% free)
|   |
|   +-< Physical Store disk3s2 4A9E7A66-914F-4321-A4E6-FD3B3B956CA7
|   |   -----------------------------------------------------------
|   |   APFS Physical Store Disk:   disk3s2
|   |   Size:                       999995129856 B (1000.0 GB)
|   |
|   +-> Volume disk4s1 54355C8A-F91B-4EEB-A954-42E626E4388E
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk4s1 (No specific role)
|   |   Name:                      SSD2 (Case-insensitive)
|   |   Mount Point:               /Volumes/SSD2
|   |   Capacity Consumed:         26257211392 B (26.3 GB)
|   |   FileVault:                 No
|   |
|   +-> Volume disk4s2 558EF49B-214C-4336-9A5C-389B37AA69D9
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk4s2 (No specific role)
|   |   Name:                      Clone (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         478165762048 B (478.2 GB)
|   |   FileVault:                 No
|   |
|   +-> Volume disk4s3 F41AAE6B-B466-4461-970D-9E323E2A6319
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk4s3 (Preboot)
|   |   Name:                      Preboot (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         21204992 B (21.2 MB)
|   |   FileVault:                 No
|   |
|   +-> Volume disk4s4 897B0313-B840-485B-913D-1EAFD5A92DFE
|   |   ---------------------------------------------------
|   |   APFS Volume Disk (Role):   disk4s4 (Recovery)
|   |   Name:                      Recovery (Case-insensitive)
|   |   Mount Point:               Not Mounted
|   |   Capacity Consumed:         515588096 B (515.6 MB)
|   |   FileVault:                 No
|   |
|   +-> Volume disk4s5 608464EF-3335-49BD-9FAD-C8E83DE91DEE
|       ---------------------------------------------------
|       APFS Volume Disk (Role):   disk4s5 (VM)
|       Name:                      VM (Case-insensitive)
|       Mount Point:               Not Mounted
|       Capacity Consumed:         20480 B (20.5 KB)
|       FileVault:                 No
|

From comments - ls -lad@ produces 20 or so versions of this, varying only in Snap # & date/time

$ ls -lad@ /Volumes/Tim\ O\'Shanter/Backups.backupdb/*/*
drwxr-xr-x@ 8 root  admin  272 30 May 19:26 /Volumes/Tim O'Shanter/Backups.backupdb/TetsMac/2021-05-30-72624 pm
    com.apple.backup.SnapshotNumber   1 
    com.apple.backup.SnapshotVersion      1 
    com.apple.backupd.SnapshotCompletionDate     17 
    com.apple.backupd.SnapshotStartDate  17 
    com.apple.backupd.SnapshotState   2 
    com.apple.backupd.SnapshotTotalBytesCopied   12 
    com.apple.backupd.SnapshotType    2
Tetsujin
  • 115,663
  • Are you trying to recover files using the Time Machine interface (Enter Time Machine) or are you using the Finder UI or Terminal to transfer the files from within the Backups.backupdb folder? Perhaps I'm not reading your question correctly. – IconDaemon May 22 '21 at 14:41
  • Either, or both - same result. i only tested the manual version after many fails at the Enter TM method. – Tetsujin May 22 '21 at 14:58
  • How did you copy the Time Machine to another disk? Because it sounds like something didn’t copy correctly. TM uses hard links and other low level file attributes that can be lost in a copy unless you take care to preserve them. – James Brickley May 22 '21 at 15:19
  • I didn't copy Time Machine to another disk - I tested copying one file, after the 'proper' restore method failed. – Tetsujin May 22 '21 at 15:29
  • I wish I knew the answer, but honestly, your best bet to get this figured out is to ask Howard Oakley if he would take a look. Contact information is at https://eclecticlight.co/about/ or @howardnoakley on Twitter. – TJ Luoma May 29 '21 at 13:56
  • Can you add some lines from the command on your TM Volume ls -lad@ /Volumes/Name_Time_Machine/Backups.backupdb/*/* ? –  Jun 01 '21 at 06:11
  • @Jean_JD - added one iteration, there are many others almost identical. This is yet another brand new backup, only been running a day. – Tetsujin Jun 01 '21 at 07:03

5 Answers5

3

UPDATE AGAIN:

A thing that sets your system apart from "ordinary" Mac systems is the fact that your boot drive is seen as external by the system, even though it is a SSD connected via PCI internally to the computer.

Depending on what kind of SSD you have, and exactly how it is connected, you could change the setup so that it is recognized as an internal boot drive by macOS. This could change what the system will recognize as okay to restore.

Vendors of some SSDs make a tool available to change whether or not the SSD is "internal" or "external". Essentially, for drives connected over a SCSI or similar-to-SCSI (e.g. USB) bus, this means that the disk controller has the field "Removable" set to 0 instead of 1. Using these tools is using hassle free.

For some drives you can make the change easily by simply overwriting the first byte on the drive. The first byte is a bit field where one specific bit is 0 for non-removable (internal) disks and 1 for removable (external) disks. In most cases you can simply write a zero to the first byte on the disk. I would advise taking a backup (i.e. dd-based backup or similar - obviously not Time Machine as that's not working for you) of the drive before doing that - especially if you're used to working a hex-editor and/or the dd command.

UPDATE:

After you've refined the question, I went back and actually tried this on an old Mojave Mac to see what I can find.

It seems the issue is that Finder is trying to be "intelligent", so whenever you try to move a Backups.backupdb folder or parts of it to a drive that is not supported as a destination by Time Machine, it will complain that the destination volume is "the wrong format".

On Mojava you couldn't use APFS volumes for Time Machine backups - and that's why you get this error. The same error message can be seen if you try to copy the file to other types of destinations that are unsupported by Time Machine, for example if you try to copy the Backups.backupdb folder into Dropbox.

If you want to copy that way, you can work-around the message by first copying the file outside the Backups.backupdb folder, and then copying it to the connected APFS drive. Or you could copy to a HFS+ drive.

It might seem weird that the Backups.backupdb folder is treated differently than any other folder, but it is not uncommon for Apple to do stuff like this in a pragmatic manner. Even the actual operating system kernel treats Backups.backupdb folder as something special, so that they for example do not generate file system events themselves as any other folder would do.

OLD ANSWER:

The bottom line is that your backup has somehow been corrupted. As you're having trouble restoring a specific file that was saved on the 6th of May, it is most probably at this time the corruption occurred.

It could be a software bug, a hardware error - or something simple like an unexpected power loss that occured to render the backup structure incomplete. Most likely the file is not really stored, but Time Machine "thinks" it is there.

You mention that you can see the specific file you're trying to restore in the folders for the 7th, 8th, etc. I would advise simply using Finder or the cp command in the Terminal to copy the file from the backup to a normal disk. You would copy the file from for example the folder for the 7th.

I'm afraid though, that most likely you'll find that the copy you make is incomplete - i.e. 0 bytes or can't copy at all. This is due to the file actually never having been saved properly in the first place (due to the one of the causes listed above).

As you have used the drive for performing backups after the corruption occurred, and now a great deal of time has passed, it is very unlikely that you will be able to use any data recovery tool to recover the data that you originally intended to store in the file.

UPDATE: You have since updated the question and the update indicates that the source volume and the Time Machine disk are somehow incompatible. This can happen for example if you have a case-sensitive Time Machine disk and try to restore it on an case-insensitive disk. That is not supported.

jksoegaard
  • 77,783
  • This has been tested with several backups, original & absolutely brand-new, as mentioned in the question. Also using multiple test files, including some made only immediately before the first TM backup. The early test files exist in another location on another disk & are fine on there. They're gone from the TM backup now as I made several new versions on different drives; eventually once more sacrificing the history [it's mostly backed up online too] – Tetsujin May 28 '21 at 11:18
  • I’m not sure I understand what you’re saying, but it is really no surprise that the problem occurs in multiple backups. Time Machine only actually stores the data for the file once, when you do further backups and the file hasn’t changed, it is not stored again and again. So if it is broken for the first day it is there, then it will be broken for each following day (until the file was changed and a new copy is made). – jksoegaard May 28 '21 at 16:57
  • You're not reading the question properly. I have done this multiple times to multiple drives. The chances of a single file being broken each time are zero, if the original can be shown to still be good. – Tetsujin May 28 '21 at 16:58
  • 1
    The question is hard to read. Are you saying that every time you start a Time Machine backup on a new drive, it will let you backup - but you cannot restore anything? – jksoegaard May 28 '21 at 18:46
  • Or are you saying that it is this one particular file that when backed up makes the Time Machine backup stop working? – jksoegaard May 28 '21 at 18:47
  • The first one. I thought that was fairly clear. If it was the second, I'd have just discarded the file. You really do need to read what I've written & the testing methiods apllied so far. – Tetsujin May 28 '21 at 18:55
  • Well, I did read it before I answered - but your question is really unclear. Have you tried just opening the files on the backup drive (I.e. not restoring them) - does that work? Do the files actually contain the expected data? – jksoegaard May 28 '21 at 23:18
  • I've added further tests to the question & honed down the issue still further. There is nothing wrong with any of the files, only the 'recovery'. – Tetsujin May 29 '21 at 06:50
  • Can you include the detailed information about your disk volume formats - please add the output of "diskutil cs list" and "diskutil apfs list" to the question. – jksoegaard May 29 '21 at 07:30
  • There's no cs list, I'm on Mojave, but I've added list & apfs – Tetsujin May 29 '21 at 07:39
  • The is no case-sensitivity issue. The issue has come down to whether the disk I'm trying to copy to is APFS or HFS+ – Tetsujin May 29 '21 at 09:49
  • Which disk name are you trying to copy from (Time Machine backup) and to (source)? I.e. the one that fails. – jksoegaard May 29 '21 at 10:26
  • I've backed to three different Time Machine instances, the one currently called Tim O'Sheen was the original, now wiped & started again from scratch. The other [no more successful] was on a brand new HD, no longer in the Mac. Recovery has been tried to KickMeHard & Clone [of course a clone of the regular boot drive]. Manual copies have been tried from Tim O'Sheen to various mounted disks in turn; consistently it will not copy to APFS but will to HFS+. Manual copies to the Time Machine volume outside the backups folder work to any disk, only files inside the backup exhibit this behaviour. – Tetsujin May 29 '21 at 10:31
  • I've tidied the question to remove earlier experiments, leaving only the current issue - recovery or copy to APFS fails, to HFS+ works. – Tetsujin May 29 '21 at 10:59
  • Really like your update. – Gilby May 31 '21 at 09:00
  • I half see the logic in the updated answer… but not entirely. The boot drive is in the now insisted-on APFS, the Time Machine drive is in the only format you can make one in Mojave, HFS+. Why on earth that raises the error as claimed utterly escapes me. I'm doing it the only way it can be done. – Tetsujin Jun 01 '21 at 10:56
  • But are you restoring the files directly to the boot drive? - Or is it an externally connected APFS drive you're trying to restore to? (this was my impression).... I'm pretty sure that your experience with this would be different if you upgraded to Catalina or even Big Sur. – jksoegaard Jun 01 '21 at 11:00
  • I have no option of upgrading. This is a Mac Pro 5,1. All drives are internal [even though one s PCI-mounted, which the Mac considers external] I have also tested with a regular SSD which the Mac doesn't think is external. All of this information is in the question already. I have now run tests on 2 physically different boot drives mounted to different physical locations, & 4 Time Machine drives. For info - I have 6 physical drives in this Mac [2 SSD, 4 HD] for a total of 16TB storage, & a boxful of spare HDs. Swapping drives in & out is a 2 minute job. – Tetsujin Jun 01 '21 at 11:13
  • There's so much information that it is hard to keep the details straight. Do you get the error when restoring the files directly to the boot drive? (I know you have all sorts of other drives and did various other tests - but I just want this simple question answered as it is hard to see from the question whether this is the case or not). – jksoegaard Jun 01 '21 at 11:36
  • Yes. I first observed the error doing a regular restore of a file from Time Machine. Later experiments have broadened where the issue might be, in that it errors when I try to Restore to any backed-up APFS volume, boot or otherwise, but it works if the backed-up volume was HFS. [As a general rule, my SSDs are all APFS, HDs are still HFS] – Tetsujin Jun 01 '21 at 11:48
  • When you say "it works if the backed-up volume was HFS" - do you mean that the target volume you're restoring to is HFS? - Or do you mean that if the source drive that you took the Time Machine backup of was HFS+, then you can restore to any drive? - I.e. the problem only occurs when the source for the original Time Machine backup was an APFS drive? – jksoegaard Jun 01 '21 at 11:51
  • The Restore volume is "the one that was originally backed up" so APFS backed to TM [which is HFS by definition] then restore to APFS - fails. HFS backed up then restored to the same HFS volume - works. The confusion really arises if I manually drag a file out of the TM hierarchy, no matter which source volume. It will drag to an HFS volume, but dragging to APFS fails with the same error. This boils down to "Time Machine will not allow me to copy any file to any APFS volume". – Tetsujin Jun 01 '21 at 11:55
  • It's not dependant on whether the boot drive is considered external or not. Already mentioned that. – Tetsujin Jun 01 '21 at 18:38
1

Six months later & I just discovered this has silently broken again, right back to square one.
APFS is just not ready for mainstream at Mojave.

Update April 2022
My current working method is to attempt recovery from Time Machine, which opens the correct, deeply-buried folder from the backup on the Desktop. Finder errors [as expected]. Manual copy to the correct location fails with the same reason…
so, first I copy to another drive, formatted HFS+, then I can copy that to the correct location. It demands admin credentials to do this, & then again to delete the intermediate copy.
This works, & other than being tedious, seems to be the 'solution'. It will just have to do for now.

I appear to have fixed this, at least for now, by taking an entirely different Mac [the now-spare Mac Pro mentioned in my question], formatting, installing a brand new clean OS, then migrating over ethernet directly from the Mac with the issue.

I had previously attempted to migrate from Time Machine after a clean install, but this was unsuccessful - see Migration Assistant fail - so I knew that was not going to be an option.

So, after a clean install + direct computer to computer migration, I now have been able to make yet another brand new Time Machine backup on Mac 'B' & it will restore deleted files from 'Enter Time Machine' back to the APFS boot drive, without error.

Joy.

Next experiment will be to clone this back to Mac 'A' & see if it remains fully functional.
Update
Yes, still fully functional after cloning back to the original Mac.

Of course, I have still no idea why this worked, but I'm happy that it did.

Tetsujin
  • 115,663
1

I found a way to recover files. Select the file or folder you want to recover, right click and chose "compress". The OS will save the zipped file on Desktop.

Chriz74
  • 111
  • 4
0

On Big Sur (11.6). I replicated the finder error message "The volume is the wrong format for a backup" by copying the root folder (2021-09-26-xxxxx) of the latest backup from a Time Machine drive to another external HD formatted as APFS.

I was actually able to copy all three folders to the APFS drive separately (Macintosh HD, Recovery and Macintosh HD - Data)

I'm considering this a valid workaround for a bug.

Wintermutetwin
  • 1
0

I was able to successfully move files that gave this message to my hard drive by first transferring them to the Trash on my MacBook Pro running Mojave. No clue why I even thought to try it. Go figure.

JJS
  • 1