ADB pull stops after first error

Question

I am trying to do a full copy of /sdcard of my unrooted OnePlus 3T to my Windows PC.

So far I've tried:

• adb pull /sdcard/ C:\mylocalfolder\

  Works OK but as soon as there is an error it stops the process, so the copy process has been truncated. The error is due to a file containing a colon ":" in the name, so it can't be copied to Windows.

---

• adb exec-out tar c sdcard > sdcard_backup.tar

  Doesn't work, it stops immediately and creates a corrupted 4kb file.

---

• Copy the files using Windows via Data Transfer mode

  Windows halts calculating time to copy files.

---

Adb pull would be the ideal solution so far if is there a way to skip/ignore errors so it can continue copying the files, or perhaps try to skip the files with certain characters.

I've tried both Adb version 1.0.32 and 1.0.41 (latest version as of July 2020).

Any suggestion?

Answer 1

Quoting is recommended to make it more clear which commands will executed on client and which pipes/redirections are designated to run on host side. Basically this command should work:

adb exec-out "tar -c /sdcard" > sdcard_backup.tar

Have a look into corrupted 4 kb file. you will see the tar header and file content. there is a leading string before the tar file begin:

removing leading '/' from member names

$ hexdump -C sdcard_backup.tar
00000000  72 65 6d 6f 76 69 6e 67  20 6c 65 61 64 69 6e 67  |removing leading|
00000010  20 27 2f 27 20 66 72 6f  6d 20 6d 65 6d 62 65 72  | '/' from member|
00000020  20 6e 61 6d 65 73 0a 73  64 63 61 72 64 00 00 00  | names.sdcard...|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000080  00 00 00 00 00 00 00 00  00 00 00 30 30 30 30 37  |...........00007|
00000090  37 37 00 30 30 30 30 30  30 30 00 30 30 30 30 30  |77.0000000.00000|
000000a0  30 30 00 30 30 30 30 30  30 30 30 30 30 30 00 30  |00.00000000000.0|
000000b0  30 30 30 30 30 30 30 30  30 30 00 30 31 34 37 35  |0000000000.01475|
000000c0  37 00 20 32 2f 73 74 6f  72 61 67 65 2f 73 65 6c  |7. 2/storage/sel|
000000d0  66 2f 70 72 69 6d 61 72  79 00 00 00 00 00 00 00  |f/primary.......|

problematic exec-out will redirect stdin and stderr. at least avoid error messages, or even better catch stderr completely

adb exec-out "tar -c sdcard" > sdcard_backup.tar
adb exec-out "tar -c sdcard 2> /dev/null" > sdcard_backup.tar

Note the skipped leading / root dir in command line. This time a 4 kb tar file is produced

$ hexdump -C sdcard_backup.tar
00000000  73 64 63 61 72 64 00 00  00 00 00 00 00 00 00 00  |sdcard..........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000060  00 00 00 00 30 30 30 30  37 37 37 00 30 30 30 30  |....0000777.0000|
00000070  30 30 30 00 30 30 30 30  30 30 30 00 30 30 30 30  |000.0000000.0000|
00000080  30 30 30 30 30 30 30 00  30 30 30 30 30 30 30 30  |0000000.00000000|
00000090  30 30 30 00 30 31 34 37  35 37 00 20 32 2f 73 74  |000.014757. 2/st|
000000a0  6f 72 61 67 65 2f 73 65  6c 66 2f 70 72 69 6d 61  |orage/self/prima|
000000b0  72 79 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |ry..............|

looks like a healthy tar archive. let tar do it's work. it reveals the file is a symbolic link

$ tar -vtf sdcard_backup.tar
lrwxrwxrwx root/root         0 1970-01-01 01:00 sdcard -> /storage/self/primary

In boot mode /sdcard is just a symlink to /storage/emulated/0
while in recovery mode /sdcard is a bind-mount of /data/media/0

tar does not traverse symlinks by default, therefore the command works in recovery mode. for boot mode use tar -h flag, or provide directory path instead of it's symlink.

furthermore on windows cmd.exe may add additional \r carriage return for each \n linefeed. therefore avoid streaming plain data, instead use compression algorithm which produces no linefeeds at all. this can be done with gzip

adb exec-out "tar -ch sdcard | gzip" > sdcard_backup.tar.gz
adb exec-out "tar -c storage/emulated/0 | gzip" > sdcard_backup.tar.gz

This tarball archive can be extracted from tar as most tar implementations will auto-detect and invoke gzip automatically.

$ tar -vtf sdcard_backup.tar.gz
drwxrwx--x root/sdcard_rw    0 2018-01-01 01:00 storage/emulated/0/
drwxrwx--x root/sdcard_rw    0 2018-01-01 01:00 storage/emulated/0/Samsung/
drwxrwx--x root/sdcard_rw    0 2018-01-01 01:00 storage/emulated/0/Samsung/Music/

For windows we can use 7-Zip as alternative unpacking program, it will handle most of compression algorithms.