Android du and df very different: File System Corrupted?

Question

On an Android Nougat (7.1.1) set top box, du and df of my /data partition are very different:

$ adb shell du -sh /data
1.0G    /data
$ adb shell df -H /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/block/sda9  10G   10G  469M  96% /data

I don’t think that anything else is mounted under /data:

$ adb shell mount | grep "\/data"
/dev/block/sda9 on /data type ext4 (rw,seclabel,nosuid,nodev,relatime,discard,noauto_da_alloc,data=ordered)
$ adb shell mount | grep "sda9"
/dev/block/sda9 on /data type ext4 (rw,seclabel,nosuid,nodev,relatime,discard,noauto_da_alloc,data=ordered)

lsof indicates to me that there are dozens of processes holding hundreds of small deleted files of the sort

init          1       root    3w      CHR               1,11       0t0      14517 /dev/__kmsg__ (deleted)
ueventd     391       root    3w      CHR               1,11       0t0      14593 /dev/__kmsg__ (deleted)
  . . .
main        756       root  mem   unknown                                         /dev/ashmem/dalvik-large object space allocation (deleted)
main        756       root  mem   unknown                                         /dev/ashmem/dalvik-large object space allocation (deleted)
  . . .
omm.times  3934     system  mem   unknown                                         /dev/ashmem/dalvik-mark sweep sweep array free buffer (deleted)
omm.times  3934     system  mem   unknown                                         /dev/ashmem/dalvik-mark sweep sweep array free buffer (deleted)

Using busybox, results are different (df shows much less usage), but there is still a large discrepancy between du and df:

$ adb shell busybox du -sh /data
389.4M  /data
$ adb shell busybox df -h /data
Filesystem                Size      Used Available Use% Mounted on
/dev/block/bootdevice/by-name/userdata
                          9.7G      8.6G      1.1G  89% /data

And using toybox, the results are similar to busybox:

$ adb shell toybox du -sh /data
389M    /data
$ adb shell toybox df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/block/sda9  10G  8.6G  1.1G  89% /data

It think that it's important to note that these numbers remain similar after a reboot.

Also, I wish to note that I cannot OTA update this device's Android due to running out of disk space, even though the OTA image is only about 1GB in size. This fact leads me to believe that the results of df are accurate in terms of actual available disk space.

All adb commands are run as root i.e., I did adb root before executing them. But at risk of making this question be too verbose, here's everything run at the adb prompt:

$ adb shell
Z:/ # whoami
root
Z:/ # du -sh /data
389M    /data
Z:/ # df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/block/sda9  10G  8.6G  1.1G  89% /data
10.197.12.14:/ # busybox du -sh /data
389.4M  /data
Z:/ # busybox df -h /data
Filesystem                Size      Used Available Use% Mounted on
/dev/block/bootdevice/by-name/userdata
                          9.7G      8.6G      1.1G  89% /data
Z:/ # toybox du -sh /data
389M    /data
Z:/ # toybox df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/block/sda9  10G  8.6G  1.1G  89% /data
Z:/ # 

A user in comments has asked for the following:

Z:/ # while read num; do (( sum += num )); done <<< $(find /data -type f -exec stat -c%b {} +); expr $sum / 2048
393

I understand that du reports free space by scanning reachable nodes, and that this could therefore mean that my file system is corrupt. However, unfortunately, this is for devices in the field, and I don’t want to have to (and don’t think I can) run fsck or e2fsck.

What might be causing the huge difference between du and df on this device, and how might this problem be resolved? I’m happy to provide further logs.

Answer 1

df stands for "Disk Free". That means, it just needs to check with the "global file system information" about free space (accessible for all users). The size of the file system itself is also "globally available". The difference logically is the "used space". Done.

du works differently: not on file system level but on file level. It needs to look up each directory and file separately, gather it's size, and sum that up. So it can only take into account what it has access to.

Now you've run both commands as "normal user" – who has no permission to traverse e.g. /data/data (where all the apps' data is stored). So it can only sum up a portion of what space is occupied on /data. Would you run both commands as root user (as the screenshot in the comment by alecxs shows), numbers would be much closer together – as usually root can access all files. Though as again Alex correctly points out: there is still SELinux which might deny full access – which according to him is a known issue for mtk-su on the Amazon FireTV stick, and might be the culprit here, and isolated mount namespaces could be yet another reason for invisible files.

TL;DR: there's neither a file system corruption, nor a broken du binary necessarily causing the difference. All might work as designed. If you want to make sure, you can run fsck (if you prefer, in read-only mode) to rule out the former – and set up a test case on a different place for the latter.

Answer 2

The /data partition was indeed corrupted.

After running e2fsck -p /dev/block/sda9, here was the updated du and df (again, after doing an adb root):

$ adb shell du -s /data
1.3G    /data
$ adb shell df -H /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/block/sda9  10G  1.5G  8.9G  15% /data

After fixing the file system, the device ran properly and updated successfully thereafter.

(Note that I found that I had to run e2fsck with the -p option; it wouldn't run interactively for some reason.)

A portion of the output of the above e2fsck command is

data contains a file system with errors, check forced.
data: Deleted inode 163845 has zero dtime.  FIXED.
data: Deleted inode 163847 has zero dtime.  FIXED.
data: Deleted inode 163849 has zero dtime.  FIXED.
   . . . [snip ] . . .
data: Deleted inode 213003 has zero dtime.  FIXED.
data: Deleted inode 262223 has zero dtime.  FIXED.
data: Deleted inode 303108 has zero dtime.  FIXED.
data: 5203/655360 files (2.0% non-contiguous), 186831/2621440 blocks

There were a total of 140 lines of the sort data: Deleted inode...

FWIW, we are guessing that this problem was possibly caused by the filesystem becoming full with logs, which then caused corruption during the next attempted download of OTA updates: Those downloads failed, but thereafter left the file system in a corrupted state. Subsequent attempts at those downloads seemed to further corrupt the file system.

Thanks very much to @alecxs for your persistence and relevant details. (However, please note that to run e2fsck, the partition /dev/block/sda9 had to be UNmounted.)