Disk usage discrepancy - Linux

[Circlotron]

I have a 1TB HDD and the used space figure is WAY different to the used on disk space figure. Also when using the Control Centre to look at Disks it gives a significantly different figure again. What could be happening?

[magic]

One thing is the idiotic "binary" versus "decimal" gigabytes issue: traditionally 1KB was considered 1024 bytes and 1GB was 1024³, which is some weird number somewhat larger than 10⁹. This likely accounts for the difference in overall disk capacity reported by different tools. HDD manufacturers used the binary convention until about 40GB, then switched to decimal in order to sell smaller disks at the same advertised capacity. At some point Apple and Microsoft did the switch too, but on Linux it's a mixup. The tool which reports 1.0TB uses the decimal convention which manufacturers use today, the other does not.

But there is still remaining a difference in percentage of free space, about 5%. I'm not entirely sure, but this is an EXT4 filesystem and 5% is the exact amount of space which is "reverved" by default, i.e. only the root user can create new files or expand existing ones when total free space falls below 5%. It's possible that one tool doesn't consider this space free, while the other does

You can check for this by running (as root):

Code: [Select]

tune2fs -l /dev/sdband looking for information about reserved blocks and block size (most likely 1 block is 4KB = 4096B).

To disable reserved blocks altogether, run

Code: [Select]

tune2fs -r 0 /dev/sdbThe filesystem may need to be unmounted for this, not sure.

[Nusa]

The disk report is using decimal numbers. 1 TB is spelled out in bytes (actually a smidge larger than). 1 TB - 448 GB = 552 GB. 552/1000 = 55.2%

The folder report is using binary numbers. 1 TB decimal (10*12, with 1000 GB per TB) = 0.9095 TB binary (2**40 with 1024 GB per TB)

magic is right the numbers don't quite match up, but they're in the ballpark for overall disk usage. Perhaps there's a trash folder full of not-completely-deleted stuff? Or some of the thoughts magic mentions.

The 130.5 GB number, that's folder specific, while the rest of the page is for the whole disk. The "on disk" number looks the same, referring to the space taken by that folder on the disk, usually only significantly different if there are many files that are smaller than the block size. In this case, they're close enough they round to the same number.

[DiTBho]

Code: [Select]

# lsprettysize *
      138 Kbyte xtrans-1.2.5.tar.bz2
       12 Kbyte xulrunner-1.9.2-patches-0.5.tar.bz2
       14 Kbyte xulrunner-1.9.2-patches-0.9.tar.bz2
      628 Kbyte xvidcore-1.2.2.tar.bz2
       94 Kbyte xvinfo-1.1.0.tar.bz2
      114 Kbyte xwd-1.0.3.tar.bz2
      107 Kbyte xwininfo-1.0.5.tar.bz2
      101 Kbyte xwud-1.0.2.tar.bz2
     1013 Kbyte xz-4.999.9beta.tar.gz
      213 Kbyte yaboot-1.3.14.tar.gz
       63 Kbyte yacc-1.9.1.tar.Z
        1 Mbyte zip30.zip
      415 Kbyte zlib-1.2.3.tar.bz2
      680 Kbyte zziplib-0.13.58.tar.bz2

years ago I developed a simple tool to list files this way.

Internally, there are two functions:
1) base10, so "1K" means "1000"
2) base2, so "1K" means "1024"

[Circlotron]

You can check for this by running (as root):

Code: [Select]

tune2fs -l /dev/sdband looking for information about reserved blocks and block size (most likely 1 block is 4KB = 4096B).

Run that and I get:

Code: [Select]

tune2fs 1.44.1 (24-Mar-2018)
tune2fs: Bad magic number in super-block while trying to open /dev/sdb
Found a gpt partition table in /dev/sdb
What is the significance of that?

[Nominal Animal]

Try
    tune2fs -l /dev/sdb1
instead.

(/dev/sdb refers to the entire disk, including partition tables; and /dev/sdb1 is its first partition, named on the Device: line in the second screenshot.)

[Nominal Animal]

If we are precise, then
    1 kB = 1,000 bytes
    1 KiB = 1,024 bytes
    1 MB = 1,000,000 bytes
    1 MiB = 1,024 KiB = 1,048,576 bytes
    1 GB = 1,000,000,000 bytes
    1 GiB = 1,024 MiB = 1,048,576 KiB = 1,073,741,824 bytes
    1 TB = 1,000,000,000,000 bytes
    1 TiB = 1,024 GiB = 1,048,576 MiB = 1,073,741,824 KiB = 1,099,511,627,776 bytes
per clause 4 of the ISO/IEC 80000-13:2008 standard.  The names are silly ("kibibyte", "mebibyte", "gibibyte", "tebibyte"), but we can be precise and unambiguous if we want, nowadays.

On the command line,
    df -h
gives you a summary of your disk usage in powers of ten (kB, MB, GB, TB) two (KiB, MiB, GiB, TiB); for example
    Filesystem                   Size  Used Avail Use% Mounted on
    udev                         7.7G     0  7.7G   0% /dev
    tmpfs                        1.6G  1.7M  1.6G   1% /run
    /dev/mapper/nvme0n1p3_crypt  464G  350G   91G  80% /
    tmpfs                        7.8G     0  7.8G   0% /dev/shm
    tmpfs                        5.0M  4.0K  5.0M   1% /run/lock
    tmpfs                        7.8G     0  7.8G   0% /sys/fs/cgroup
    /dev/nvme0n1p2               3.7G  445M  3.1G  13% /boot
    /dev/nvme0n1p1               749M  6.1M  743M   1% /boot/efi
    tmpfs                        1.6G   56K  1.6G   1% /run/user/1000

On the other hand, use an upper-case H,
    df -H
and you get the summary in powers of two (KiB, MiB, GiB, TiB) ten (kB, MB, GB, TB):
    Filesystem                   Size  Used Avail Use% Mounted on
    udev                         8.3G     0  8.3G   0% /dev
    tmpfs                        1.7G  1.8M  1.7G   1% /run
    /dev/mapper/nvme0n1p3_crypt  499G  376G   98G  80% /
    tmpfs                        8.4G     0  8.4G   0% /dev/shm
    tmpfs                        5.3M  4.1k  5.3M   1% /run/lock
    tmpfs                        8.4G     0  8.4G   0% /sys/fs/cgroup
    /dev/nvme0n1p2               4.0G  467M  3.3G  13% /boot
    /dev/nvme0n1p1               785M  6.4M  779M   1% /boot/efi
    tmpfs                        1.7G   58k  1.7G   1% /run/user/1000

Edited: kjpye noticed I got them swapped myself.    For details, see man 1 df.

[Zero999]

The following question was asked at a pub quiz last week: "How many megabytes in a terabyte?"

I told my team mate the quiz master didn't specify whether it was TB or TiB, but he didn't know the difference, so we answered 1024. Fortunately it was right, but if not we should have been able to argue it. Afterwards, he Googled it.

[magic]

1048576

[eutectique]

years ago I developed a simple tool to list files this way.

From man ls:

Code: [Select]


    -h, --human-readable
        with -l and -s, print sizes like 1K 234M 2G etc.



[DiTBho]

years ago I developed a simple tool to list files this way.

From man ls:

Code: [Select]


    -h, --human-readable
        with -l and -s, print sizes like 1K 234M 2G etc.



I wrote a simplified version of "ls" because I some need formatting that isn't offered by ls by default, and I don't have any desire to hack "ls", since then I would have to fork an ebuild (ls is part of sys-apps/coreutils) into an Overlay for the only purpose of adding a patch to ls.

Code: [Select]

Portage: sys-apps/coreutils
Binary: /bin/ls


Code: [Select]

Overlay: dev-tool/lsprettysize
Binary: /usr/bin/lsprettysize

That's annoying on Gentoo with complex ebuilds like sys-apps/coreutils, so I usually end up that I'd better create my own tools, especially because they take less time to compile.

Anyway, the point is: even a customized and simplified "ls", I found that there are two ways to show the filesize! 

And the problem is how to avoid to make confusion with the base { 10, 2 } the Eng constants { K, M, G, ... } are referring to.

Above there is a solution, names are a bit silly, but actually they offer a solution.

[kjpye]

On the command line,
    df -h
gives you a summary of your disk usage in powers of ten (kB, MB, GB, TB); for example
...

On the other hand, use an upper-case H,
    df -H
and you get the summary in powers of two (KiB, MiB, GiB, TiB):
...

Are you sure about that? On my system it's the other way around, and the numbers in your second example are larger than in the first example, and GiB and friends should give smaller numbers than GB and co.

[IanB]

What I don't see anyone mention is allocation size.

In all file systems, the space is managed (allocated and freed) in blocks of a fixed size. For example, suppose the block size is 4K, and you create a file containing 100 bytes of data. Then the "file size" will be 100 bytes, but the size on disk will be 4 kilobytes.

It follows that all files occupy more disk space than the actual data they contain.

Take it to the extreme, and create 1000 files of size 1 byte. You will have 1000 bytes of data occupying 1000 x 4K = 4 MB of disk space. Of course, 4 MB is lost in the noise of a 1 TB drive, and so people don't really worry about this very much, especially since real files are much bigger than 1 byte. But it's worth knowing.

When I was in high school, we had a time share service on a PDP/11 at a local college. A typical account shared by a whole computer science class was allocated 100 blocks of 512 bytes. So 100 files of size 1 byte would fill it up. While you were logged in, you could use much more space, but you could not log out until you had deleted files down to the 100 block limit.

I wrote a compress/decompress program that would take all the files in the directory and pack them into one file like zip without compression. So assuming 6.3 file names, those 100 1-byte files could be compressed down to a single file of a little more than 1000 bytes, or 2 disk blocks instead of 100 disk blocks. The class teacher could therefore run compress before logging out, and run decompress when logging in for the next class. Of course, real files had more than a byte, but typical student BASIC programs were very small and had a lot of wasted allocation space. Something like 3:1 compression was not atypical.

[IanB]

per clause 4 of the ISO/IEC 80000-13:2008 standard.  The names are silly ("kibibyte", "mebibyte", "gibibyte", "tebibyte"), but we can be precise and unambiguous if we want, nowadays.

I've got to say, that nobody is forced to adopt published standards. Standards find acceptance if they are useful and solve problems. If they create more problems than they solve, then nobody needs to use them.

In this case, it was (and is) universally accepted in computer science that 1 KB = 1024 bytes, 1 MB = 1024 * 1024 bytes, and so on. The ISO "standard" has created ambiguity by leaving it uncertain whether 1 MB is 1024x1024 bytes, or 1000x1000 bytes. Standards are supposed to remove ambiguity, not create it.

The so-called standard would have been much more useful if it had proposed new nomenclature for the decimal sizes, for example, 1 KeB = 1 000 bytes, 1 MeB = 1 000 000 bytes, and so on.

Some people who are stuck in an insular worldview would be astonished to learn that 1 Mlb = 1000 pounds (not one million pounds, which would be 1 MMlb).

[SiliconWizard]

As Nominal said, I also find the names silly, but that's a standard and see no reason not to adopt it. I have.

Now feel free not to adopt it, as long as you are explicit about your units in your documents/communication. But experience shows that people are not, precisely because they assume that it's all obvious, when obviously, it isn't, and is stil a source of endless confusion as this very thread shows.

I have heard on a regular basis, from people against the standard, that it was all obvious from the context anyway. Like, talk about RAM size, it's obvious KB is KiB, right? Talk about storage size, and it's obvious that KB is 1000 bytes, right? Or maybe not? What about throughput in MB/s? Is it MiB/s or not?

I understand that old habits are hard to beat, and when the new stuff sounds silly, it doesn't help, but really, when units depend on context to have a full meaning, then it's not science or engineering, it's just literature.

[DiTBho]

in my lsprettysize tool, I just added a legenda ( --help ) to tell the user about the base { base10, base2 }.

[IanB]

I have heard on a regular basis, from people against the standard, that it was all obvious from the context anyway. Like, talk about RAM size, it's obvious KB is KiB, right? Talk about storage size, and it's obvious that KB is 1000 bytes, right? Or maybe not? What about throughput in MB/s? Is it MiB/s or not?

When a standard comes along and says that the thing that has always been 1024 bytes is now 1000 bytes, that is not a standard, that is a recipe for chaos.

We can see the truth about the chaos from the endless arguments the attempted standard has produced. A good standard would produce harmony, not disruption.

There was never any disagreement between memory space and disk space either. Reference my note above concerning disk allocation sizes. They were (and still are) in powers of two. From 512 byte blocks, to 1024 byte blocks, to 4096 byte blocks, and so on.

[magic]

What I don't see anyone mention is allocation size.

In all file systems, the space is managed (allocated and freed) in blocks of a fixed size. For example, suppose the block size is 4K, and you create a file containing 100 bytes of data. Then the "file size" will be 100 bytes, but the size on disk will be 4 kilobytes.

Because it (hopefully) isn't the source of OP's confusion, but you have now added some more

[Nominal Animal]

On the command line,
    df -h
gives you a summary of your disk usage in powers of ten (kB, MB, GB, TB); for example
...

On the other hand, use an upper-case H,
    df -H
and you get the summary in powers of two (KiB, MiB, GiB, TiB):
...

Are you sure about that?

No, I got them swapped. 

See man 1 df for details; -h is shorthand for --human-readable, and means powers-of-two scale, and -H is shorthand for --si, meaning powers-of-ten scale.

Will edit that post; thanks.

[Circlotron]

Because it (hopefully) isn't the source of OP's confusion,

This is the source of the confusion, the difference between these two figures.

[tunk]

What does df say, and what linux distro?
Do you have the same problem with the other disks?
Just speculation, it could be a bug in the software.

[Nominal Animal]

Because it (hopefully) isn't the source of OP's confusion,

This is the source of the confusion, the difference between these two figures.

You have 130.5GB in the root directory of the ONE_TB volume, and a total of 586.3GB overall in that volume.

Let's say you have ONE_TB mounted in the typical location in Linux,
    /media/circlotron/ONE_TB
and in it,
    /media/circlotron/ONE_TB/movie1.mkv
    :
    /media/circlotron/ONE_TB/movieN.mkv
that total to 130.5GB.  Then, you have a subdirectory in it,
    /media/circlotron/ONE_TB/not-porn/movie1.mkv
    :
    /media/circlotron/ONE_TB/not-porn/movieN.mkv
that total to 455.8GB.

Many Linux file managers only show you the folder statistics when you right-click and select properties.  The summary is then of only that folder, not including any subfolders.  Above, you have 586.3GB of stuff on a volume named ONE_TB, about 130.5GB in the topmost directory, the rest, about 455.8GB, somewhere in subdirectories.

Me, I use Nemo right now, and that starts scanning the subdirectories interactively when you look at the properties of a folder, updating the total in real time as you wait.  I don't know which file manager or desktop environment Circlotron is using, and there are several that can be configured to look like that, so it could also be a funky bug in one instead.

[magic]

XFCE counts subdirectories too, and it takes quite a while on large disks for obvious reason.

One thing that could still cause such recursive count to be wrong is lacking permissions to access some subdirectories.

OTOH, I don't think the discrepancy can be completely (or even mostly) explained by the aforementioned issue of 4KB block allocation, because it would take a huge lot of small files to waste so much space in this manner.


At the end of the day, I would trust df to accurately report available space and du to accurately report overall disk usage of a directory (provided that it doesn't report failures due to lack of permissions). Meanwhile with those GUI tools you always have to guess what they are doing and what they might be doing wrong...

[DiTBho]

my "lsprettysize" can be used like "du", but in this case it doesn't sum the actual size of the file but rather it sums how many blocks each file uses, so also including metadata blocks!

This way, by iterating hierarchically from a sub-root, and adding up how many blocks are used in total, you get a better idea of how much real space you have consumed!

[madires]

Another thing to know it that linux usually reserves 5% of a partition for root.