ZFS and disk usage

[Mark Schouten]

Hi,

I'm having some issues with a FreeBSD box using ZFS to serve iscsi to other boxes.

[root@storage ~]# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
storage 1.77T 431G 1.34T 23% ONLINE -

As you can see, the zpool is at only 23% of it's capacity. However, if you get a list of filesystems with "zfs list", you see that there is only 138GB free space left.
[root@storage ~]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
storage 1.60T 138G 431G /storage
storage/ZFS_FS_1 20G 158G 16K -
storage/ZFS_FS_2 20G 158G 16K -
storage/ZFS_FS_3 100G 238G 16K -
storage/ZFS_FS_4 20G 158G 16K -
storage/ZFS_FS_5 1G 139G 16K -
storage/ZFS_FS_6 400G 538G 16K -
storage/ZFS_FS_7 20G 158G 16K -
storage/ZFS_FS_8 400G 538G 16K -
storage/ZFS_FS_9 20G 158G 16K -
storage/ZFS_FS_10 20G 158G 16K -
storage/ZFS_FS_11 20G 158G 16K -
storage/ZFS_FS_12 150G 288G 16K -
storage/ZFS_FS_13 20G 158G 16K -

These are fiesystems that are created with the following command.
zfs create -V ${size}GB ${ZFS_ROOT}/${diskname}

Now, it seems that zpool only counts the data that is actually written on the disk, and that zfs counts both the sum of the individual filesystems *and* the data actually written on disk. If I was to create a new filesystem of 138GB, the filesystem would be full, eventhough that that's not the case.

This seems weird, but I'm not sure if it's me doing something wrong, or if its a bug. Please enlighten me, thanks.



Some more info:
[root@storage ~]# uname -a
FreeBSD storage.storage 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Thu Feb 17 02:41:51 UTC 2011 ro...@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64

[root@storage ~]# zpool get all storage
NAME PROPERTY VALUE SOURCE
storage size 1.77T -
storage used 431G -
storage available 1.34T -
storage capacity 23% -
storage altroot - default
storage health ONLINE -
storage guid 10905194744545589060 default
storage version 15 default
storage bootfs - default
storage delegation on default
storage autoreplace off default
storage cachefile - default
storage failmode wait default
storage listsnapshots off default

[root@storage ~]# zfs get all storage
NAME PROPERTY VALUE SOURCE
storage type filesystem -
storage creation Tue May 10 11:59 2011 -
storage used 1.60T -
storage available 138G -
storage referenced 431G -
storage compressratio 1.00x -
storage mounted yes -
storage quota none default
storage reservation none default
storage recordsize 128K default
storage mountpoint /storage default
storage sharenfs off default
storage checksum on default
storage compression off default
storage atime on default
storage devices on default
storage exec on default
storage setuid on default
storage readonly off default
storage jailed off default
storage snapdir hidden default
storage aclmode groupmask default
storage aclinherit restricted default
storage canmount on default
storage shareiscsi off default
storage xattr off temporary
storage copies 1 default
storage version 4 -
storage utf8only off -
storage normalization none -
storage casesensitivity sensitive -
storage vscan off default
storage nbmand off default
storage sharesmb off default
storage refquota none default
storage refreservation none default
storage primarycache all default
storage secondarycache all default
storage usedbysnapshots 0 -
storage usedbydataset 431G -
storage usedbychildren 1.18T -
storage usedbyrefreservation 0 -

--
Dit bericht is verzonden via https://www.hyperdesktop.nl/. Alles, overal!

Mark Schouten | Tuxis Internet Engineering
KvK: 09218193 | http://www.tuxis.nl/
T: 0318 200208 | in...@tuxis.nl
M: 06 53463918 | ma...@tuxis.nl
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"

[Volodymyr Kostyrko]

`zfs create -V` withous `-s` creates reserved volume that eats all
needed space immediately. Technically zfs pool is filled only for 23%,
but logically you have only 138G left unassigned.

--
Sphinx of black quartz judge my vow.

[Mark Schouten]

Hi,

Op Vrijdag, 13-04-2012 om 15:11 schreef Volodymyr Kostyrko:

> > These are fiesystems that are created with the following command.
> > zfs create -V ${size}GB ${ZFS_ROOT}/${diskname}
>
> `zfs create -V` withous `-s` creates reserved volume that eats all
> needed space immediately. Technically zfs pool is filled only for 23%,
> but logically you have only 138G left unassigned.

I understand. However, the created volumes should use a total of 1211GB. That's not 1.6TB like zfs list says. But 1211 + 431 (referred) does come close to 1.6TB.n And 1.6 TB still isn't the 1.77TB that's in the zpool.

I have this feeling that zfs has reserved the space for each volume, but counts data written to the volumes in usage of the main filesystem. Mainly because zfs list shows me that the volumes have only 16KB referenced, where /storage has 431GB referenced.

--
Dit bericht is verzonden via https://www.hyperdesktop.nl/. Alles, overal!

Mark Schouten | Tuxis Internet Engineering
KvK: 09218193 | http://www.tuxis.nl/
T: 0318 200208 | in...@tuxis.nl
M: 06 53463918 | ma...@tuxis.nl

[Peter Maloney]

Please run this (my script I call zfsdf):

zfs list -o
name,used,referenced,usedbychildren,usedbydataset,usedbysnapshots,available,mountpoint,quota,reserv,refquota,refreserv
"$@" | sed -r "s/none/ -/g"

zpool list -o name,size,allocated,free,capacity

It may give some small hints (similar to zfs get all <poolname> but for
all datasets, telling about refquota, snap used, etc.),

but...

I think it won't tell us enough, and the problem is 8.2-RELEASE. You
should definitely upgrade to 8-STABLE, or 8.3-rc2. [with regression
testing of course, but not as much as you would need for 9.x] [btw, I
found that 8-STABLE in Sept 2011 would hang renaming snapshots with
ZVOLS, but not 8-STABLE from Feb 2012, so be extra careful with zvols]


On my 8-STABLE systems, the numbers make sense:

semi-new system:

# zfs list
NAME USED AVAIL REFER MOUNTPOINT

tank 37.7T 10.2T 67.5K /tank

# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
tank 63.2T 49.0T 14.3T 77% 1.00x ONLINE -

# bc
scale=5
37.7/(10.2+37.7)
.78705


year old system (scripts create and destroy 1 recursive snapshot every
20 minutes):

# zfs list
NAME USED AVAIL REFER MOUNTPOINT

tank 14.5T 17.5T 5.99G /tank

# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
tank 43.5T 19.4T 24.1T 44% 1.00x ONLINE -

# bc
scale=5
14.5/(14.5+17.5)
.45312

On 04/13/2012 02:21 PM, Mark Schouten wrote:
> Hi,
>
> I'm having some issues with a FreeBSD box using ZFS to serve iscsi to other boxes.
>
> [root@storage ~]# zpool list
> NAME SIZE USED AVAIL CAP HEALTH ALTROOT
> storage 1.77T 431G 1.34T 23% ONLINE -
>
> As you can see, the zpool is at only 23% of it's capacity. However, if you get a list of filesystems with "zfs list", you see that there is only 138GB free space left.
> [root@storage ~]# zfs list
> NAME USED AVAIL REFER MOUNTPOINT
> storage 1.60T 138G 431G /storage
> storage/ZFS_FS_1 20G 158G 16K -
> storage/ZFS_FS_2 20G 158G 16K -
> storage/ZFS_FS_3 100G 238G 16K -
> storage/ZFS_FS_4 20G 158G 16K -
> storage/ZFS_FS_5 1G 139G 16K -
> storage/ZFS_FS_6 400G 538G 16K -
> storage/ZFS_FS_7 20G 158G 16K -
> storage/ZFS_FS_8 400G 538G 16K -
> storage/ZFS_FS_9 20G 158G 16K -
> storage/ZFS_FS_10 20G 158G 16K -
> storage/ZFS_FS_11 20G 158G 16K -
> storage/ZFS_FS_12 150G 288G 16K -
> storage/ZFS_FS_13 20G 158G 16K -
>

> These are fiesystems that are created with the following command.
> zfs create -V ${size}GB ${ZFS_ROOT}/${diskname}
>

--------------------------------------------
Peter Maloney
Brockmann Consult
Max-Planck-Str. 2
21502 Geesthacht
Germany
Tel: +49 4152 889 300
Fax: +49 4152 889 333
E-mail: peter....@brockmann-consult.de
Internet: http://www.brockmann-consult.de
--------------------------------------------

[Johannes Totz]

On 13/04/2012 14:28, Mark Schouten wrote:
> Hi,
>
> Op Vrijdag, 13-04-2012 om 15:11 schreef Volodymyr Kostyrko:

>>> These are fiesystems that are created with the following
>>> command. zfs create -V ${size}GB ${ZFS_ROOT}/${diskname}
>>

>> `zfs create -V` withous `-s` creates reserved volume that eats all
>> needed space immediately. Technically zfs pool is filled only for
>> 23%, but logically you have only 138G left unassigned.
>
> I understand. However, the created volumes should use a total of
> 1211GB. That's not 1.6TB like zfs list says. But 1211 + 431
> (referred) does come close to 1.6TB.n And 1.6 TB still isn't the
> 1.77TB that's in the zpool.
>
> I have this feeling that zfs has reserved the space for each volume,
> but counts data written to the volumes in usage of the main
> filesystem. Mainly because zfs list shows me that the volumes have
> only 16KB referenced, where /storage has 431GB referenced.

Without checking the numbers myself...
Note that zpool and zfs do not agree on (free) space accounting: zpool
shows "raw" space, whereas zfs includes metadata overhead for itself.

Small rant: I dont understand why zpool and zfs show different things.
If you have an integrated storage stack then why not show consistent
numbers? Is there any use for this extra (mis-)information that
zpool-vs-zfs provides?

[Tom Evans]

On Fri, Apr 13, 2012 at 4:27 PM, Johannes Totz <joha...@jo-t.de> wrote:
> Small rant: I dont understand why zpool and zfs show different things.
> If you have an integrated storage stack then why not show consistent
> numbers? Is there any use for this extra (mis-)information that
> zpool-vs-zfs provides?
>

They are supposed to be different things. `zpool list` shows the size
of the raw storage in the pool, where as `zfs list` shows the size of
file systems on that storage.

zpool deals with raw storage blocks, why would it talk about the size
of the overlying filesystem?

Cheers

Tom

[Freddie Cash]

On Fri, Apr 13, 2012 at 8:27 AM, Johannes Totz <joha...@jo-t.de> wrote:
> Without checking the numbers myself...
> Note that zpool and zfs do not agree on (free) space accounting: zpool
> shows "raw" space, whereas zfs includes metadata overhead for itself.
>

> Small rant: I dont understand why zpool and zfs show different things.
> If you have an integrated storage stack then why not show consistent
> numbers? Is there any use for this extra (mis-)information that
> zpool-vs-zfs provides?

There's a great posting about the differences in the zfs-discuss
mailing list archives, although I can't find a reference to it at the
moment. Going from memory, the breakdown is something like:

zpool shows "raw storage available to the pool across all vdevs",
without counting any redundancy. This should be approx. "size of
drives * num of drives".

zfs shows "storage space available for use", after removing all
redundancy, extra space for metadata, checksums, etc. This is what's
available for programs to use, before compression and dedupe take
effect.

df shows "storage space available to userspace programs" after all
compressions, dedupe, metadata, checksums, etc have been removed.
This is the actual space that users can access.

"ls -l" shows the "size of files" (as in, uncompressed, rehydrated,
the size it would be if you copied it to a floppy).

They each work on different layers of the storage stack (DMU, ZPL,
userspace, etc). Hence, they show different values. But once you
think about what each layer of the stack is doing ... the numbers make
perfect sense.

--
Freddie Cash
fjw...@gmail.com

[Freddie Cash]

On Fri, Apr 13, 2012 at 8:45 AM, Freddie Cash <fjw...@gmail.com> wrote:
> On Fri, Apr 13, 2012 at 8:27 AM, Johannes Totz <joha...@jo-t.de> wrote:
>> Without checking the numbers myself...
>> Note that zpool and zfs do not agree on (free) space accounting: zpool
>> shows "raw" space, whereas zfs includes metadata overhead for itself.
>>
>> Small rant: I dont understand why zpool and zfs show different things.
>> If you have an integrated storage stack then why not show consistent
>> numbers? Is there any use for this extra (mis-)information that
>> zpool-vs-zfs provides?
>
> There's a great posting about the differences in the zfs-discuss
> mailing list archives, although I can't find a reference to it at the
> moment.  Going from memory, the breakdown is something like:

Here's one of them:
http://mail.opensolaris.org/pipermail/zfs-discuss/2010-April/040180.html

Message details:
On Sun, Apr 18, 2010 at 20:08, Harry Putnam <reader at newsguy.com> wrote:
> Seems like you can get some pretty large discrepancies in sizes of
> pools. and directories.
They all answer different things, sure, but they're all things that an
administrator might want to know.

> zpool list
"How many bytes are in use on the storage device? How many
unallocated bytes are there?"

> zfs list
"If I have to ship this filesystem to another box (uncompressed and
not deduped) how many bytes is that?"

> du
"How many bytes are used to store the contents of the files in this directory?"

and "ls -l":
"How many bytes are addressable in this file?"

> Do no other administrators feel the
> need to know accurate sizes?
It's important to consider what you want this data for. Considering
upgrading your storage to get more room? Check out "zpool list".
Need to know whether accounting or engineering is using more space?
Look at "zfs list". Looking at a sparse or compressed file, and want
to know how many bytes are allocated to it? "du" does the trick.
Planning to email someone a file, and want to know if it'll fit in
their 10MB quota? "ls -l" is the relevant command.

In short, there are many commands because there are many answers, and
many questions. No single tool has all the information available to
it.

Will

[Volodymyr Kostyrko]

Mark Schouten wrote:
> Hi,
>
> Op Vrijdag, 13-04-2012 om 15:11 schreef Volodymyr Kostyrko:
>>> These are fiesystems that are created with the following command.
>>> zfs create -V ${size}GB ${ZFS_ROOT}/${diskname}
>>
>> `zfs create -V` withous `-s` creates reserved volume that eats all
>> needed space immediately. Technically zfs pool is filled only for 23%,
>> but logically you have only 138G left unassigned.
>
> I understand. However, the created volumes should use a total of 1211GB. That's not 1.6TB like zfs list says. But 1211 + 431 (referred) does come close to 1.6TB.n And 1.6 TB still isn't the 1.77TB that's in the zpool.

After reinventing calculator I'm just puzzled and shamed. This looks
strange.

Indeed not just a single byte from this files should count directly for
the main filesystem.

Just curious, what `ls -la /storage/' and `ls -la /storage/.zfs/s*` show?

>
> I have this feeling that zfs has reserved the space for each volume, but counts data written to the volumes in usage of the main filesystem. Mainly because zfs list shows me that the volumes have only 16KB referenced, where /storage has 431GB referenced.
>

--
Sphinx of black quartz judge my vow.

[Ronald Klop]

On Fri, 13 Apr 2012 17:27:10 +0200, Johannes Totz <joha...@jo-t.de> wrote:

> On 13/04/2012 14:28, Mark Schouten wrote:
>> Hi,
>>
>> Op Vrijdag, 13-04-2012 om 15:11 schreef Volodymyr Kostyrko:
>>>> These are fiesystems that are created with the following
>>>> command. zfs create -V ${size}GB ${ZFS_ROOT}/${diskname}
>>>
>>> `zfs create -V` withous `-s` creates reserved volume that eats all
>>> needed space immediately. Technically zfs pool is filled only for
>>> 23%, but logically you have only 138G left unassigned.
>>
>> I understand. However, the created volumes should use a total of
>> 1211GB. That's not 1.6TB like zfs list says. But 1211 + 431
>> (referred) does come close to 1.6TB.n And 1.6 TB still isn't the
>> 1.77TB that's in the zpool.
>>

>> I have this feeling that zfs has reserved the space for each volume,
>> but counts data written to the volumes in usage of the main
>> filesystem. Mainly because zfs list shows me that the volumes have
>> only 16KB referenced, where /storage has 431GB referenced.
>

> Without checking the numbers myself...
> Note that zpool and zfs do not agree on (free) space accounting: zpool
> shows "raw" space, whereas zfs includes metadata overhead for itself.
>
> Small rant: I dont understand why zpool and zfs show different things.
> If you have an integrated storage stack then why not show consistent
> numbers? Is there any use for this extra (mis-)information that
> zpool-vs-zfs provides?
>

It isn't integrated that far as you might think. That is why you have
zpool and zfs tools. These are 2 separate things. You can also put a zvol
+ ufs or zvol + iscsi on the zpool and not use zfs. How would zpool know
about ufs or iscsi usage? Or how would ufs know about the underlying
zpool+zvol?

Ronald.

[Mark Schouten]

Hi,

Op Vrijdag, 13-04-2012 om 18:32 schreef Volodymyr Kostyrko:

> After reinventing calculator I'm just puzzled and shamed. This looks
> strange.
>
> Indeed not just a single byte from this files should count directly for
> the main filesystem.
>
> Just curious, what `ls -la /storage/' and `ls -la /storage/.zfs/s*` show?

[root@storage ~]# zfs list -o name,used,referenced,usedbychildren,usedbydataset,usedbysnapshots,available,mountpoint,quota,reserv,refquota,refreserv "$@" | sed -r "s/none/ -/g"
NAME USED REFER USEDCHILD USEDDS USEDSNAP AVAIL MOUNTPOINT QUOTA RESERV REFQUOTA REFRESERV
storage 1.60T 431G 1.18T 431G 0 138G /storage - - - -
storage/av.https-ict.nl.OS 20G 16K 0 16K 0 158G - - - - 20G
storage/bestandonline.https-ict.nl.OS 20G 16K 0 16K 0 158G - - - - 20G
storage/crm.fastbenelux.DATA 100G 16K 0 16K 0 238G - - - - 100G
storage/crm.fastbenelux.OS 20G 16K 0 16K 0 158G - - - - 20G
storage/default.do.not.remove 1G 16K 0 16K 0 139G - - - - 1G
storage/mail.https-ict.nl.DATA 400G 16K 0 16K 0 538G - - - - 400G
storage/mail.https-ict.nl.OS 20G 16K 0 16K 0 158G - - - - 20G
storage/webserver.https-ict.nl.DATA 400G 16K 0 16K 0 538G - - - - 400G
storage/webserver.https-ict.nl.OS 20G 16K 0 16K 0 158G - - - - 20G
storage/webserver2.https-ict.nl.DATA 20G 16K 0 16K 0 158G - - - - 20G
storage/webserver2.https-ict.nl.OS 20G 16K 0 16K 0 158G - - - - 20G
storage/xlmail.xladsl.nl.DATA 150G 16K 0 16K 0 288G - - - - 150G
storage/xlmail.xladsl.nl.os 20G 16K 0 16K 0 158G - - - - 20G

[root@storage /storage]# ls -al
total 452101743
drwxr-xr-x 2 root wheel 18 Mar 24 17:42 .
drwxr-xr-x 20 root wheel 512 May 10 2011 ..
-rw-r--r-- 1 root wheel 21474836480 Apr 16 12:40 av.https-ict.nl.OS
-rw-r--r-- 1 root wheel 155692564480 Mar 24 17:42 bestandonline.https-ict.nl.DATA
-rw-r--r-- 1 root wheel 21474836480 Apr 5 12:19 bestandonline.https-ict.nl.OS
-rw-r--r-- 1 root wheel 107374182400 Apr 16 12:40 crm.fastbenelux.DATA
-rw-r--r-- 1 root wheel 21474836480 Apr 16 12:39 crm.fastbenelux.OS
-rw-r--r-- 1 root wheel 1073741824 Apr 5 12:19 default.do.not.remove
-rw-r--r-- 1 root wheel 10737418240 May 12 2011 documentatievm.https-ict.nl
-rw-r--r-- 1 root wheel 429496729600 Apr 15 02:07 mail.https-ict.nl.DATA
-rw-r--r-- 1 root wheel 21474836480 Apr 16 12:40 mail.https-ict.nl.OS
-rw-r--r-- 1 root wheel 16106127360 May 16 2011 testvm1.https-ict.nl
-rw-r--r-- 1 root wheel 429496729600 Apr 16 12:40 webserver.https-ict.nl.DATA
-rw-r--r-- 1 root wheel 21474836480 Apr 16 12:40 webserver.https-ict.nl.OS
-rw-r--r-- 1 root wheel 21474836480 Apr 5 12:19 webserver2.https-ict.nl.DATA
-rw-r--r-- 1 root wheel 21474836480 Apr 16 12:40 webserver2.https-ict.nl.OS
-rw-r--r-- 1 root wheel 161061273600 Apr 16 12:40 xlmail.xladsl.nl.DATA
-rw-r--r-- 1 root wheel 21474836480 Apr 16 12:40 xlmail.xladsl.nl.os

[root@storage /storage]# ls -la /storage/.zfs/s*
/storage/.zfs/shares:
total 2
dr-xr-xr-x 2 root wheel 2 May 10 2011 .
dr-xr-xr-x 4 root wheel 4 May 10 2011 ..

/storage/.zfs/snapshot:
total 0
dr-xr-xr-x 2 root wheel 2 May 10 2011 .
dr-xr-xr-x 4 root wheel 4 May 10 2011 ..


It definitly looks like the reserved space for the zvols is counted in /storage, but the written data is counted for /storage as well, instead of the zvols..

Will an upgrade fix this?

--
Dit bericht is verzonden via https://www.hyperdesktop.nl/. Alles, overal!

Mark Schouten | Tuxis Internet Engineering
KvK: 09218193 | http://www.tuxis.nl/
T: 0318 200208 | in...@tuxis.nl
M: 06 53463918 | ma...@tuxis.nl

[Volodymyr Kostyrko]

Mark Schouten wrote:
>> Just curious, what `ls -la /storage/' and `ls -la /storage/.zfs/s*` show?

It's plain weird.

>
> [root@storage ~]# zfs list -o name,used,referenced,usedbychildren,usedbydataset,usedbysnapshots,available,mountpoint,quota,reserv,refquota,refreserv "$@" | sed -r "s/none/ -/g"
> NAME USED REFER USEDCHILD USEDDS USEDSNAP AVAIL MOUNTPOINT QUOTA RESERV REFQUOTA REFRESERV
> storage 1.60T 431G 1.18T 431G 0 138G /storage - - - -

> storage/av.https-ict.nl..OS 20G 16K 0 16K 0 158G - - - - 20G

> storage/bestandonline.https-ict.nl.OS 20G 16K 0 16K 0 158G - - - - 20G

Let's take this one for example. It's not mounted! It shouldn't be
listed in next command output.

> storage/crm.fastbenelux.DATA 100G 16K 0 16K 0 238G - - - - 100G
> storage/crm.fastbenelux.OS 20G 16K 0 16K 0 158G - - - - 20G
> storage/default.do.not.remove 1G 16K 0 16K 0 139G - - - - 1G
> storage/mail.https-ict.nl.DATA 400G 16K 0 16K 0 538G - - - - 400G
> storage/mail.https-ict.nl.OS 20G 16K 0 16K 0 158G - - - - 20G
> storage/webserver.https-ict.nl.DATA 400G 16K 0 16K 0 538G - - - - 400G
> storage/webserver.https-ict.nl.OS 20G 16K 0 16K 0 158G - - - - 20G
> storage/webserver2.https-ict.nl.DATA 20G 16K 0 16K 0 158G - - - - 20G
> storage/webserver2.https-ict.nl.OS 20G 16K 0 16K 0 158G - - - - 20G
> storage/xlmail.xladsl.nl.DATA 150G 16K 0 16K 0 288G - - - - 150G
> storage/xlmail.xladsl.nl.os 20G 16K 0 16K 0 158G - - - - 20G
>
> [root@storage /storage]# ls -al
> total 452101743
> drwxr-xr-x 2 root wheel 18 Mar 24 17:42 .
> drwxr-xr-x 20 root wheel 512 May 10 2011 ..
> -rw-r--r-- 1 root wheel 21474836480 Apr 16 12:40 av.https-ict.nl.OS
> -rw-r--r-- 1 root wheel 155692564480 Mar 24 17:42 bestandonline.https-ict.nl.DATA
> -rw-r--r-- 1 root wheel 21474836480 Apr 5 12:19 bestandonline.https-ict.nl.OS

This is not a zfs dataset. This is a file. It has no relevance to named
dataset. And it drains space from /storage.

> -rw-r--r-- 1 root wheel 107374182400 Apr 16 12:40 crm.fastbenelux.DATA
> -rw-r--r-- 1 root wheel 21474836480 Apr 16 12:39 crm.fastbenelux.OS
> -rw-r--r-- 1 root wheel 1073741824 Apr 5 12:19 default.do.not.remove
> -rw-r--r-- 1 root wheel 10737418240 May 12 2011 documentatievm.https-ict.nl
> -rw-r--r-- 1 root wheel 429496729600 Apr 15 02:07 mail.https-ict.nl.DATA
> -rw-r--r-- 1 root wheel 21474836480 Apr 16 12:40 mail.https-ict.nl.OS
> -rw-r--r-- 1 root wheel 16106127360 May 16 2011 testvm1.https-ict.nl
> -rw-r--r-- 1 root wheel 429496729600 Apr 16 12:40 webserver.https-ict.nl.DATA

> -rw-r--r-- 1 root wheel 21474836480 Apr 16 12:40 webserver..https-ict.nl.OS

> -rw-r--r-- 1 root wheel 21474836480 Apr 5 12:19 webserver2.https-ict.nl.DATA
> -rw-r--r-- 1 root wheel 21474836480 Apr 16 12:40 webserver2.https-ict.nl.OS
> -rw-r--r-- 1 root wheel 161061273600 Apr 16 12:40 xlmail.xladsl.nl.DATA
> -rw-r--r-- 1 root wheel 21474836480 Apr 16 12:40 xlmail.xladsl.nl.os
>
> [root@storage /storage]# ls -la /storage/.zfs/s*
> /storage/.zfs/shares:
> total 2
> dr-xr-xr-x 2 root wheel 2 May 10 2011 .
> dr-xr-xr-x 4 root wheel 4 May 10 2011 ..
>
> /storage/.zfs/snapshot:
> total 0
> dr-xr-xr-x 2 root wheel 2 May 10 2011 .
> dr-xr-xr-x 4 root wheel 4 May 10 2011 ..

There's also another discrepancies in your output like:
* "av.https-ict.nl..OS" != "av.https-ict.nl.OS", seems like
copy-and-paste error(?);
* "bestandonline.https-ict.nl.DATA" and "documentatievm.https-ict.nl"
doesn't exist in datasets, it looks like correspondent datasets were
already removed, but not files.

> It definitly looks like the reserved space for the zvols is counted in /storage, but the written data is counted for /storage as well, instead of the zvols..
>
> Will an upgrade fix this?
>

--

Sphinx of black quartz judge my vow.

[Mark Schouten]

Hi,

Op Dinsdag, 17-04-2012 om 8:32 schreef Volodymyr Kostyrko:

> > [root@storage ~]# zfs list -o name,used,referenced,usedbychildren,usedbydataset,usedbysnapshots,available,mountpoint,quota,reserv,refquota,refreserv "$@" | sed -r "s/none/ -/g"
> > NAME USED REFER USEDCHILD USEDDS USEDSNAP AVAIL MOUNTPOINT QUOTA RESERV REFQUOTA REFRESERV
> > storage 1.60T 431G 1.18T 431G 0 138G /storage - - - -
> > storage/av.https-ict.nl..OS 20G 16K 0 16K 0 158G - - - - 20G
> > storage/bestandonline.https-ict.nl.OS 20G 16K 0 16K 0 158G - - - - 20G
>
> Let's take this one for example. It's not mounted! It shouldn't be
> listed in next command output.

They are zvols, they shouldn't be mounted. They're exported by iscsitgt as devices.

> This is not a zfs dataset. This is a file. It has no relevance to named
> dataset. And it drains space from /storage.

That's absolutely true. There are indeed three files that could be removed.
bestandonline.https-ict.nl.DATA
documentatievm.https-ict.nl
testvm1.https-ict.nl

But still, there is only 16K referenced by the zvols...

--
Dit bericht is verzonden via https://www.hyperdesktop.nl/. Alles, overal!

Mark Schouten | Tuxis Internet Engineering
KvK: 09218193 | http://www.tuxis.nl/
T: 0318 200208 | in...@tuxis.nl
M: 06 53463918 | ma...@tuxis.nl

[Volodymyr Kostyrko]

Mark Schouten wrote:
>>> [root@storage ~]# zfs list -o name,used,referenced,usedbychildren,usedbydataset,usedbysnapshots,available,mountpoint,quota,reserv,refquota,refreserv "$@" | sed -r "s/none/ -/g"
>>> NAME USED REFER USEDCHILD USEDDS USEDSNAP AVAIL MOUNTPOINT QUOTA RESERV REFQUOTA REFRESERV
>>> storage 1.60T 431G 1.18T 431G 0 138G /storage - - - -
>>> storage/av.https-ict.nl..OS 20G 16K 0 16K 0 158G - - - - 20G
>>> storage/bestandonline.https-ict.nl.OS 20G 16K 0 16K 0 158G - - - - 20G
>>
>> Let's take this one for example. It's not mounted! It shouldn't be
>> listed in next command output.
>
> They are zvols, they shouldn't be mounted. They're exported by iscsitgt as devices.

As they are zvols they do exists at /dev/zvol/storage/ and not
/storage/. Where did all that files came from?

Can you post your iscsitgt setup?

>
>> This is not a zfs dataset. This is a file. It has no relevance to named
>> dataset. And it drains space from /storage.
>
> That's absolutely true. There are indeed three files that could be removed.
> bestandonline.https-ict.nl.DATA
> documentatievm.https-ict.nl
> testvm1.https-ict.nl
>
> But still, there is only 16K referenced by the zvols...
>

--

Sphinx of black quartz judge my vow.

[Mark Schouten]

Hi,

Op Dinsdag, 17-04-2012 om 11:55 schreef Volodymyr Kostyrko:

> As they are zvols they do exists at /dev/zvol/storage/ and not
> /storage/. Where did all that files came from?
>
> Can you post your iscsitgt setup?

Ok, so now I feel like an ass. :) I followed this [1] link to setup this box. But I understand that I'm doing that all wrong?

istgt.conf has the lun's defined like this:
LUN0 Storage /storage/default.do.not.remove 1GB
LUN1 Storage /storage/mail.https-ict.nl.OS 20GB
LUN2 Storage /storage/mail.https-ict.nl.DATA 400GB

That should point to the /dev/zvol/storage/ devices?

[1]: http://www.techjolts.com/2011/01/14/iscsi-target-with-zfs-freebsd/

--
Dit bericht is verzonden via https://www.hyperdesktop.nl/. Alles, overal!

Mark Schouten | Tuxis Internet Engineering
KvK: 09218193 | http://www.tuxis.nl/
T: 0318 200208 | in...@tuxis.nl
M: 06 53463918 | ma...@tuxis.nl

[Volodymyr Kostyrko]

Mark Schouten wrote:
> Hi,
>
> Op Dinsdag, 17-04-2012 om 11:55 schreef Volodymyr Kostyrko:
>> As they are zvols they do exists at /dev/zvol/storage/ and not
>> /storage/. Where did all that files came from?
>>
>> Can you post your iscsitgt setup?
>
> Ok, so now I feel like an ass. :) I followed this [1] link to setup this box. But I understand that I'm doing that all wrong?
>
> istgt.conf has the lun's defined like this:
> LUN0 Storage /storage/default.do.not.remove 1GB
> LUN1 Storage /storage/mail.https-ict.nl.OS 20GB
> LUN2 Storage /storage/mail.https-ict.nl.DATA 400GB
>
> That should point to the /dev/zvol/storage/ devices?
>
> [1]: http://www.techjolts.com/2011/01/14/iscsi-target-with-zfs-freebsd/

As I never worked with istgt myself I can't really say, but yes this
somehow feels wrong to me.

As all your data could be potentially stored in raw files let's test it
another way. Could you create another zfs volume/istgt target and point
it at /dev/zvol/storage/new_volume?

1. Would access to this new device work?
2. Would some new file be created?
3. Would some data be accounted to new_volume?

--
Sphinx of black quartz judge my vow.

[Mark Schouten]

Hi,

Op Dinsdag, 17-04-2012 om 12:11 schreef Volodymyr Kostyrko:

> As I never worked with istgt myself I can't really say, but yes this
> somehow feels wrong to me.

Can someone confirm that I'm stupid? :P

> As all your data could be potentially stored in raw files let's test it
> another way. Could you create another zfs volume/istgt target and point
> it at /dev/zvol/storage/new_volume?
>
> 1. Would access to this new device work?
> 2. Would some new file be created?
> 3. Would some data be accounted to new_volume?

The box is running a production environment now, so I can't really fool around with it. It will be migrated soon though, which will give me some time to experiment.

Thanks a bunch for the help!

--
Dit bericht is verzonden via https://www.hyperdesktop.nl/. Alles, overal!

Mark Schouten | Tuxis Internet Engineering
KvK: 09218193 | http://www.tuxis.nl/
T: 0318 200208 | in...@tuxis.nl
M: 06 53463918 | ma...@tuxis.nl