ZFS - Abyssal slow on copying

[O. Hartmann]

Running 12-CURRENT (FreeBSD 12.0-CURRENT #32 r306579: Sun Oct 2 09:34:50 CEST 2016 ), I
have a NanoBSD setup which creates an image for a router device.

The problem I face is related to ZFS. The system has a system's SSD (Samsung 850 Pro,
256GB) which has an UFS filesystem. Aditionally, I have also a backup and a data HDD,
both WD, one 3 TB WD RED Pro, on 4 TB WD RED (the backup device). Both the sources for
the NanoBSD and the object tree as well as the NANO_WORLDDIR are residing on the 3 TB
data drive.

The box itself has 8 GB RAM. When it comes to create the memory disk, which is ~ 1,3 GB
in size, the NanoBSD script starts creating the memory disk and then installing world
into this memory disk. And this part is a kind of abyssal in terms of the speed.

The drive sounds like hell, the heads are moving rapidly. The copy speed is incredibly
slow compared to another box I usually use in the lab with UFS filesystem only (different
type of HDD).

The whole stuff the nanbsd is installed from and to is on a separate ZFS partition, but
in the same pool as everything else. When I first setup the new partitions, I switched on
deduplication, but I quickly deactivated it, because it had a tremendous impact on the
working speed and memory consumption on that box. But something seems not right since
then - as I initially described, the copy/initialisation speed/bandwith is abyssal. Well,
I also fear that I did something wrong when I firt initialised the HDD - there is this
125bytes/4k block discussion and I do not know how to check whether I'm affected to that
or not (or even causing the problems) and how to check whether DEDUPLICATION is
definitely OFF (apart from the usual stuff list features via "zfs get all").

As an example: the nanbosd script takes ~ 1 minute to copy /boot/loader from source to
memory disk and the HDD makes sounds like hell and close to loosing the r/w heads. On
other boxes this task is done in a blink of an eye ...

Thanks for your patience,

Regards,
oh

[Allan Jude]

Turning deduplication off, only stops new blocks from being
deduplicated. Any data written while deduplication was on, are still
deduplicated. You would need to zfs send | zfs recv, or
backup/destroy/restore to get the data back to normal.

If the drive is making that much noise, have you considered that the
drive might be failing?

--
Allan Jude
_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

[O. Hartmann]

Am Sun, 2 Oct 2016 15:30:41 -0400
Allan Jude <alla...@freebsd.org> schrieb:

Hello Jude.

Thank you for your answer.

> Turning deduplication off, only stops new blocks from being
> deduplicated. Any data written while deduplication was on, are still
> deduplicated. You would need to zfs send | zfs recv, or
> backup/destroy/restore to get the data back to normal.

All right, that confirms my fear - I had dedup on for a while when experimenting ...

>
> If the drive is making that much noise, have you considered that the
> drive might be failing?

The drive seems all right, I think it's the reading/writing of many small blocks at once
when copying from that specific ZFS partition containing the nanbsd installation with
dedup on once. The noise is "classical", but unusual in that specific task when supposed
not to do so many read/writes. But, anyway, it is an optimistic guess and somehow also
wishful thinking ;-) I will take a drive failure also in consideration - I hopefully have
backups.

Kind reagrds,

Oliver
>

[O. Hartmann]

Am Sun, 2 Oct 2016 15:30:41 -0400
Allan Jude <alla...@freebsd.org> schrieb:

Hello.

Might there be any hint I can investigate on that ZFS partition showing me that the
particular partition is still doing deduplication? If I wouldn't know that I switch
dedup on and later off, I would blame the OS instead. So, for further forensik analysis
in the future, it would be nice to know how to look at it - if it is doable via simple
checking the features of the ZFS partition ...

Thanks,
oh

[Daniel Braniss]

> On 3 Oct 2016, at 12:30 PM, O. Hartmann <ohar...@zedat.fu-berlin.de> wrote:
>
> Am Sun, 2 Oct 2016 15:30:41 -0400

> Allan Jude <alla...@freebsd.org <mailto:alla...@freebsd.org>> schrieb:

not really an answer, but zpool has a nice command: history, it sometimes helps to find what and when
nfs commands where given.

danny

[Tomoaki AOKI]

Looks helpful to confirm 'when and how changed', but the output
contains 'all' changes such as snapshots. Would be hard to determine
what is the current configuration.

To determine what is the current settings, `zfs get all zroot` for case
that pool name is zroot.

Pool name can include child dataset like zroot/local.
To list all children, use `zfs get -r all zroot` instead.

To focus on dedup `zfs get -p dedup zroot` can be better.
If you want children datasets, '-r' shall be specified before '-p'.

Unfortunately, this (sub)command cannot show change logs.

HTH.


--
Tomoaki AOKI <junc...@dec.sakura.ne.jp>

[Evan Susarret]

In case you are still trying to get this resolved:

Since you have enabled deduplication in the past, unfortunately the deduplication tables must be loaded into memory for any and all writes. ZFS must determine if all writes match a previous block, to 1) increment the dedup count for matches, or 2) decrement the dedup count for a no-longer needed block.
This often means that the table must be loaded from disk repeatedly, especially if the table is large enough that it is purged from memory often. It is even possible to get your pool into a state where it cannot be imported due to the amount of memory required.
From the disk activity and slowness you described, it sounds like the dedup table (DDT) is being loaded, dumped, and loaded again repeatedly.

I don't know of the top of my head if you can use the 'zdb' command to examine the current DDT table, but there is a way to simulate the effect of enabling dedup on a given dataset using zdb in order to estimate the size of the DDT table that would result.

The only resolution, as Allan stated, is to zfs send and zfs recv the dataset into a new copy that does not have dedup enabled, or recreate it from scratch. Depending if you enabled dedup on single dataset, multiple datasets, or the whole pool, it may be easiest to recreate the whole pool. This is a side-effect of dedup that some have referred to as 'toxic', and some googling will show that it is a common issue encountered when experimenting with dedup (on all platforms).
Some guides will recommend only enabling dedup if you know that your data will have many duplicated blocks, and also that the tables will easily fit into memory. There are also guides that can help to estimate the amount of ram required to hold a DDT based on the blocksize and number of blocks in the dataset. This can be complicated by many factors, so it is usually best to avoid turning dedup on unless you are sure that it will help rather than hinder usage.

Hope this helps!
Evan

[Fabian Keil]

"O. Hartmann" <ohar...@zedat.fu-berlin.de> wrote:

> Running 12-CURRENT (FreeBSD 12.0-CURRENT #32 r306579: Sun Oct 2 09:34:50 CEST 2016 ), I
> have a NanoBSD setup which creates an image for a router device.
>
> The problem I face is related to ZFS. The system has a system's SSD (Samsung 850 Pro,
> 256GB) which has an UFS filesystem. Aditionally, I have also a backup and a data HDD,
> both WD, one 3 TB WD RED Pro, on 4 TB WD RED (the backup device). Both the sources for
> the NanoBSD and the object tree as well as the NANO_WORLDDIR are residing on the 3 TB
> data drive.
>
> The box itself has 8 GB RAM. When it comes to create the memory disk, which is ~ 1,3 GB
> in size, the NanoBSD script starts creating the memory disk and then installing world
> into this memory disk. And this part is a kind of abyssal in terms of the speed.

Can you reproduce the issue if you replace the memory disk with
a zvol or tmpfs?

While I agree that depulication could be the cause of the problem,
it probably wouldn't hurt to rule out a memory-disk related issue
before recreating the pool.

Fabian