ATA 4K sector issues
Mohacsi Janos
What is the situation with ATA 4K dirves in FreeBSD? Are there any
support for them in fdisk or disklabel?
More information at:
http://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
Janos Mohacsi
Head of HBONE+ project
Network Engineer, Deputy Director of Network Planning and Projects
NIIF/HUNGARNET, HUNGARY
Key 70EF9882: DEC2 C685 1ED4 C95A 145F 4300 6F64 7B00 70EF 9882
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"
Andriy Gapon
> Dear FreeBSD hackers,
> What is the situation with ATA 4K dirves in FreeBSD? Are there any
> support for them in fdisk or disklabel?
> More information at:
> http://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
Did you mean to say gpart(1)? :)
AFAIK, the tool(s) do not auto-align on 4K boundaries, but they give user an
ability to do that by hand.
Besides, some 4K sector disks lie that they have 512 sectors.
--
Andriy Gapon
Kent Stewart
> on 17/03/2010 12:16 Mohacsi Janos said the following:
> > Dear FreeBSD hackers,
> > What is the situation with ATA 4K dirves in FreeBSD? Are there
> > any support for them in fdisk or disklabel?
> > More information at:
> > http://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
>
> Did you mean to say gpart(1)? :)
> AFAIK, the tool(s) do not auto-align on 4K boundaries, but they give
> user an ability to do that by hand.
> Besides, some 4K sector disks lie that they have 512 sectors.
An article on the effect is located at
http://storageeffect.media.seagate.com/2010/03/storage-effect/y2-011k-looming-for-windows-xp-users/
Kent
--
Kent Stewart
Richland, WA
http://users.owt.com/kstewart/index.html
Mohacsi Janos
On Wed, 17 Mar 2010, Dag-Erling Sm?rgrav wrote:
> Mohacsi Janos <moh...@niif.hu> writes:
>> What is the situation with ATA 4K dirves in FreeBSD? Are there
>> any support for them in fdisk or disklabel?
>
> Hmm, didn't we discuss this already? All we need to do is change the
> defaults in fdisk so it rounds partition offsets and sizes to a multiple
> of 8 sectors (or 16 for future-proofing) instead of aligning them with
> fictitious cylinder boundaries. Bsdlabel, as disklabel is properly
> known, already DTRT: by default, the first partition starts at offset
> 16. Just make sure you specify sizes that are divisible by 8 or 16
> blocks (not an issue if you use the M or G suffixes). Sysinstall
> operates in megabytes.
Thanks for the information.
Dag-Erling Smørgrav
> I had problem with ZFS.
What kind of problem?
DES
--
Dag-Erling Smørgrav - d...@des.no
Thiago Damas
above 400ms was common, with peaks of 3000ms per write). Tested on two
different machines.
Disks WD10EARS in single disk and mirror configurations.
Thiago
2010/3/17 Dag-Erling Smørgrav <d...@des.no>
Thiago Damas
use those disks.
2010/3/17 Mohacsi Janos <moh...@niif.hu>
Bruce Cran
> Dear FreeBSD hackers,
> What is the situation with ATA 4K dirves in FreeBSD? Are there any
> support for them in fdisk or disklabel?
# mdconfig -a -f ddfile -S 4096
md0
# fdisk /dev/md0
fdisk: could not detect sector size
# mdconfig -d -u 0
# mdconfig -a -f ddfile -S 1024
md0
# fdisk /dev/md0
******* Working on device /dev/md0 *******
parameters extracted from in-core disklabel are:
cylinders=130 heads=255 sectors/track=63 (16065 blks/cyl)
parameters to be used for BIOS calculations are:
cylinders=130 heads=255 sectors/track=63 (16065 blks/cyl)
fdisk: invalid fdisk partition table found
fdisk: /boot/mbr: length must be a multiple of sector size
So it seems there's still work to do to get fdisk working, but I can't try
gpart since I don't have a real disk.
--
Bruce Cran
Olivier Smedts
> I had problem with ZFS.
> With gnop -S 4096, it works well on /dev/ad{a}X.nop; but I decided to not
> use those disks.
So maybe this was not a ZFS problem but a partition misalignment problem ?
On a properly aligned partition with a physical sector size of 4KB and
a logical sector size of 512 bytes, will ZFS try to use blocksizes of
less than 4KB ? Blocksize in ZFS seems to be dynamic (at least when
not told to use a fixed blocksize), but I didn't see somewhere in the
manpage or Sun's website which minimum blocksize ZFS would use for
small files, and if there is a lower limit on blocksizes to use.
--
Olivier Smedts _
ASCII ribbon campaign ( )
e-mail: oli...@gid0.org - against HTML email & vCards X
www: http://www.gid0.org - against proprietary attachments / \
"Il y a seulement 10 sortes de gens dans le monde :
ceux qui comprennent le binaire,
et ceux qui ne le comprennent pas."
Olivier Smedts
> On Wednesday 17 March 2010 10:16:16 Mohacsi Janos wrote:
>> Dear FreeBSD hackers,
>> What is the situation with ATA 4K dirves in FreeBSD? Are there any
>> support for them in fdisk or disklabel?
>
> # mdconfig -a -f ddfile -S 4096
> md0
> # fdisk /dev/md0
> fdisk: could not detect sector size
> # mdconfig -d -u 0
> # mdconfig -a -f ddfile -S 1024
> md0
> # fdisk /dev/md0
> ******* Working on device /dev/md0 *******
> parameters extracted from in-core disklabel are:
> cylinders=130 heads=255 sectors/track=63 (16065 blks/cyl)
>
> parameters to be used for BIOS calculations are:
> cylinders=130 heads=255 sectors/track=63 (16065 blks/cyl)
>
> fdisk: invalid fdisk partition table found
> fdisk: /boot/mbr: length must be a multiple of sector size
>
> So it seems there's still work to do to get fdisk working, but I can't try
> gpart since I don't have a real disk.
Why not on geom_md ?
# mdconfig -a -f ddfile -S 4096
# gpart create -s gpt md0
# gpart list md0
Geom name: md0
fwheads: 32
fwsectors: 1
last: 25594
first: 6
entries: 128
scheme: GPT
Consumers:
1. Name: md0
Mediasize: 104857600 (100M)
Sectorsize: 4096
Mode: r0w0e0
Same results with MBR scheme.
>
> --
> Bruce Cran
> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"
>
--
Olivier Smedts _
ASCII ribbon campaign ( )
e-mail: oli...@gid0.org - against HTML email & vCards X
www: http://www.gid0.org - against proprietary attachments / \
"Il y a seulement 10 sortes de gens dans le monde :
ceux qui comprennent le binaire,
et ceux qui ne le comprennent pas."
Thiago Damas
34, 35, 36, 37, 38, 39 etc
Even 64
gpart create -s gpt ad4
gpart add -s 64 -t freebsd-zfs ad4
With UFS, I previouly did some tests (in portuguese):
Particao sem alinhar inicio
=> 34 1953525101 ad4 GPT (932G)
34 1953525101 1 freebsd-ufs (932G)
newfs -U (padrao -f 2048 -b 16384)
# time tar xfj
/install/releases/i386/8.0-RELEASE/8.0-RELEASE/ports/ports.tgz
2.342u 5.013s 20:35.65 0.5% 64+1091k 6623+2734io 0pf+0w
newfs -U -f 4096 -b 32768
# time tar xfj
/install/releases/i386/8.0-RELEASE/8.0-RELEASE/ports/ports.tgz
2.590u 5.263s 22:06.32 0.5% 63+1087k 10911+1676io 0pf+0w
Particao alinhada em multiplos de 4096 bytes (blocos logicos de 512
bytes, entao, multiplo de 8 blocos)
=> 34 1953525101 ad4 GPT (932G)
34 30 - free - (15K)
64 1953525071 1 freebsd-ufs (932G)
newfs -U (padrao -f 2048 -b 16384)
# time tar xfj
/install/releases/i386/8.0-RELEASE/8.0-RELEASE/ports/ports.tgz
2.392u 5.000s 10:24.15 1.1% 64+1095k 8375+2735io 0pf+0w
newfs -U -f 4096 -b 32768
# time tar xfj
/install/releases/i386/8.0-RELEASE/8.0-RELEASE/ports/ports.tgz
2.547u 5.567s 1:25.50 9.4% 64+1088k 11880+1755io 0pf+0w
Bruce Cran
> Why not on geom_md ?
Thanks! After getting a "no such geom" message when I tried a couple of
commands without having created any partitions I presumed it was looking for a
DISK provider.
Dag-Erling Smørgrav
> With UFS, I previouly did some tests (in portuguese):
Some commentary would be nice.
DES
--
Dag-Erling Smørgrav - d...@des.no
Thiago Damas
Partition without begin aligned:
=> 34 1953525101 ad4 GPT (932G)
34 1953525101 1 freebsd-ufs (932G)
newfs -U (default -f 2048 -b 16384)
# time tar xfj
/install/releases/i386/8.0-RELEASE/8.0-RELEASE/ports/ports.tgz
2.342u 5.013s 20:35.65 0.5% 64+1091k 6623+2734io 0pf+0w
newfs -U -f 4096 -b 32768
# time tar xfj
/install/releases/i386/8.0-RELEASE/8.0-RELEASE/ports/ports.tgz
2.590u 5.263s 22:06.32 0.5% 63+1087k 10911+1676io 0pf+0w
Partition aligned on 4k boundaries
=> 34 1953525101 ad4 GPT (932G)
34 30 - free - (15K)
64 1953525071 1 freebsd-ufs (932G)
newfs -U (default -f 2048 -b 16384)
# time tar xfj
/install/releases/i386/8.0-RELEASE/8.0-RELEASE/ports/ports.tgz
2.392u 5.000s 10:24.15 1.1% 64+1095k 8375+2735io 0pf+0w
newfs -U -f 4096 -b 32768
# time tar xfj
/install/releases/i386/8.0-RELEASE/8.0-RELEASE/ports/ports.tgz
2.547u 5.567s 1:25.50 9.4% 64+1088k 11880+1755io 0pf+0w
Aligning the partition, and ajusting the block/fragment size, I went
from 22minutes to 1m25s.
But, I tried every combination possible with ZFS, and always get
slow performance (something like 2, 3 MB/s write speed)
2010/3/17 Dag-Erling Smørgrav <d...@des.no>
Matthew Dillon
the sector offset to 2 but I came to the conclusion that it was better
to do the alignment in disklabel / gpt / whatever higher-level
partitioner floats your boat and not mess with anything the BIOS
uses to boot the machine
My recommendation is to use a 1MB physical base alignment. That's what
I adjusted DragonFly's disklabel64 to do. It's definitely best to
have the partitioner deal with it instead of having to mess around
manually because the partitioner can calculate the actual physical
alignment by querying the kernel's disk subsystem regardless of the
topology.
There are several reasons for using a large alignment:
* A variety of media already uses much larger physical block sizes.
MLC flash uses 128K and SLC uses 64K blocks. See the note below
on why this matters even though SSDs do write combining.
* A larger alignment is more likely to work well as a default in
RAID configurations and doesn't hurt non-RAID.
* The kernel cluster I/O subsystem wants to collect stuff into 64K-256K
clusters for reading and writing (writing being the most important).
A larger alignment plus some minor tweeks in the cluster code will
cause the cluster writes to also be well aligned.
* Even though UFS does not take advantage of cluster alignment
(because BMAP tends to align only to the UFS block size which
is a fairly small <= 32K usually), filesystems such as ZFS (with
128K blocks I believe) and HAMMER (with 64K blocks and 8MB super
blocks) will. And fixing up UFS isn't difficult. One might need
to mess with the cylinder group alignment and make some minor tweeks
to the bmap allocator but that's about it.
* A large alignment hurts nothing. Who cares about ~512K-1MB of wasted
space at the beginning of the drive? I don't.
This is particularly important for SSDs. Even though SSDs do write
combining a properly aligned write will theoretically greatly improve
write endurance by reducing internal fragmentation, reducing write
amplification effects, and also reducing the amount of internal
rewriting the drive does to defragment and wear-level. It is hard
to test this but I am seeing wear rates condusive with a 100TB write
endurance on 40G Intel drives vendor-speced for a 35TB write endurance.
So even though you might not see a major difference in performance
you could very well see a big difference in write endurance. It isn't
possible to benchmark this with a standard benchmark which keeps the
SSD 100% active so I've been using real work loads and it just takes
forever to tick-down the SSDs wear-meter. The SSD also needs idle
time to implement internal defragmentation and wear leveling efficiently
(This seems more apparent in the OCZs than in the Intels).
There are a lot of moving parts in the kernel related to alignment.
The cluster code and the filesystem block allocation code are the two
biggest issues and adjustments have to made to take proper advantage
of it, particularly for SSDs.
So the answer is: Aligning things certainly isn't going to hurt
anything so you might as well kick it hard (use a large alignment)
so you don't have to revisist the problem again a year from now.
--
For hard drives with larger physical sector sizes it shouldn't matter
for asynchronous writes. It really shouldn't. And nearly all of UFS's
writes are asynchronous. That said:
I read Thiago's posting. I will note something specifics about a ports
tarball. Ports has 261,000+ files in it, mostly small. UFS and the
cluster code CANNOT COMBINE those writes (because the buffer-cache for
file data is per-vnode), so UFS will wind up doing a very large number
of fragment-sized writes.
These fragment-sized writes (4K in Thiago's aligned test that ran in
1:25, and 2K in Thiago's aligned test that ran in 10:24) should STILL
be write-combined in the drive. That is, UFS STILL has good write
linearity even with the small writes.
So I suspect the issue here is that the drive is not properly
write-combining the writes, possibly coupled with additional issues
in UFS's bmap and inode allocator that might not be presenting the
drive with enough write-combinable data that fits in the drive's cache,
forcing the drive to do a lot of read-before-write.
In terms of write-combinable data and UFS it could be a cylinder-group
alignment issue. Bitmap blocks are a particular problem because they
use an odd-sized block size (typically 6K if I remember right), though
I'm not sure how the filesystem fragment size effects it.
You would have to instrument the write activity to determine how
good the linearity is verses the size of the drive's ram cache.
There are definitely several possible explanations for the horrible
performance when using 2K fragments.
ZFS (and also HAMMER) would not have this particular problem. ZFS
clearly has other issues in those tests but I don't know enough about
its internals to guess, other than maybe it is a ZIL tuning issue.
-Matt
Matthew Dillon
drive's zone cache. Drive zone caches work very well for combing
adjacent sectors when the write zones are perfectly linear (when
the writes within each zone being tracked are perfectly linear).
But the drive zone caches I've tested tend to break down very
quickly when the data is written out of order, even if all the data winds
up being present (could be combined into a linear result).
I posted a program a while back that showed that. Basically writing
512-byte sectors ordered 0, 1, 2, 3, 4, ... is very fast, but the
moment you start writing out of order, e.g. 3, 2, 1, 0, 7, 6, 5, 4,
things start to break down very quickly.
I would suspect the issue with UFS and a 2K fragment size on a 4K
physical sector drive is precisely this. When UFS mixes fragmented
writes with full block writes (related to different files), they
tend to have locality of reference but they also tend to be NOT
perfectly linear. 2K+ gaps will be created during the unpack and
filled later on, but the drive cache can't handle it.
Having lots of directories with a few small files in each probably
doesn't help matters any either but that can't be 'fixed' without
messing up UFS's ability to maintain a relatively unfragmented
filesystem over long periods of time.
This is probably what is tripping the drive up. This implies that
you absolutely must use a 4K fragment size (32K block size) and an
aligned partition when using UFS with 4K physical sector drives.
Thiago Damas
But I also remember that zfs block size its 128k, but metadata can be of
dynamic size. And we can use compressed files too.
There is a sysctl, md_compress, that I turned out in my tests, but not
working as expected.
Why using gnop -S 4096 works well?
Thiago
Matthew Dillon
: There is a sysctl, md_compress, that I turned out in my tests, but not
:working as expected.
: Why using gnop -S 4096 works well?
:
:Thiago
You are setting the sector size to 4K with gnop -S 4096 so presumably
ZFS will not do any fragmented writes smaller than that. I'm not
sure why that would matter except possibly for ZIL writes. In the
case of ZIL if ZFS is using sector-sized writes (I don't know what it
actually uses) then setting the sector size to 4K would be more
efficient as the drive would not have to issue a read-before-write
when the disk cache is flushed after the ZIL write.
One important aspect of having the filesystem use a larger logical
block size, such as 4K or 16K or 32K etc, is that the filesystem
itself knows whether any trailing data is garbage or not and will
avoid doing a read-before-write when writing small amounts of data.
Most of the time if the filesystem is allocating space from its blockmap
it knows the trailing data in the block is garbage and will zero it
instead of performing a read-before-write. Also, the buffer cache covers
hundreds of megabytes verses the hard drive cache which is typically
only 8-64MB (though the OCZ Colosus has 128M). Still, this means the
kernel will do a much better job write-combining than the drive.
The drive has no knowledge of what is garbage and what is not at the
drive level, so the moment this stuff moves out of the drive and into
the kernel you reap rewards on these larger physical sector-sized drives.
-Matt
Andrew Stesin
> you absolutely must use a 4K fragment size (32K block size) and an
> aligned partition when using UFS with 4K physical sector drives.
Sorry for my ignorance, but what's wrong with making this the default
setting for newfs regardless of whatever the drive is?
Thanks,
Andrew
Thiago Damas
minimum of 2K, wasting 1k. With 4k frag size it will waste 3k.
2010/3/17, Andrew Stesin <ste...@gmail.com>:
Dag-Erling Smørgrav
> We experimented a bit with aligning fdisk (dos slices) by changing the
> sector offset to 2 but I came to the conclusion that it was better to
> do the alignment in disklabel / gpt / whatever higher-level
> partitioner floats your boat and not mess with anything the BIOS uses
> to boot the machine
Not sure what you mean by "changing the sector offset to 2", but the
BIOS doesn't care where partitions start or end. It just obeys the
partition table.
> My recommendation is to use a 1MB physical base alignment. That's
> what I adjusted DragonFly's disklabel64 to do. It's definitely best
> to have the partitioner deal with it instead of having to mess around
> manually because the partitioner can calculate the actual physical
> alignment by querying the kernel's disk subsystem regardless of the
> topology.
The disk system doesn't necessarily know. Disks have been known to lie
about their physical sector size. The best strategy is to consistently
use a large alignment "just in case". Using a 4k or 8k or even 1M
alignment has no negative impact on 512b disks (other than that it
wastes a very small amount of disk space), but for a 4k disk or an SSD,
it can make the difference between "dog slow" and "lightning fast".
> * A variety of media already uses much larger physical block sizes.
> MLC flash uses 128K and SLC uses 64K blocks. See the note below
> on why this matters even though SSDs do write combining.
Some SSDs do RAID0 internally and therefore have larger block sizes.
ISTR the OCZ Vertex is made up of four SATA SSDs with a microcontroller
in front.
DES
--
Dag-Erling Smørgrav - d...@des.no
Olivier Smedts
> I'll try tomorrow more zfs tests, with 1M alignment on begining of disk.
> But I also remember that zfs block size its 128k, but metadata can be of
> dynamic size. And we can use compressed files too.
ZFS block size for data is not fixed at 128k, search for recordsize in
the zfs man page.
> There is a sysctl, md_compress, that I turned out in my tests, but not
> working as expected.
> Why using gnop -S 4096 works well?
Interesting thread for issues with actual 4k disks :
http://opensolaris.org/jive/thread.jspa?messageID=447926
I'm waiting for a jumper on those 4k disks to turn off the "legacy" 512b mode.
>
> Thiago
> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"
>
--
Olivier Smedts _
ASCII ribbon campaign ( )
e-mail: oli...@gid0.org - against HTML email & vCards X
www: http://www.gid0.org - against proprietary attachments / \
"Il y a seulement 10 sortes de gens dans le monde :
ceux qui comprennent le binaire,
et ceux qui ne le comprennent pas."