raw vs sector mode

[Anton Gavriliuk]

Hi team

Which mode (raw or sector) should be choosen for non-dax Db/App access ?

Will performance be the same for example random read 4k in raw & sector modes ?

Anton

[Andy Rudoff]

Hi Anton,

I'm going to give you a little bit of a long answer...

When a write is in-flight to a block storage device, what happens if you lose power?  Many programs assume that a sector will not be torn by power failure, so afterwards, the sector contains either the old data or the new data or, perhaps in rare cases, some sort of error saying the sector's ECC shows it got corrupted.

But in reality this was not guaranteed on traditional storage for years, it was just an assumption made by lots of SW and the case is so rare, the SW was never fixed to deal with torn sectors.  More recent specifications, like the NVMe SSD specification, require at least a single sector power fail atomicity, so you know a single sector won't be torn by power failure.

With persistent memory, when you use the storage APIs (like read(2) and write(2) to the device), those just turn into memcpy() calls in the driver.  So if you lose power, you are likely to have some of the old data, some of the new data, and no indication of a corrupted block.  Again, that should be fine if the SW is designed to handle torn sectors (for example, if it uses checksums or something similar to detect and recover from that case).  But there's lots of SW out there written that erroneously thinks sectors cannot be torn by a power failure, including some file systems!  And since the memcpy() calls in the driver make this case more common, we invented the BTT algorithm (block translation table) for making sector writes power fail atomic.  When you use a "sectored" device for pmem, you are getting the power fail atomic sector writes added for you by the kernel.  It has an overhead, depending on the workload, I would expect an extra three 32-bit writes for every sector write.  For 4k sectors, that's a small overhead, but it is definitely there.  There's an extra "map" lookup (another 32-bit read) that happens on each block read as well.

If you have SW that checksums its log, or uses other mechanisms to detect torn writes, then you can turn off the sectored mode and just get that performance back.  But it depends on the SW and if you are not sure, I recommend sticking to the sectored mode to be safe.  Lots of database-like applications do indeed write their changes to a log and include a CRC or something similar, so lots of people have figured out the correct way to deal with this already.

The sectored support is not available when you're using DAX.  So even if you stick to the standard storage APIs, read and write on DAX will not be protected (and will not use the page cache, since that is what DAX is all about).  And, of course, if you memory map a DAX file, the application is taking on full responsibility for detecting torn writes, and only the align 8-byte store is power fail atomic.  This is where PMDK comes in, to help make that programming easier.

All that said, we have found applications that check for torn writes already, benefit from skipping the page cache, so run great by just running the unmodified app on top of a DAX file system using standard storage APIs.  

Hope that helps,

-andy

[Anton Gavriliuk]

Hi Andy

Thank you for the detailed answer.  While we have fast-growing list of DAX aware Db/App, currently any I/O intensive Db/App running on pmem should get huge performance boost even comparing to latest AFA arrays. That's why I'm asking about raw/secotr modes.

I have found that for random reads around 20+% slower in sector mode comparing to raw mode; for random writes 50+% slower comparing to raw mode.

Ok, it's Db/App dependent and must be discussed BEFORE run :-)

Anton

вт, 26 мар. 2019 г. в 18:39, Andy Rudoff <an...@rudoff.com>:

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/c8ddc3e7-9f2e-46a1-9ef6-16cf2eedb653%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeff Moyer]

Anton Gavriliuk <antos...@gmail.com> writes:

> Hi team
>
> Which mode (raw or sector) should be choosen for non-dax Db/App access ?

You should almost never use raw mode. If you don't want to use the btt,
you should still configure the namespace in fsdax mode.

> Will performance be the same for example random read 4k in raw & sector
> modes ?

No. You should test the performance of your application to find out the
exact overhead, though, as it will be workload dependent. I've seen btt
overheads as high as 30%.

Cheers,
Jeff

[Anton Gavriliuk]

Hi Jeff,

>  You should almost never use raw mode.  If you don't want to use the btt,
you should still configure the namespace in fsdax mode.

Well, I don't want being limited by fsdax.  How about Oracle ASM who likes block devices ?, so raw or sector for Oracle ASM ?

Anton

вт, 26 мар. 2019 г. в 19:05, Jeff Moyer <jmo...@redhat.com>:

[Jeff Moyer]

Anton Gavriliuk <antos...@gmail.com> writes:

> Hi Jeff,
>
>> You should almost never use raw mode. If you don't want to use the btt,
> you should still configure the namespace in fsdax mode.
>
> Well, I don't want being limited by fsdax. How about Oracle ASM who likes
> block devices ?, so raw or sector for Oracle ASM ?

You don't have to put a file system on the device.

The difference between raw mode and fsdax mode is that the former does
not have struct page-s describing the persistent memory. This leads to
failures when trying to do things like DMA, ptrace (gdb), etc.

Cheers,
Jeff

[Anton Gavriliuk]

Ok, now back to the question - which mode is the best for Oracle ASM ?  There are many customers with such setup, so this question definatly will be soon.

вт, 26 мар. 2019 г. в 19:16, Jeff Moyer <jmo...@redhat.com>:

[Adrian Jackson]

Looks like they're are targeting PM themselves, so might be worth asking them:

https://blogs.oracle.com/exadata/persistent-memory-accelerator

--

You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/CAAiJnjrVidSJNCmxN-OHmCY-KCQtyz5qyevPNhrsw3KfdGXSvw%40mail.gmail.com.

[Jeff Moyer]

Anton Gavriliuk <antos...@gmail.com> writes:

> Ok, now back to the question - which mode is the best for Oracle ASM ?
> There are many customers with such setup, so this question definatly will
> be soon.

Ask Oracle.

-Jeff