Setting up Ubuntu 18.04 to use the Optane memory module - /dev/dax0.0 is missing

62 views
Skip to first unread message

markf

unread,
Dec 29, 2018, 9:29:51 PM12/29/18
to pmem
My computer has a 32GB Optane and is running Ubuntu 18.04.

I have compiled and installed pmdk from www.github.com/pmem/pmdk

I have installed everything including daxio.

Then I looked up the pcj repository:


It says that to actually use the optane hardware memory, or emulated persistent memory, one must set config.properties:


there should be a device in /dev something like /dev/dax0.0

However, when I looked up /dev, I couldn't find anything like dax*

So how can I let the OS see my optane memory module and load it to /dev?


Also, where can I find documentation about setting up the OS to use the hardware optane module?




Andy Rudoff

unread,
Dec 30, 2018, 1:23:00 PM12/30/18
to pmem
Hi,

The Optane Media is used to make two classes of products.  One class is a block device, an SSD, only supporting block I/O.  The other case is a persistent memory module, supporting direct load/store access.  The persistent memory products have "PM" or "Persistent Memory" in the name (i.e. Optane DC Persistent Memory).  It is currently only available in the server-class, Xeon machines (the "DC" stands for "Data Center").

So one possibility is that you have the Optane product that is block accessible.  PMDK is a suite of libraries designed for use with the persistent memory programming model provided by NVDIMMs and the Optane DC Persistent Memory modules.  Things like PMDK (and PCJ, built on top of PMDK) are designed to take advantage of the very small granularity access to persistence using loads and stores, so it won't perform very well on top of Optane SSDs.

You can use the ndctl command to list any pmem devices in your Linux machine.  Here's a link to some ndctl documentation that may help: https://docs.pmem.io/ndctl-users-guide

If ndctl doesn't list it, then it isn't persistent memory.

Hope that helps,

-andy

Qichang Liang

unread,
Dec 30, 2018, 5:40:28 PM12/30/18
to pmem
I also have a computer with the Gigabyte Z370XP SLI motherboard, which has a M2 socket, fitted with the 32GB Optane memory module MEMPEK1W032GA 32 GB

Therefore, this is in fact a memory module, but it does not show up in /dev/ also "ndctl list" does not list anything.

The computer is also fitted with a 480GB Optane 900P SSD.  It shows up as /dev/nvme0 and nvme0n1


After I looked at:

http://dlcdnmkt.asus.com/microsite/Optane-Memory-SOP.pdf

I changed the SATA mode selection from ...... to AHCI

Then after rebooting, the 32GB Optane memory module shows us as /dev/nvme1 and /dev/nvme1n1

However, it does not show up as /dev/pmem

Also, "ndctl list" still does not show anything.


So how can I get Ubuntu 18.04 to see my 32GB Optane memory module as /dev/pmem and get ndctl to see the memory module as pmem?

Andy Rudoff

unread,
Dec 30, 2018, 5:59:41 PM12/30/18
to pmem
Hi,

As I said earlier in the thread, the products that use Optane Media for persistent memory have "PM" or "Persistent Memory" in their name and are initially supported on servers only.

Anything that speaks SATA, AHCI, or NVMe is speaking block protocols and can only be accessed as a block device.  There's no way to make it show up as /dev/pmem -- you have to use a persistent memory device that connects to the memory bus, such as an NVDIMM-N, or an Optane DC Persistent Memory module.

Hope that helps,

-andy

steve

unread,
Dec 30, 2018, 10:35:44 PM12/30/18
to Andy Rudoff, pmem
On Sun, 30 Dec 2018 10:22:59 -0800 (PST), Andy Rudoff <an...@rudoff.com> wrote:

>Hi,
>
>The Optane Media is used to make two classes of products. One class is a
>block device, an SSD, only supporting block I/O. The other case is a
>persistent memory module, supporting direct load/store access. The
>persistent memory products have "PM" or "Persistent Memory" in the name
>(i.e. Optane DC Persistent Memory). It is currently only available in the
>server-class, Xeon machines (the "DC" stands for "Data Center").

So it is available for those who are willing to buy a Xeon server-class machine?

If so, is there a price list somewhere?
------------
Steve Heller

Qichang Liang

unread,
Dec 30, 2018, 10:39:34 PM12/30/18
to pmem
Hi Andy,

This is the optane memory that I got:


So are you saying that this "optane memory" CANNOT be used as persistent memory despite it is marketed as such, although it has been used to accelerate hard drives?  (Note: I am NOT talking about the Optane 900P SSD here)

Is this "optane memory" still a block device, which another member describes as running "1000x slower" during random accesses?


If the above is true, then is this the Optane DC Persistent Memory you refer to?


If so, then where can I buy it and how much it costs?  I need to know the price in order to get funding to get one for my project.

Thanks!

Qichang

jackv...@gmail.com

unread,
Dec 30, 2018, 11:39:08 PM12/30/18
to pmem
Hi all,
Hey all - Intel Optane DC Persistent Memory is not generally available yet, it will launch with the next generation Xeon processor, code named Cascade Lake in early 2019. This is the product that will utilize the persistent memory programming model. You will have to wait for the product launch or reach out to your hardware vendor or cloud service provider of choice for pricing guidance. 

This article may help you in understanding the differences between the different Optane products - https://software.intel.com/en-us/articles/3d-xpoint-technology-products.

-Jack

Qichang Liang

unread,
Dec 31, 2018, 4:36:57 AM12/31/18
to pmem
Thanks for pointing out this link:


It says that the Intel optane memory module, which is mounted onto an M.2 socket (the one I got and mentioned in my previous posts in this thread), is in fact byte addressable:

"Intel Optane memory bridges the functionality of DRAM and storage in that it is high performance, byte-addressable, and offers persistent storage."

So if the Linux kernel fails to acknowledge this byte-addressable M.2 mounted optane memory as /dev/pmem, and instead assign it as a block device, then can this be considered as a bug in the Linux kernel?

It will be a big handicap of the Linux kernel and pmdk if they cannot recognize this byte-addressable Optane memory module as persistent memory and use it accordingly.

Otto Chow

unread,
Dec 31, 2018, 5:21:25 AM12/31/18
to Andy Rudoff, pmem
Hi Mark,

The 32GB Optane Memory product you are using is designed for desktop/laptop usage as disk cache for HDD/SSD.  PMDK does not support this type of Optane memory product.

PMDK only supports the server grade Optane memory product which it has not been launched yet. Optane DC Persistent Memory modules will work with the next generation of Xeon processors which is not available on the market yet.

If you want to develop software with PMDK, You can consider to use the PM emulator.

Thank you.

Regards,
Otto

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/619ba0bb-a7a7-478a-9567-74d022954214%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Andy Rudoff

unread,
Dec 31, 2018, 11:37:14 AM12/31/18
to pmem
Hi Qichang,

Although the underlaying Optane media is byte-addressable, the product you have is not.  The specs you linked to state it is a PCIe NVMe drive.  The only way to do I/O to such a device is using block-based NVMe commands.  There's no amount of code you could add to the Linux kernel to make this device show up as /dev/pmemX.

Currently the only devices supporting the persistent memory programming model are NVDIMM-N products (that have been shipping for quite a while from various vendors) and the upcoming Optane DC Persistent Memory designed to work with Cascade Lake servers as Jack mentioned.

-andy

Qichang Liang

unread,
Dec 31, 2018, 4:53:58 PM12/31/18
to pmem
I see.... then is it possible to emulate the pmem by making pmem to allocate persistent memory on a particular NVMe device, such as the M.2 mounted 32GB Optane memory?

Also, you said there are NVDIMM-N products around for a long time, can you please name some of the NVDIMM-N products?

Tanks!

Qichang

Amnon Izhar

unread,
Dec 31, 2018, 5:09:52 PM12/31/18
to Qichang Liang, pmem

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/f15183e4-ba17-4913-844a-024670d82370%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Amnon Izhar

steve

unread,
Dec 31, 2018, 5:32:56 PM12/31/18
to Amnon Izhar, Qichang Liang, pmem
Hi Amnon,

That part is not an NVDIMM, just a normal DIMM, as far as I can tell from the Amazon description. I'm pretty sure an NVDIMM would be far more
expensive.

On Mon, 31 Dec 2018 17:09:37 -0500, Amnon Izhar <aiz...@gmail.com> wrote:

>Qichang,
>
>As outlined b Andy & Otto - any device that is presented as block device
>(dev/nvme or dev/sda etc.) are block device and only expose block
>interface, not memory. As such you cannot use them as persistent memory.
>For NVDIMM-N you can find multiple of them online - here is one example
>https://www.amazon.com/Crucial-PC4-17000-Unbuffered-288-Pin-Memory/dp/B00MMLUYQK/ref=asc_df_B00MMLUYQK/?tag=hyprod-20&linkCode=df0&hvadid=309777534894&hvpos=1o2&hvnetw=g&hvrand=384091815704806650&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9002064&hvtargid=pla-456286650128&psc=1&tag=&ref=&adgrpid=58425267301&hvpone=&hvptwo=&hvadid=309777534894&hvpos=1o2&hvnetw=g&hvrand=384091815704806650&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9002064&hvtargid=pla-456286650128
>Amnon
>
><https://www.amazon.com/Crucial-PC4-17000-Unbuffered-288-Pin-Memory/dp/B00MMLUYQK/ref=asc_df_B00MMLUYQK/?tag=hyprod-20&linkCode=df0&hvadid=309777534894&hvpos=1o2&hvnetw=g&hvrand=384091815704806650&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9002064&hvtargid=pla-456286650128&psc=1&tag=&ref=&adgrpid=58425267301&hvpone=&hvptwo=&hvadid=309777534894&hvpos=1o2&hvnetw=g&hvrand=384091815704806650&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9002064&hvtargid=pla-456286650128>
>> <https://groups.google.com/d/msgid/pmem/f15183e4-ba17-4913-844a-024670d82370%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>--
>Amnon Izhar
------------
Steve Heller

Andy Rudoff

unread,
Dec 31, 2018, 5:35:11 PM12/31/18
to pmem
Since an NVMe device can only speak blocks, you can't emulate pmem to the extent where you are actually doing loads and stores directly to the media.  But of course, the whole idea of the programming model is to build on storage APIs.  So if you just create a file system on the NVMe device and mmap a file, you will page into DRAM as necessary and libraries like PMDK detect this situation and use msync() to flush changes instead of CLWB.  It will just be much slower since each flush must write an entire page instead of a cache line like real pmem.  Still, it is good enough to develop SW, just not good enough to run any benchmarks.

You can also just use DRAM to emulate pmem, and then you do get to use /dev/pmemX and file system DAX, etc, you just don't really have persistence.  We have a blog entry about this here: http://pmem.io/2016/02/22/pm-emulation.html

For an NVDIMM-N, just google for it.  Here's one such product: http://www.vikingtechnology.com/products/nvdimm/

-andy

Amnon Izhar

unread,
Dec 31, 2018, 5:44:47 PM12/31/18
to st...@steveheller.org, Qichang Liang, pmem
Hello Steve
You are correct - i pulled the wrong link from my browser. I meant to put in this one http://www.netlist.com/products/vault-memory-storage/nvvault-ddr4-nvdimm/ 
Amnon
--
Amnon Izhar

Yonghong Yan

unread,
Jan 8, 2019, 10:18:41 AM1/8/19
to Andy Rudoff, pmem

Andy, 

How about the marketed Intel Memory Drive Technology (https://www.intel.com/content/www/us/en/products/docs/memory-storage/solid-state-drives/optane-ssd-dc-p4800x-mdt-brief.html)? How that works? Are they enabled byte-addressability from firmware or not possible at all since it is SSD. 

Yonghong

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.

Steve Scargall

unread,
Jan 8, 2019, 12:07:31 PM1/8/19
to pmem
Hi Yonghong,

The P4800X devices are not byte addressable as the 3D XPoint Media sits behind a block device controller/interface.  From the URL you referenced - "Together, DRAM and Intel® Optane™ SSDs with Intel® Memory Drive Technology emulate a single volatile memory pool".

HTH,
    Steve

Phani Arava

unread,
Jan 8, 2019, 8:39:20 PM1/8/19
to Yonghong Yan, Andy Rudoff, pmem
This is one more trick off Intel's marketing. The product is a spin-off of scalemps more extender.

They use a small hypervisor and extend the memory using SSD as the backend.

The benchmark results they provide are interesting though

Reply all
Reply to author
Forward
0 new messages