Write pending queues' (WPQs) buffering policy

[Raqib Islam]

Hi Everyone,

As we all know that, the 3D-XPoint physical media access granularity is 256 bytes. This causes a write amplification problem for writes smaller than 256 bytes. To avoid this problem, the WPQ buffer queues up to 256 B data issued from a single thread. I am wondering how this queueing up happen?

For example, consider we are writing data in an 8-Byte integer array. At first, we write data at the 31st index (in 0-based index). My question is, how the WPQ will queue up the future writes? Here are some of the possible options:

1. Writes from the index range [0-31] will be queued up (as 32 8-Bytes integer combines 256 Bytes).
2. Writes from the index range [31-62] will be queued up (again as writes start at the index-31 and 32 8-Bytes integer combines 256 Bytes).
3. Writes will be queued up based on the cache line alignment.

Can you please help me with this?

--

Abdullah Al Raqibul Islam
Ph.D. Student
University of North Carolina at Charlotte

[Andy Rudoff]

Hi,

The term WPQ, or Write Pending Queue, refers to the queue of outstanding writes in the memory controller (part of the CPU chip).  Each entry in the Write Pending Queue is a 64-byte cacheline and for PMem, the power supply must store enough energy to allow these queues to drain on power loss so that stores sitting in the WPQs can be considered persistent by software (known as the ADR feature, a platform feature required for supporting PMem).

I think what you are asking about isn't the WPQs, but instead the Write Data Buffers (WDBs) which are buffers located on the PMem device, used to hold data while the device is collecting adjacent cache lines to construct a full ECC block.  We don't publish detailed information about the internal architecture of the PMem devices, but you can probably glean quite a bit of information about it by running tests and gathering stats using ipmwatch.  The ipmwatch program is open source and you'll see a brief mention of the WDB in the documentation in the repo:

https://github.com/intel/intel-pmwatch

Note that one of the reasons the internal details are not published is that they may change significantly between generations, including the size and number of WDBs, as well as the algorithm for managing when data moves from a WDB to the media.

Hope that helps.

-andy