Persistent memory support for Go

[Jerrin Shaji George]

Hi,

 

I am part of a small team at VMware working on projects related to persistent

memory (others in CC). We have recently been working on adding persistent memory

support to the Go programming language, and I wanted to spread the word about

couple of these projects.

 

1) Go-pmem-transaction

The go-pmem-transaction project introduces a new programming model for

developing applications in Go for persistent memory. It consists of two packages

- pmem and transaction.

 

The pmem package provides methods to initialize persistent memory and an

interface to set and retrieve objects in persistent memory. The transaction

package provides undo and redo transaction logging APIs to support

crash-consistent updates to persistent memory data.

 

Project page - https://github.com/vmware/go-pmem-transaction

 

2) Go-pmem

The Go-pmem project adds native persistent memory support to Go.

Some of the features of the persistent memory support added to Go are:

                * Support for persistent memory allocations

                * Garbage collector now collects objects from persistent heap and volatile

                heap

                * Runtime automatically swizzles pointers if the memory mapping address

                changes on an application restart

                * The persistent memory heap is dynamically sized and supports automatic

                heap growth depending on memory demand

 

Project page - https://github.com/jerrinsg/go-pmem

 

The project pages contains links to further documentation. We welcome the

community to try out these projects and send any feedback our way!

 

Also see the blog post at https://blogs.vmware.com/opensource/2019/04/03/persistent-memory-with-go/

 

Thanks,

Jerrin

[Robert Engels]

I think the project needs to be distributed as a patch to the Go codebase - too much to review/maintain for security controls. 

Also, I’m curious about the Api. In reviewing the example, it looks no different than any ORM - just flatter -and these don’t need a customized runtime. 

What would be the advantage of using the Api directly rather than using Redis (with the support there)?

No criticism, just not obvious to me. Maybe a link describing the persistent memory capabilities in detail?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Engels]

Never mind on the “what is pmem” as I found the VMware docs. 

I am curious though why you need specialized runtime support rather than mmap on a block device? I am assuming that the api is slightly easier to use from a standard Go perspective but given the variable performance characteristics of pmem I don’t see why it shouldn’t be treated like any other external resource. 

[Robert Engels]

Also, I’ve used pmem for a very long time (I have a bg in retail point of sale systems where it is very common) and we always had a driver/library on top of the physical device to multiplex across consumers, and we used it from Java, and never needed specialized JVM to use it efficiently. Again, just curious, maybe the use cases have changed. 

On Apr 3, 2019, at 5:47 PM, Robert Engels <ren...@ix.netcom.com> wrote:

[Robert Engels]

Thanks for the info!

On Apr 3, 2019, at 6:32 PM, Mohit Verma <moh...@vmware.com> wrote:

Hi Robert,

 

Sorry for the confusion. Our changes are to make byte-addressable persistent memory support available to Go community. You can find more information about this technology here:

1. http://pmem.io/

2. https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html

 

>>> I think the project needs to be distributed as a patch to the Go codebase - too much to review/maintain for security controls. 

We eventually plan to distribute our changes as patches to Go codebase, but before we do that we wanted to gather feedback here 😊

 

>>> Also, I’ve used pmem for a very long time (I have a bg in retail point of sale systems where it is very common) and we always had a driver/library on top of the physical device to multiplex across consumers, and we used it from Java, and never needed specialized JVM to use it efficiently. Again, just curious, maybe the use cases have changed.

The recent technological changes have put this technology close to the access speeds of DRAM,  while retaining the persistence property. Quoting from the blog post in Jerrin’s original e-mail:

“Ad hoc libraries often provide a difficult programming interface. As memory management for RAM has increasingly become a part of programming languages, we believe managing persistent memory should be addressed in the same manner. And so, our programming model includes changes to Go itself ”.

 

There has been a lot of work around this technology in the past few years, including efforts to change C and Java, and adhoc libraries for C/C++, Java and Python. These changes are our efforts to make persistent memory accessible to Go developers.

 

Thanks!

Mohit

[Mohit Verma]

Hi Robert,

 

Sorry for the confusion. Our changes are to make byte-addressable persistent memory support available to Go community. You can find more information about this technology here:

1. http://pmem.io/

2. https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html

 

>>> I think the project needs to be distributed as a patch to the Go codebase - too much to review/maintain for security controls. 

We eventually plan to distribute our changes as patches to Go codebase, but before we do that we wanted to gather feedback here 😊

 

>>> Also, I’ve used pmem for a very long time (I have a bg in retail point of sale systems where it is very common) and we always had a driver/library on top of the physical device to multiplex across consumers, and we used it from Java, and never needed specialized JVM to use it efficiently. Again, just curious, maybe the use cases have changed.

The recent technological changes have put this technology close to the access speeds of DRAM,  while retaining the persistence property. Quoting from the blog post in Jerrin’s original e-mail:

“Ad hoc libraries often provide a difficult programming interface. As memory management for RAM has increasingly become a part of programming languages, we believe managing persistent memory should be addressed in the same manner. And so, our programming model includes changes to Go itself ”.

 

There has been a lot of work around this technology in the past few years, including efforts to change C and Java, and adhoc libraries for C/C++, Java and Python. These changes are our efforts to make persistent memory accessible to Go developers.

 

Thanks!

Mohit

 

From: Robert Engels <ren...@ix.netcom.com>
Date: Wednesday, April 3, 2019 at 4:00 PM
To: Jerrin Shaji George <jshaji...@vmware.com>
Cc: "golan...@googlegroups.com" <golan...@googlegroups.com>, Mohit Verma <moh...@vmware.com>, Pratap Subrahmanyam <pra...@vmware.com>, Rajesh Venkatasubramanian <vra...@vmware.com>
Subject: Re: [go-nuts] Persistent memory support for Go

 

Also, I’ve used pmem for a very long time (I have a bg in retail point of sale systems where it is very common) and we always had a driver/library on top of the physical device to multiplex across consumers, and we used it from Java, and never needed specialized JVM to use it efficiently. Again, just curious, maybe the use cases have changed. 

[Robert Engels]

And no need to be sorry. You didn’t do anything wrong at all. 

[r...@golang.org]

Out of curiosity what HW/OS is this being developed on? I need new HW and might as well get the same since it will make playing around with this smoother.

[Jerrin Shaji George]

Hi,

 

Sorry I missed this email as I am not subscribed to email updates in this group.

 

This project currently only supports being built on Linux 64 bit. It is being developed on a machine with Intel Optane DC Persistent Memory

(https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html)

 

If persistent memory is not available, it can also be emulated using DRAM. Please see instructions at https://pmem.io/2016/02/22/pm-emulation.html.

 

Thanks,

Jerrin

 

PS: please reply-all when replying to this message