[ANN] gommap for Go 1

519 views
Skip to first unread message

Gustavo Niemeyer

unread,
Apr 17, 2012, 10:06:27 PM4/17/12
to golang-nuts
Sorry, this one took a while. I've just updated gommap to Go 1, and
also polished it a little bit further so that it uses properly typed
variables for each of the flag sets and uses []bool for the result of
InCore.

Please note that gommap can eat your lunch. The garbage collector
doesn't manage the data that backs the []byte slice; unmapping it and
touching the slice itself or any other slices created based on it will
crash the application.

I've run tests on Ubuntu/amd64, and I believe it should work on 386 as
well. Tests and patches welcome.

The project is go-gettable at launchpad.net/gommap, and docs are available at:

http://godoc.labix.org/gommap

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/plus
http://niemeyer.net/twitter
http://niemeyer.net/blog

-- I'm not absolutely sure of anything.

Michael Jones

unread,
Apr 17, 2012, 11:07:01 PM4/17/12
to Gustavo Niemeyer, golang-nuts
Perhaps petty, but how about IsResident() rather than InCore()? I've
only ever programmed one machine with core memory, and it's been a
while. It was a CDC 3300 and had a 24-bit word size. Nice for its day,
but that day has long passed.

each one a bit of RAM:
http://www.computermuseum.li/Testpage/Core-Memory-CDC.htm

How:
http://www.cs.ubc.ca/~hilpert/e/coremem/index.html

--
Michael T. Jones | Chief Technology Advocate  | m...@google.com |  +1
650-335-5765

Thomas Bushnell, BSG

unread,
Apr 17, 2012, 11:36:07 PM4/17/12
to Michael Jones, golang-nuts, Gustavo Niemeyer

Reminds me of /dev/drum on BSD 4.3.

Michael Jones

unread,
Apr 18, 2012, 1:41:01 AM4/18/12
to Thomas Bushnell, BSG, golang-nuts
On Apr 17, 2012 10:39 PM, "Michael Jones" <m...@google.com> wrote:

Just realized that I've used a second machine, a CDC 7600 at Bell Labs, which also had core. Just once as a test of Kronos and NOS/BE RJE. Nothing since and that was 1977. It may be time to retire 'core'.

dlin

unread,
Apr 18, 2012, 1:52:21 AM4/18/12
to golan...@googlegroups.com

Could you put some read/write example on the doc page?

If possible the wrong example of the following  description is also useful.

IMPORTANT NOTE (1): The MMap type is backed by an unsafe memory region, which is not covered by the normal rules of Go's memory management. If a slice is taken out of it, and then the memory is explicitly unmapped through one of the available methods, both the MMap value itself and the slice obtained will now silently point to invalid memory. Attempting to access data in them will crash the application.

Jan Mercl

unread,
Apr 18, 2012, 2:51:21 AM4/18/12
to golan...@googlegroups.com
On Wednesday, April 18, 2012 4:06:27 AM UTC+2, Gustavo Niemeyer wrote:

... and docs are available at:

    http://godoc.labix.org/gommap

No docs there. Just a blank page. Let's look at the source:

<html><script>window.location="http://gopkgdoc.appspot.com/pkg/launchpad.net"+window.location.pathname+window.location.hash;</script><body></body></html>

I have Javascript disabled by default and no, I will not enable it for unknown (non trusted) sites, so the "docs" link is of no use for me and perhaps some others too. Is there some specific reason why not to simply post the direct link to gopkgdoc.appspot.com/whatever instead?


Gustavo Niemeyer

unread,
Apr 18, 2012, 10:07:14 AM4/18/12
to Michael Jones, golang-nuts
On Wed, Apr 18, 2012 at 00:07, Michael Jones <m...@google.com> wrote:
> Perhaps petty, but how about IsResident() rather than InCore()? I've

Sounds sensible. I was following the system call naming, but I've
already changed the result to be more friendly to Go programs, so that
should be good too.

I've just replaced and pushed.

Nice, thanks for the references. Quite amazing how simple the
mechanism turns out to be. I also find curious that to read the state
of a core it must be forced into a given state.

Gustavo Niemeyer

unread,
Apr 18, 2012, 10:28:13 AM4/18/12
to Jan Mercl, golan...@googlegroups.com
On Wed, Apr 18, 2012 at 03:51, Jan Mercl <0xj...@gmail.com> wrote:
> I have Javascript disabled by default and no, I will not enable it for
> unknown (non trusted) sites, so the "docs" link is of no use for me and
> perhaps some others too. Is there some specific reason why not to simply
> post the direct link to gopkgdoc.appspot.com/whatever instead?

Thanks for the note, Jan. I'm using it so that I can preserve links
working irrespective of what else happens. The reason to use
Javascript is to preserve the fragments across redirects. Please feel
free to go straight to gopkgdoc or to your local godoc site, though.

Jan Mercl

unread,
Apr 18, 2012, 11:24:48 AM4/18/12
to golan...@googlegroups.com, Jan Mercl
On Wednesday, April 18, 2012 4:28:13 PM UTC+2, Gustavo Niemeyer wrote:

Thanks for the note, Jan. I'm using it so that I can preserve links
working irrespective of what else happens.

So, IIUC, to "preserve links working irrespective of what else happens" is implemented by making them not working preemptively. Interesting.

-jan 

 

Gustavo Niemeyer

unread,
Apr 18, 2012, 11:48:46 AM4/18/12
to Jan Mercl, golan...@googlegroups.com
On Wed, Apr 18, 2012 at 12:24, Jan Mercl <0xj...@gmail.com> wrote:
> So, IIUC, to "preserve links working irrespective of what else happens" is
> implemented by making them not working preemptively. Interesting.

Sorry about that. I'll add a visible link so people still living in
the 90s can click on it.

Message has been deleted

Jan Mercl

unread,
Apr 18, 2012, 12:08:42 PM4/18/12
to golan...@googlegroups.com, Jan Mercl
On Wednesday, April 18, 2012 5:48:46 PM UTC+2, Gustavo Niemeyer wrote:
On Wed, Apr 18, 2012 at 12:24, Jan Mercl wrote:
> So, IIUC, to "preserve links working irrespective of what else happens" is
> implemented by making them not working preemptively. Interesting.

Sorry about that. I'll add a visible link so people still living in
the 90s can click on it.

That's what I am! ;-)

Actually, in the 90s there was yet very few reasons to disable JS in a browser. Nowadays it's all different. I routinely blacklist non trusted sites trying to send me somewhere else w/o my consent (some browsers catch such attempts, don't they?), so no worries about me.

Michael Jones

unread,
Apr 18, 2012, 12:58:29 PM4/18/12
to Gustavo Niemeyer, golang-nuts
DRAMs have a similar destructiveness.

--

Gustavo Niemeyer

unread,
Apr 18, 2012, 1:16:13 PM4/18/12
to Michael Jones, golang-nuts
On Wed, Apr 18, 2012 at 13:58, Michael Jones <m...@google.com> wrote:
> DRAMs have a similar destructiveness.

As I recall DRAMs have to be refreshed, but they don't have to be
forced into a state to read the side effects of that. But then, my
electronics are far from trustworthy.

Michael Jones

unread,
Apr 18, 2012, 1:26:43 PM4/18/12
to Gustavo Niemeyer, golang-nuts
you are right. there were (are) some DRAM's where reading was
semi-destructive so the refresh cycle was really a
read-and-leave-uncertain, then write back to make certain. That
changed, though, over time and recent ones actually hold data for a
LONG time. (i.e. after a power outage for multiple seconds and longer,
compared to microsecond refresh)

--

lamvak

unread,
Apr 19, 2012, 1:47:36 AM4/19/12
to golang-nuts
Do you use still launchpad gommap's site for issues reporting or
feature requests? Documentations leads there.
> Gustavo Niemeyerhttp://niemeyer.nethttp://niemeyer.net/plushttp://niemeyer.net/twitterhttp://niemeyer.net/blog

Gustavo Niemeyer

unread,
Apr 19, 2012, 5:05:06 AM4/19/12
to lamvak, golang-nuts
On Thu, Apr 19, 2012 at 02:47, lamvak <lam...@gmail.com> wrote:
> Do you use still launchpad gommap's site for issues reporting or
> feature requests? Documentations leads there.

Yep, still maintained there, and I did see your ticket about Go 1 some
time ago. It's more that this project was behind than where it is
maintained.

Is it working for you now?

minux

unread,
Apr 19, 2012, 7:10:09 AM4/19/12
to Gustavo Niemeyer, Michael Jones, golang-nuts
On Wed, Apr 18, 2012 at 10:07 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
> How:
Nice, thanks for the references. Quite amazing how simple the
mechanism turns out to be. I also find curious that to read the state
of a core it must be forced into a given state.
In fact, several RAM technologies all use this kind of destructive-read.
AFAIK, the technique is used by as old as Williams tube, all the way to DRAM, and now FeRAM.

Gustavo Niemeyer

unread,
Apr 19, 2012, 8:38:02 AM4/19/12
to minux, Michael Jones, golang-nuts
On Thu, Apr 19, 2012 at 08:10, minux <minu...@gmail.com> wrote:
> In fact, several RAM technologies all use this kind of destructive-read.
> AFAIK, the technique is used by as old as Williams tube, all the way to
> DRAM, and now FeRAM.

Do you have a reference?

minux

unread,
Apr 19, 2012, 8:49:03 AM4/19/12
to Gustavo Niemeyer, Michael Jones, golang-nuts
On Thu, Apr 19, 2012 at 8:38 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
On Thu, Apr 19, 2012 at 08:10, minux <minu...@gmail.com> wrote:
> In fact, several RAM technologies all use this kind of destructive-read.
> AFAIK, the technique is used by as old as Williams tube, all the way to
> DRAM, and now FeRAM.

Do you have a reference?

Gustavo Niemeyer

unread,
Apr 19, 2012, 9:13:51 AM4/19/12
to minux, Michael Jones, golang-nuts
On Thu, Apr 19, 2012 at 09:49, minux <minu...@gmail.com> wrote:
> http://en.wikipedia.org/wiki/Dynamic_random-access_memory#Operations_to_read_a_data_bit_from_a_DRAM_storage_cell

As I mentioned before in this thread, it talks about refreshing, which
isn't the same as forcing a cell into a state to read the side effects
of it. I may be missing something, but the link hasn't fixed my
blindness.

minux

unread,
Apr 19, 2012, 10:30:22 AM4/19/12
to Gustavo Niemeyer, Michael Jones, golang-nuts
On Thu, Apr 19, 2012 at 9:13 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
On Thu, Apr 19, 2012 at 09:49, minux <minu...@gmail.com> wrote:
> http://en.wikipedia.org/wiki/Dynamic_random-access_memory#Operations_to_read_a_data_bit_from_a_DRAM_storage_cell

As I mentioned before in this thread, it talks about refreshing, which
isn't the same as forcing a cell into a state to read the side effects
of it. I may be missing something, but the link hasn't fixed my
blindness.
Note that we are now using 1T DRAM cell almost exclusively, which has destructive read.

Gustavo Niemeyer

unread,
Apr 19, 2012, 10:38:17 AM4/19/12
to minux, Michael Jones, golang-nuts
On Thu, Apr 19, 2012 at 11:30, minux <minu...@gmail.com> wrote:
> http://docencia.ac.upc.edu/master/DTM/docs/04-Memory%20Structures-2.pdf
> Note that we are now using 1T DRAM cell almost exclusively, which has
> destructive read.

"read and refresh operations are necessary for correct operation."

The document is great, btw, but this is still not the same as forcing
the state to be able to read it as a side-effect. I'll stop repeating
that now since we're well out of topic.

Hotei

unread,
Apr 19, 2012, 12:12:16 PM4/19/12
to golan...@googlegroups.com, minux, Michael Jones
Gustavo,
Think of it as a door.  You want to know if the door is locked or unlocked.  So you push on it.  The door opens -> it was unlocked.  The door doesn't open -> it was locked.  This is not refresh, this is force a state (door is now always going to be open) in order to read the previous state (locked or unlocked).   If you want to add refresh, all you do is write back the state you just read (close the door and lock or unlock it using previous state).

Maybe a dumb analogy, but it works for me. BTW,  I worked on two core based systems, NCR Century 100 was what I used in high school.  CDC 6600 was my freshman year of college.  NCR wasn't really core, but it used ferrite rods that replaced donut core.  Not sure if NCR used a 3-wire write to read method or not. I think CDC-6600 used donut core but I never got close enough to inspect the hardware.   InCore() brings back good memories.

Gustavo Niemeyer

unread,
Apr 19, 2012, 1:08:28 PM4/19/12
to Hotei, golan...@googlegroups.com, minux, Michael Jones
On Thu, Apr 19, 2012 at 13:12, Hotei <hote...@gmail.com> wrote:
> Think of it as a door.  You want to know if the door is locked or unlocked.
>  So you push on it.  The door opens -> it was unlocked.  The door doesn't
> open -> it was locked.  This is not refresh, this is force a state (door is
> now always going to be open) in order to read the previous state (locked or
> unlocked).   If you want to add refresh, all you do is write back the state
> you just read (close the door and lock or unlock it using previous state).

Thanks, that's what I was trying to say as well, in an unclear way admittedly.

> Maybe a dumb analogy, but it works for me. BTW,  I worked on two core based
> systems, NCR Century 100 was what I used in high school.  CDC 6600 was my
> freshman year of college.  NCR wasn't really core, but it used ferrite rods
> that replaced donut core.  Not sure if NCR used a 3-wire write to read
> method or not. I think CDC-6600 used donut core but I never got close enough
> to inspect the hardware.   InCore() brings back good memories.

Oh, nice. It's quite telling that you *could* get close enough to
inspect the hardware! :-)

Michael Jones

unread,
Apr 19, 2012, 1:56:16 PM4/19/12
to Hotei, golan...@googlegroups.com, minux
Hotei, the CDC 6600 and 7600 used core. The Cyber 170 was the first
with semiconductor memory.

On Thu, Apr 19, 2012 at 12:12 PM, Hotei <hote...@gmail.com> wrote:
:


> I think CDC-6600 used donut core but I never got close enough
> to inspect the hardware.   InCore() brings back good memories.

--

lamvak

unread,
May 2, 2012, 4:16:53 PM5/2/12
to golan...@googlegroups.com, lamvak
Hi.
Yes, now it installs without a fuss and loads a file. Sorry for the delay, though.
And I have another issue - but I think it might be issue with something else than gommap.
I cannot write to a file through a mmaped slice. I did not forget to Sync the mmap. I tried to make the calls through syscall to create and sync mmaped slice and it didn't work either. The code:
####
package main
import (
        `fmt`
        `os`
)
func main() {
        file, _ := os.OpenFile(`test.file`, os.O_RDWR, 0666)
        mmap, _ := gommap.Map(file.Fd(), gommap.PROT_READ | gommap.PROT_WRITE, gommap.MAP_PRIVATE)
        fmt.Printf("mmap: %s\n", mmap)
        mmap[1] = byte('q')
        fmt.Printf("mmap: %s\n", mmap)
        mmap.Sync(gommap.MS_SYNC)
}
#####
file "test.file" contains just the string abcd
after running the code from above it should contain the string aqcd, but doesn't change.
What do I do wrong?

Kyle Lemons

unread,
May 2, 2012, 4:33:43 PM5/2/12
to lamvak, golan...@googlegroups.com
On Wed, May 2, 2012 at 1:16 PM, lamvak <lam...@gmail.com> wrote:
Hi.
Yes, now it installs without a fuss and loads a file. Sorry for the delay, though.
And I have another issue - but I think it might be issue with something else than gommap.
I cannot write to a file through a mmaped slice. I did not forget to Sync the mmap. I tried to make the calls through syscall to create and sync mmaped slice and it didn't work either. The code:
####
package main
import (
        `fmt`
        `os`
)
func main() {
        file, _ := os.OpenFile(`test.file`, os.O_RDWR, 0666)
        mmap, _ := gommap.Map(file.Fd(), gommap.PROT_READ | gommap.PROT_WRITE, gommap.MAP_PRIVATE)

Underscores should be the first thing you look at when something is misbehaving.

lamvak

unread,
May 2, 2012, 4:45:06 PM5/2/12
to golan...@googlegroups.com, lamvak
That wasn't too helpful. I would not paste it if that would be of any help.

Devon H. O'Dell

unread,
May 2, 2012, 4:49:48 PM5/2/12
to lamvak, golan...@googlegroups.com
Op 2 mei 2012 16:45 heeft lamvak <lam...@gmail.com> het volgende geschreven:
> That wasn't too helpful. I would not paste it if that would be of any help.

So if you s/_/err/, and then add:

if err != nil {
fmt.Println(e.Error())
return
}

under the calls to OpenFile and Map, main still completes and you get
no error output that you could provide us to actually give you some
help?

--dho

lamvak

unread,
May 2, 2012, 4:54:11 PM5/2/12
to golan...@googlegroups.com, lamvak
func main() {
        file, err := os.OpenFile(`test.file`, os.O_RDWR, 0666)
        if err != nil {
                fmt.Printf("err: %s\n", err.Error())
        }
        mmap, err := gommap.Map(file.Fd(), gommap.PROT_READ | gommap.PROT_WRITE, gommap.MAP_PRIVATE)
        if err != nil {
                fmt.Printf("err: %s\n", err.Error())
        }
        fmt.Printf("mmap: %s\n", mmap)
        mmap[1] = byte('q')
        fmt.Printf("mmap: %s\n", mmap)
        err = mmap.Sync(gommap.MS_SYNC)
        if err != nil {
                fmt.Printf("err: %s\n", err.Error())
        }
        fmt.Printf("mmap: %s\n", mmap)
        file.Close()
}
gives:
----- 
mmap: abcd

mmap: aqcd

mmap: aqcd

------
So, still nothing.

Dave Cheney

unread,
May 2, 2012, 6:02:16 PM5/2/12
to lamvak, golan...@googlegroups.com, lamvak
I'm away from my terminal atm, but I think MAP_PRIVATE is the culprit. 

lamvak

unread,
May 2, 2012, 6:26:06 PM5/2/12
to golan...@googlegroups.com, lamvak
Well, that's a surprise for me!
Now, changing MAP_PRIVATE to MAP_SHARED did the trick. Penny for thoughts though, how and why? Might that be caused by the map being created in one thread/process and somehow used in another? Is there a particular doc to wrap head around this properly?
Ah, and thank you very much!

Dave Cheney

unread,
May 2, 2012, 6:28:11 PM5/2/12
to lamvak, golan...@googlegroups.com
Just a hunch,

http://linux.die.net/man/2/mmap

Cheers

Dave

lamvak

unread,
May 3, 2012, 2:27:24 AM5/3/12
to golan...@googlegroups.com, lamvak
The sad thing: I've checked that man earlier. My reading skills astonish me, at times.
Thanks again.
Reply all
Reply to author
Forward
0 new messages