which one came first

[Joe keane]

[Unix] shared memory or memory-mapped files

Pretty much everyone has both these days (different interface to common
function). What systems supported one but not the other?

Also when did people get the idea to implement 'malloc' in terms of
'mmap' rather than [somewhat awkward] 'sbrk'?

(feel free to comment on other operating systems)

[David Dyer-Bennet]

TOPS-20 had memory-mapped files as the preferred form of IO. I never
wrote for TENEX so I can't confirm it goes back that far from personal
knowledge.
--
David Dyer-Bennet, dd...@dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

[Scott Lurndal]

j...@panix.com (Joe keane) writes:
>[Unix] shared memory or memory-mapped files

For Unix systems, shared memory came first. The system V IPC subsystem which was
designed by a group at Bell Labs included shm, msg queues and semaphores. This
was eventually bolted on to unix (with an oddball namespace).

mmap came with SunOS and was merged into SVR4 when sun and USL developed
SVR4 jointly.

Posix later specified its own form of shared memory, message queues and
sync primitives which used a 'path-like' namespace.

>Also when did people get the idea to implement 'malloc' in terms of
>'mmap' rather than [somewhat awkward] 'sbrk'?

With SVR4, mmap of /dev/zero was used to allocate anonymous space (linux
later added MAP_ANON so you didn't have to open /dev/zero first) and became
the underpinnings of brk/sbrk.

Of course, with create and map segment, VMS had memory mapped files from
the start.

scott

[Barry Margolin]

In article <ylfkty0a...@dd-b.net>,

David Dyer-Bennet <dd...@dd-b.net> wrote:

> j...@panix.com (Joe keane) writes:
>
> > [Unix] shared memory or memory-mapped files
> >
> > Pretty much everyone has both these days (different interface to common
> > function). What systems supported one but not the other?
> >
> > Also when did people get the idea to implement 'malloc' in terms of
> > 'mmap' rather than [somewhat awkward] 'sbrk'?
> >
> > (feel free to comment on other operating systems)
>
> TOPS-20 had memory-mapped files as the preferred form of IO. I never
> wrote for TENEX so I can't confirm it goes back that far from personal
> knowledge.

If you're going to count non-Unix systems, Multics had memory-mapped
files back in the 60's. In fact, that was the ONLY way to access files
on Multics -- the equivalents of open(), read() and write() were in a
library that used memory mapping.

Multics was the system that inspired Unix, but they didn't copy this
aspect, I think because the early Unix hardware didn't have sufficient
virtual memory features.

--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

[Anne & Lynn Wheeler]

Barry Margolin <bar...@alum.mit.edu> writes:
> If you're going to count non-Unix systems, Multics had memory-mapped
> files back in the 60's. In fact, that was the ONLY way to access files
> on Multics -- the equivalents of open(), read() and write() were in a
> library that used memory mapping.
>
> Multics was the system that inspired Unix, but they didn't copy this
> aspect, I think because the early Unix hardware didn't have sufficient
> virtual memory features.

tss/360 ("official" system for 360/67) was all memory mapped files
... but wasn't very smart about it ... just mapped and then lots of 4k
page faults. along with other inefficiencies ... tss/360 ran a four user
fortran compile & execute slower than cp67/cms ran equivalent 30 user
fortran compile & execute (on identical hardware). tss/360 never did get
very far and was decommitted ... although a few customers did managed to
keep in alive and justify tss/370. Later in the early 80s, there was a
special project for AT&T to take a stripped down tss/370 kernel (SSUP)
that would have unix layer on top.

in the first half of the 70s, company did Future System effort ... which
was going to replace 360/370 ... and was suppose to be all memory mapped.

i was at the science center on 4th flr (multics was on the 5th flr) and
did a memory mapped file system for cms. I had learned quite a bit what
"not to do" watching tss/360 ... I also figured that i could also do as
well as people on the 5th flr (as well as critical of future system not
really knowing what they were doing ... drawing analogy with long
playing cult film down at central sq). misc. past posts mentioning
545 tech sq
http://www.garlic.com/~lynn/subtopic.html#545tech

misc. past posts doing cms memory mapped filesystem ... was deployed
internally, but didn't make it into product.
http://www.garlic.com/~lynn/submain.html#mmap

I could get 3 times the thruput (of standard cms filesystem) for
moderate i/o application. part of the issue was cms running in virtual
machine used channel programs for i/o (including file i/o). 360/370
channel programs were "real" ... so channel programs created in virtual
machine had to be "copied" into real channel programs with real
addresses. Going to memory mapped filesystem allowed me to eliminate a
whole layer of emulation and enabled a doing a lot of optimization (that
wasn't possible with a channel program emulation paradigm).

however in the wake of both tss/360 and future system memory map issues
... there was lots of resistance to memory map paradigm (even if there
was lots of internal sites running it and showing improved performance).

--
virtualization experience starting Jan1968, online at home since Mar1970

[Peter Flass]

OS/2 had (maybe has) shared memory but not memory-mapped files.

--
Pete

[Peter Flass]

Actually OS/2 mmaps the executables. They page directly from the LX file.

--
Pete

[Anne & Lynn Wheeler]

re:
http://www.garlic.com/~lynn/2012f.html#28 which one came first

360/67 had 1mbyte segments, 256 4k pages ... and 24bit and 32bit virtual
addressing modes (16 1mbyte and 4096 1mbyte)

370 just had 24bit virtual addressing ... but had 1mbyte and 64kbyte
virtual segment sizes and 4kbyte and 2kbyte virtual page size options.

the original 370 virtual memory specification had a bunch of additional
options ... included r/o shared segment bit in segment table entries.
370s were announced and shipped before virtual memory feature was
announced ... virtual memory later had to be retrofitted. most machines
implemented the full 370 virtual memory specification ... and in the
cp67/cms to vm370/cms transition, cms was reorganized to make use of the
64kbyte "read-only" (protected) shared segments (16 4kbyte shared
pages). problem arose retrofitting the full 370 virtual memory
specification to 370/165 ... and in order to get back six months in
schedule ... it was decided to drop several virtual memory features
... included r/o shared segment protection (370 models that had already
implemented the full set of features had to go back and remove the
dropped features).

vm370/cms was then stuck with how to have protected shared segments when
the hardware support had been dropped.

360 (and 370) had 2k "storage-protect" keys ... left over from 360
real-storage protection ... 4bit key for every 2kbyte of storage and PSW
field ... if PSW field matched the storage key ... store operations were
allowed (psw key of zero turned off store protect).

vm370/cms retreated to real hack to emulate segment protect ... cms
virtual machine with any shared "segments" always had its psw key set to
"15" and all non-shared storage set to "15" ... and all 2k areas in
"shared segment" were set to zero (preventing all stores in shared
pages).

w/o memory-mapped filesystem ... cp67 & vm370 had a real hack ... special
area in paging system ... appropriate privileges could execute "savesys"
command that wrote image of virtual pages to the special area. The
machine "IPL" command was allowed to be a name of the special area (as
alternative to device address) ... which would map virtual memory to the
saved named area. "named systems" have special configuration specifying
the pages saved/restored ... and the segments designated as "shared"
(& protected)

i could elimiante all that in cp67 with memory-mapped filesystem ...
and then moved the support to vm370 base. One of the big problems I
constantly fought was os/360 "relocatable address constants". CMS
borrowed compilers and applications from os/360 ... as well as many
conventions. os/360 relocatable address constants were application on
disk ... the application is fully read into (emulated real) storage
... and all relocatable address constants were swizzled to correspond
with the address where application was loaded.

memory-mapped filesystem should allow executiable applications mapped to
any available addres w/o having to thread way through lots of the image
... swizzling all the address constants (making them absolute). Swizzled
"absolute" addresses (after loading) also results in problems with
sharing exact same image in different virtual address spaces.

tss/360 addressed the design by having the address fields in application
images as relative to some virtual address space specific field. For me,
I was constantly battling the os/360 relocatable address constant
convention used by cms ... misc. past posts
http://www.garlic.com/~lynn/submain.html#adcon

[Jeff Jonas]

>>[Unix] shared memory or memory-mapped files

> For Unix systems, shared memory came first.
> The system V IPC subsystem which was
> designed by a group at Bell Labs included
> shm, msg queues and semaphores.

Long long ago on a PDP11/45 not so far away,
I used Unix version 6.
The only inter process communication was anonymous pipes
via pipe(2), which was used by the shell.

I don't remember what Unix System III offered for IPC,
so I'll have to agree that by AT&T Unix System V
(pronounced "system five"), IPC was the big 3:
shared memory, message queues and semaphores.

>This was eventually bolted on to unix (with an oddball namespace).

No argument there. Most of today's Linux systems still have them
for compatibility and legacy reasons.
Just try
ipcs -a
and see how every item has an ID and key
that's in a private namespace, not accessable via a name.

[sarr.b...@alum.dartmouth.org]

In alt.folklore.computers Jeff Jonas <je...@panix.com> wrote:

: Long long ago on a PDP11/45 not so far away,

: I used Unix version 6.
: The only inter process communication was anonymous pipes
: via pipe(2), which was used by the shell.

Didn't v6 have the fork/exec pair. V7 certainly did. And in
that sense shared memory was easy; a fork with no exec.

--
--------
Sarr Blumson sarr.b...@alum.dartmouth.org
http://www-personal.umich.edu/~sarr/

[Barry Margolin]

In article <jotmla$lq2$1...@dont-email.me>,

sarr.b...@alum.dartmouth.org wrote:

> In alt.folklore.computers Jeff Jonas <je...@panix.com> wrote:
>
> : Long long ago on a PDP11/45 not so far away,
> : I used Unix version 6.
> : The only inter process communication was anonymous pipes
> : via pipe(2), which was used by the shell.
>
> Didn't v6 have the fork/exec pair. V7 certainly did. And in
> that sense shared memory was easy; a fork with no exec.

Unix has had fork/exec practically since day 1. But when you fork, the
child process gets a COPY of your memory, it's not shared. Exec is
irrelevant in this discussion.

[Ahem A Rivet's Shot]

On Tue, 15 May 2012 13:45:14 +0000 (UTC)
sarr.b...@alum.dartmouth.org wrote:

> In alt.folklore.computers Jeff Jonas <je...@panix.com> wrote:
>
> : Long long ago on a PDP11/45 not so far away,
> : I used Unix version 6.
> : The only inter process communication was anonymous pipes
> : via pipe(2), which was used by the shell.
>
> Didn't v6 have the fork/exec pair. V7 certainly did. And in
> that sense shared memory was easy; a fork with no exec.

No, a fork with no exec still creates a new process with a separate
memory map (almost identical at the time fork returns). Usually
copy-on-write is used to make it efficient, I'm not sure how far back that
goes. Fork itself goes back to the PDP-7 unix see
http://cm.bell-labs.com/who/dmr/hist.html.

--
Steve O'Hara-Smith | Directable Mirror Arrays
C:>WIN | A better way to focus the sun
The computer obeys and wins. | licences available see
You lose and Bill collects. | http://www.sohara.org/

[Charles Richmond]

"Ahem A Rivet's Shot" <ste...@eircom.net> wrote in message
news:20120515152908....@eircom.net...

> On Tue, 15 May 2012 13:45:14 +0000 (UTC)
> sarr.b...@alum.dartmouth.org wrote:
>
>> In alt.folklore.computers Jeff Jonas <je...@panix.com> wrote:
>>
>> : Long long ago on a PDP11/45 not so far away,
>> : I used Unix version 6.
>> : The only inter process communication was anonymous pipes
>> : via pipe(2), which was used by the shell.
>>
>> Didn't v6 have the fork/exec pair. V7 certainly did. And in
>> that sense shared memory was easy; a fork with no exec.
>
> No, a fork with no exec still creates a new process with a separate
> memory map (almost identical at the time fork returns). Usually
> copy-on-write is used to make it efficient, I'm not sure how far back that
> goes. Fork itself goes back to the PDP-7 unix see
> http://cm.bell-labs.com/who/dmr/hist.html.
>

Yes, it's my understanding that in bringing up Unix, the "init" process is
the only one *not* fork/exec'ed. (Is that process ID 0???) All other
processes are decendants of this one. So the whole system kind of builds up
by fork'ing and exec'ing.

--

numerist at aquaporin4 dot com

[Casper H.S. Dik]

"Charles Richmond" <nume...@aquaporin4.com> writes:

>"Ahem A Rivet's Shot" <ste...@eircom.net> wrote in message
>news:20120515152908....@eircom.net...

>> No, a fork with no exec still creates a new process with a separate
>> memory map (almost identical at the time fork returns). Usually
>> copy-on-write is used to make it efficient, I'm not sure how far back that
>> goes. Fork itself goes back to the PDP-7 unix see
>> http://cm.bell-labs.com/who/dmr/hist.html.

vfork() was introduced because of a bug in the VAX-730 because the
copy-on-write thing didn't always work. Before that, Unix wasn't
demand paged, it was swap based and there was sharing of pages
between processes.

>Yes, it's my understanding that in bringing up Unix, the "init" process is
>the only one *not* fork/exec'ed. (Is that process ID 0???) All other
>processes are decendants of this one. So the whole system kind of builds up
>by fork'ing and exec'ing.

Process "1". Process 0 is the virtual process that is the kernel.

Casper

[Ahem A Rivet's Shot]

That's how I understand it too, but init is pid 1 so that 0 is
available for "no process" and negative for errors in code dealing with
pids - at least that's my theory.

[John Levine]

>vfork() was introduced because of a bug in the VAX-730 because the
>copy-on-write thing didn't always work. Before that, Unix wasn't
>demand paged, it was swap based and there was sharing of pages
>between processes.

VAX-11/750 actually, read-only pages in the stack didn't work. In
retrospect a simple way to patch around it would have to make stack
pages copy-on-touch which would have certainly worked . It's a rare
stack page that's touched and not written so there'd be little
performance difference, but I guess Bill thought of the brute force
approach first.

Shared memory came surprisingly late to Unix. First there was the SHM
kludge which I gather was invented for one of the phone switches. Real
shared memory integrated into the pager didn't come along until 4.4BSD.

R's,
John
--
Regards,
John Levine, jo...@iecc.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly