GrayBox
Leif Strand
[switching to classic-mac-dev]
Cameron Kaiser wrote:
> By the way, I am very impressed by GrayBox.
Thanks! It was, and is, much more work than I thought it was going to
be. (Isn't that always the truth?)
I thought the transparency thing (to make classic windows look like
peers to native windows) would be easy, but it turned out to be quite
frustrating and messy. In fact, I'm planning to throw out that code and
replace it with a WDEF-based approach: I think I can bridge a Carbon
custom window definition to the emulated M68K WDEFs underneath. That's
the easy part, actually; the hard part is, it requires full QuickDraw
emulation so that M68K QuickDraw calls end up calling Carbon QuickDraw
when drawing to the screen.
But the thing that really blindsided me was the File Manager. "Easy", I
thought, "I'll just intercept the File Manager's A-line traps, just as
if they were Unix system calls, and make them call into to a Carbon
'kernel' instead." If only it were that easy! On the M68K side, apps
and Standard File make direct memory accesses to File Manager data
structures (e.g., Volume Control Blocks), which of course were fully
documented in _Inside Macintosh_ (d'oh!). On the Carbon side, I was
surprised to discover that Apple had eliminated "working directories",
so I had to emulate those. That's GrayBox's biggest weakness at
present: my File Manager implementation is only good enough for
Standard File... it's still pretty crappy... HFS-aware software, like
HyperCard, simply does not work.
Joshua Juran wrote:
> I've only looked at the source code (since there doesn't appear to be
> a downloadable binary) but it certainly looks promising. It should
> be possible to back-port it to OS 9 by replacing just the app shell,
> and unless I'm missing something a Mach-O build shouldn't be too hard
> either, and even native-Intel isn't out of the question, though
> obviously that will require some byte-swapping. The latter is less
> important, as performance isn't likely to be an issue, but enabling
> the use of a freeware toolchain will open up development beyond just
> those with CodeWarrior.
All true. I'm planning to do an OS 9 port myself. OS 9 will be easier
than Carbon/OS neXt in many ways, because many of the aforementioned
problems do not exist. OS 9 has both B&W QuickDraw and HFS working
directories, so I can simply use them!
"Why?", one might ask. MacDraw (for instance) runs just fine on Mac OS
9 -- and it runs quickly, too! Much faster than under OS X/GrayBox.
Well...
This may sound insane, but GrayBox is actually a proof-of-concept for a
still more ambitious project: adding memory protection to classic Mac
OS. After all, on OS X, once I have a bunch of classic apps each
running in their own protected memory space -- with no hardware or
driver emulation whatsoever -- it isn't much of a leap to imagine a new
MultiFinder implementation which fork()s. Despite all the technical
difficulties I encountered with GrayBox, I still believe I can get
memory protection working under System 6, and hopefully System 7. Even
if only 10% of classic apps work in this new environment, that's still
better than what Snow Leopard gives you, which is 0%.
--Leif
Joshua Juran
> Cameron Kaiser wrote:
>> By the way, I am very impressed by GrayBox.
>
> Thanks! It was, and is, much more work than I thought it was going
> to be. (Isn't that always the truth?)
Well, I've been working on MacRelix for over a decade, but since I
didn't have a clear end goal in mind, I never thought about when I
expected it to be 'done'. Just last night I decided it made sense to
port FUSE to MacRelix, so I could then port the GUI filesystem to FUSE
and have it work on OS X (in MacFUSE) as well as MacRelix, as well as
take advantage of existing filesystems targeting FUSE.
> I thought the transparency thing (to make classic windows look like
> peers to native windows) would be easy, but it turned out to be
> quite frustrating and messy. In fact, I'm planning to throw out
> that code and replace it with a WDEF-based approach: I think I can
> bridge a Carbon custom window definition to the emulated M68K WDEFs
> underneath. That's the easy part, actually; the hard part is, it
> requires full QuickDraw emulation so that M68K QuickDraw calls end
> up calling Carbon QuickDraw when drawing to the screen.
As opposed to classic QuickDraw running in emulation? On the plus
side, Having factored QuickDraw calls into native space, they could be
reimplemented for a framebuffer or in terms of other graphics systems
like Quartz and GTK.
One other benefit would be supporting monochrome QuickDraw's eight
colors. :-)
> But the thing that really blindsided me was the File Manager.
> "Easy", I thought, "I'll just intercept the File Manager's A-line
> traps, just as if they were Unix system calls, and make them call
> into to a Carbon 'kernel' instead." If only it were that easy! On
> the M68K side, apps and Standard File make direct memory accesses to
> File Manager data structures (e.g., Volume Control Blocks), which of
> course were fully documented in _Inside Macintosh_ (d'oh!). On the
> Carbon side, I was surprised to discover that Apple had eliminated
> "working directories", so I had to emulate those. That's GrayBox's
> biggest weakness at present: my File Manager implementation is only
> good enough for Standard File... it's still pretty crappy... HFS-
> aware software, like HyperCard, simply does not work.
Basilisk II uses the File System Manager (FSM) for this purpose but
the result is buggy, and I think it requires System 7.1 or later. It
looks like you'll have to emulate volume control blocks and file
control blocks.
> Joshua Juran wrote:
>> I've only looked at the source code (since there doesn't appear to be
>> a downloadable binary) but it certainly looks promising. It should
>> be possible to back-port it to OS 9 by replacing just the app shell,
>> and unless I'm missing something a Mach-O build shouldn't be too hard
>> either, and even native-Intel isn't out of the question, though
>> obviously that will require some byte-swapping. The latter is less
>> important, as performance isn't likely to be an issue, but enabling
>> the use of a freeware toolchain will open up development beyond just
>> those with CodeWarrior.
>
> All true. I'm planning to do an OS 9 port myself. OS 9 will be
> easier than Carbon/OS neXt in many ways, because many of the
> aforementioned problems do not exist. OS 9 has both B&W QuickDraw
> and HFS working directories, so I can simply use them!
Reification for the win!
> "Why?", one might ask. MacDraw (for instance) runs just fine on Mac
> OS 9 -- and it runs quickly, too! Much faster than under OS X/
> GrayBox. Well...
There are a few things that don't work in Apple's 68K emulator, such
as CFM-68K, Open Transport timer tasks, and handling bus error
exceptions (as well as any MMU-reprogramming tricks). These aren't a
big deal, though, unless you are actually trying to protect memory.
> This may sound insane, but GrayBox is actually a proof-of-concept
> for a still more ambitious project: adding memory protection to
> classic Mac OS.
This has been done before (by MacMiNT, PATMOS, MachTen, and Jasik's
Debugger) for 68K apps running on a 68K box by reprogramming the MMU,
which is much more efficient than emulation (and would be critical to
performance on actual 68K Macs). On PowerPC, you're already emulating
anyway, so using VMs makes some sense. The tradeoff is that Apple's
emulator (in later OS versions) has JIT compiling. On the other hand,
so does Basilisk II, so maybe that could be adapted. I don't remember
the license offhand.
> After all, on OS X, once I have a bunch of classic apps each running
> in their own protected memory space -- with no hardware or driver
> emulation whatsoever
Because you're emulating further up the stack, at the level of Toolbox
call...
> -- it isn't much of a leap to imagine a new MultiFinder
> implementation which fork()s.
Wait, I don't follow you here. I'm not sure what you mean, since
fork() is a POSIX call and exists in a different context than Mac
processes. The two only cross when the Process Manager runs over Unix
(A/UX, OS X), or Unix is implemented over Mac OS (MachTen et all,
MacRelix). Okay, it looks like you're talking OS X again. Do you
mean implementing LaunchApplication(), so e.g. HyperCard could launch
another app? I don't see how fork() first into the picture.
I have thought about implementing fork() in MacRelix, though. The
first step is to bottleneck all process memory usage through mmap(),
which is already doable for stack and heap. Then enumerate each
unshared mapped region and create copies for the parent (assuming the
child runs first). If the child hasn't exited by the time the parent
runs, make new copies for the child and swap in the parent's copies.
The next time the child runs, swap in its version of the memory
blocks, etc., until one of them exits. Nasty, but it should work.
One possible improvement is reprogramming the MMU for
68020+851/030/040 (as has been done multiple times before), but it
would also be possible to use Mini vMac's extension mechanism for
virtual memory allocation, allowing (a) different processes to have
distinct blocks of real memory mapped into the same address space, and
(b) a multiple processes running on a 4MB virtual machine to each
allocate multiple multi-megabyte blocks -- although a program would
have be explicitly coded to take advantage of this, so realistically
we're talking only about MacRelix. But again, it should work, and
this time without swapping.
> Despite all the technical difficulties I encountered with GrayBox, I
> still believe I can get memory protection working under System 6,
> and hopefully System 7. Even if only 10% of classic apps work in
> this new environment, that's still better than what Snow Leopard
> gives you, which is 0%.
Don't you implicitly have memory protection by running in emulation?
I'm not sure what you're trying to do.
Josh
Leif Strand
> This has been done before (by MacMiNT, PATMOS, MachTen, and Jasik's
> Debugger) for 68K apps
Yeah, but the first three are all POSIX environments -- they don't offer
memory protection between Mac apps. My understanding of Jasik's MMU
thing is that it adds guard pages here and there; and maybe it protects
the stack and heap of other apps while your app is running... I don't
remember exactly... in any case, I'm quite sure it doesn't put each app
in its own address space, which is what I'm talking about.
I want nothing short of full memory protection between Classic Mac apps:
each app has its own address space. When an app crashes or goes
haywire, there is no danger of having to reboot; no messing with SIZE
resources... all that good stuff. My impression is that this is
generally regarded as impossible within the realm of Classic Mac OS, but
I've always rejected that line of thinking.
> Don't you implicitly have memory protection by running in emulation?
Exactly. Well, sort of. I find emulators easier to work with, but
emulation is irrelevant in the end.
I started my long journey with a thought experiment. Suppose I have two
Mac emulators/VMs running side-by-side on the same computer. The host
OS doesn't really matter, but let's assume the host OS is a "modern" OS
with memory protection. Each Mac emulator/VM is running a different Mac
app.
Now, have I not achieved memory protection between those two apps?
It might sound like I'm being silly or facetious, but I'm really not!
Obviously, two instances of vMac/Basilisk/SheepShaver/whatever are two
completely different worlds. So, what would it take to bring them
together into a single, coherent environment... without sacrificing the
memory protection between the two?
As I see it, merging the two worlds simply involves factoring out all
the stuff traditionally associated with a Unix kernel:
1) Factor out the filesystem, so that the two emulators/VMs share the
same disks.
2) Likewise, factor out networking. Each environment shares the same
network stack, IP address, etc.
3) Share the display/keyboard/mouse in full-screen mode.
4) Share the clipboard so you can copy & paste between apps. (Okay,
Unix kernels aren't responsible for the clipboard, but we're being
pragmatic here.)
Then, intercept LaunchApplication() so that it spawns a new
VM/emulator... i.e., a new process in the underlying host OS. Q.E.D.!
Classic Mac OS with memory protection. To hell with OS neXt -- I've
created my own alternative upgrade path!!! :-)
To elaborate:
1) Filesystem. GrayBox roughly demonstrates how this would be done:
intercept File Manager calls and trap into the kernel. Unlike almost
every other Mac emulator, GrayBox does not require disk images! It
boots off the host filesystem; therefore, the filesystem is shared
between GrayBox instances. Also, the experience of writing GrayBox
exposed a flaw in my thinking. I thought PBOpen/PBOpenRF would be the
equivalent of open(2) on Unix: I thought they would be system calls.
But, it turns out they are more like fopen(3) because of all of the File
Manager's public data structures. There has to be a user-mode File
Manager implementation that maintains all the data structures that apps
expect to see, like VCBs. The trap into kernel mode has to be buried
within this FM implementation (but still far above the device driver level).
2) Networking. I don't know much about MacTCP or Open Transport, but in
principle, it should work the same as the File Manager.
3) Display/keyboard/mouse. Again, GrayBox demonstrates how
keyboard/mouse would work: intercept GetOSEvent() and turn it into a
kernel trap. For the display, simply put the framebuffer into shared
memory. Each process gets its own instance of QuickDraw. Each process
gets its own instance of the Window Manager; the Window Manager code
would have to be tweaked so that apps don't clobber each other visually
(perhaps by exchanging visible regions using IPC).
4) Clipboard. Not sure yet. This might also involve shared memory.
Now, I say "kernel trap" but obviously, right now, in GrayBox, I'm just
intercepting certain opcodes in an M68K emulator. But to focus on the
emulation misses the point. In principle, I could be implementing a
kernel using native trap instructions on the host processor (whether it
be M68K or PowerPC).
The whole of the Mac ROM, the whole of Mac OS, becomes one gigantic
shared library. For the most part, each app gets its own copy of Mac
OS; each app is living in its own Mac VM. But, through highly selective
patching, a single, cohesive foundation is created underneath.
Forget about X11/NeXT-style "display servers". Forget about making
every damn data structure opaque, as Carbon does. Just do the minimum
amount necessary to obtain modern OS stability.
It is likely that I will never write my own kernel. That's beyond my
abilities, at least at present. So I'm content to rely on a host OS.
Darwin/OS X is a good choice, because (AFAIK) it is the only Unix that
can boot off HFS, and it is preinstalled on later Power Macs. Once I go
full screen, though, you'll never know that a host OS is underneath.
--Leif
Cameron Kaiser
>
> Exactly. Well, sort of. I find emulators easier to work with, but
> emulation is irrelevant in the end.
I did the same idea with my virtual 6502-on-6502 code (in my KIM-1 emulator).
It seemed the easiest way to "virtualize" a processor that was never
intended to be virtualized.
Besides, then you get certain things like apparent preemptive multitasking
for free, as long as the various emulator instances play nice with each
other, naturally. Run something like MachTen alongside and that would be
great stuff "once" it supports PowerPC.
--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- A Freudian slip is when you say one thing, but mean your mother. -----------
Joshua Juran
> I want nothing short of full memory protection between Classic Mac
> apps: each app has its own address space. When an app crashes or
> goes haywire, there is no danger of having to reboot; no messing
> with SIZE resources... all that good stuff. My impression is that
> this is generally regarded as impossible within the realm of Classic
> Mac OS, but I've always rejected that line of thinking.
Unless you hack the MMU, classic Mac OS has a flat address space
shared by all applications. But yeah, you can work around it by
booting one Mac OS instance per app.
> Joshua Juran wrote:
>
>> Don't you implicitly have memory protection by running in emulation?
>
> Exactly. Well, sort of. I find emulators easier to work with, but
> emulation is irrelevant in the end.
>
> I started my long journey with a thought experiment. Suppose I have
> two Mac emulators/VMs running side-by-side on the same computer.
> The host OS doesn't really matter, but let's assume the host OS is a
> "modern" OS with memory protection. Each Mac emulator/VM is running
> a different Mac app.
>
> Now, have I not achieved memory protection between those two apps?
I think we're in violent agreement. Your proof-of-concept package of
MacDraw demonstrates that a classic application can run preemptively
and with memory protection among other apps. As I see it, you've
already accomplished this in principle.
My question was, what was it you were intending to do with fork()?
> Then, intercept LaunchApplication() so that it spawns a new VM/
> emulator... i.e., a new process in the underlying host OS. Q.E.D.!
> Classic Mac OS with memory protection.
I think what you're getting at is booting Mac OS in emulation once, up
to the point of launching the Finder or alternate shell application,
then forking a child process for each application launched by the user.
I imagine there are applications that expect to poke around in the
memory of other processes, which need to be run in the same VM in
order to work. Off the top of my head I can think of ZoneRanger
(which is useless on its own) and MWDebug or the CodeWarrior IDE in
debugger mode. There will have to be some way for LaunchApplication()
to determine if it should spawn a new VM or just punt the trap back to
the emulator. Aside from debuggers, there may be apps that use
another process like a coroutine or send pointers via Apple events.
> To hell with OS neXt -- I've created my own alternative upgrade
> path!!! :-)
My ultimate goal (well, one of them) is to have applications that work
on any of classic Mac OS, OS X, and Linux. On the command-line side,
the path of least resistance is to implement POSIX in classic Mac OS.
The GUI side is more complex, and my approach there is to map the GUI
into the filesystem. MacRelix includes a prototype of this.
Josh
Joshua Juran
>>> Don't you implicitly have memory protection by running in emulation?
>>
>> Exactly. Well, sort of. I find emulators easier to work with, but
>> emulation is irrelevant in the end.
>
> I did the same idea with my virtual 6502-on-6502 code (in my KIM-1
> emulator).
> It seemed the easiest way to "virtualize" a processor that was never
> intended to be virtualized.
Which should work just as well with an OS that was never intended to
be virtualized.
> Besides, then you get certain things like apparent preemptive
> multitasking
> for free, as long as the various emulator instances play nice with
> each
> other, naturally.
Now that would be ironic, if the virtual environments intended to
protect apps from each other were themselves the source of trouble.
> Run something like MachTen alongside and that would be
I heard MachTen had robustness issues and occasionally ate
filesystems. I'd suggest using something actively supported, whose
developer fixes bugs. ;-)
> great stuff "once" it supports PowerPC.
You mean running classic apps built for PowerPC either virtualized or
emulated? That would be useful, since the less ancient Metrowerks
tools are PPC-only. It would be really nice if it supported processor
exceptions, so I can handle exceptions per thread and use the sc
instruction for trapping system calls.
Josh
Leif Strand
> I did the same idea with my virtual 6502-on-6502 code (in my KIM-1 emulator).
Wow -- I've just been reading up on the history. I cut my teeth on
Apple II assembly programming, but had never heard the story of the
KIM-1 before. Didn't even know there was a 6501. Cool stuff.
I once knew a guy who wrote an x86-on-x86 emulator called Chaperon, but
it was for Valgrind-style debugging on Linux, rather than
virtualization. Unfortunately, Chaperon was swallowed by the company I
worked for at the time.
--Leif
Leif Strand
> Unless you hack the MMU, classic Mac OS has a flat address space shared
> by all applications. But yeah, you can work around it by booting one
> Mac OS instance per app.
In a way, I plan to rewind history a bit. Each process space will look
like pre-System 7 without MultiFinder.
> My question was, what was it you were intending to do with fork()?
Sorry, I should have been more specific. I just meant, "host
environment creates a new process space" -- in response to
LaunchApplication().
> I think what you're getting at is booting Mac OS in emulation once, up
> to the point of launching the Finder or alternate shell application,
> then forking a child process for each application launched by the user.
Yup yup yup. I call this the "template process". Every time
LaunchApplication() is called, it just sends an IPC message to the
template process, which forks itself, and the spawned app is launched in
the new child. (This hasn't been a problem for GrayBox so far, because
System 6 boots really fast! So it simply boots it every time.)
On an unrelated subject, it would be possible to "punch a hole" in the
VM environment, thus giving access to the host OS. If the host OS is
Darwin, for example, one could imagine a PEF stub library that acts as a
bridge, allowing access to the underlying POSIX or Mach APIs. Of
course, the first app to use such a bridge would be a Classic version of
Terminal.app.
> I imagine there are applications that expect to poke around in the
> memory of other processes, which need to be run in the same VM in order
> to work.
Yeah, I've thought about this some. As you say, the work-around is to
make it configurable: invent some way to say, "these apps run together
the same VM".
--Leif