netbsd-5.1 cpu probe/attach for 11/7xx?
Brad Parker
11/730 support. I don't think the
cpu probe wiring for the 11/7xx line is correct.
If I use a reasonably stock simh and boot a 5.1 kernel it doesn't seem
to call the attach_cpu
function. It does boot and run (in sim), so that's something.
I'm not exactly sure how the cpu should be attached to be honest. The
bus structure has
always been slightly confusing to me (a picture would sure help).
I ended up adding it on mainbus0 by adding the line "cpu* at mainbus0"
to the kernel config file.
That seemed to work. That appears to be the simplest thing. I did
have to change the match function
in autoconf.c to look for "cpu", however.
Index: autoconf.c
===================================================================
RCS file: /cvsroot/src/sys/arch/vax/vax/autoconf.c,v
retrieving revision 1.90
diff -u -r1.90 autoconf.c
--- autoconf.c 11 Mar 2008 05:34:03 -0000 1.90
+++ autoconf.c 25 Jun 2011 16:59:21 -0000
@@ -121,7 +121,7 @@
{
struct mainbus_attach_args * const ma = aux;
if (name)
- aprint_normal("%s at %s", ma->ma_type, name);
+ aprint_normal("%s at %s ", ma->ma_type, name);
return UNCONF;
}
@@ -173,7 +173,7 @@
{
struct mainbus_attach_args *ma = aux;
- return strcmp(cpu_cd.cd_name, ma->ma_type) == 0;
+ return strcmp("cpu", ma->ma_type) == 0;
}
void
I have diffs for 11/730 booting (in sim) for 3.1.1, 5.1 and -current if
anyone wants them. I've recently
moved and dusted off my 11/730 so I hope to try them on real iron in the
next few weeks.
-brad
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-...@muc.de
Johnny Billquist
one CPU, and it is implicitly already attached.
The system starts by trying to figure out what general type of system it
is (checking the SID), and then just start initializing pmap, and all
other stuff from there.
Check locore.c, which have the initial CPU type setup code.
So, there is no cpu probe code being done (or needed) for an 11/780, nor
any other Unibus VAX.
Johnny
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: b...@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
der Mouse
> be one CPU, and it is implicitly already attached.
NetBSD/vax doesn't support the 782? And I suppose the George Gobel's
hack-job would be dreaming?
> So, there is no cpu probe code being done (or needed) for an 11/780,
> nor any other Unibus VAX.
I thought the 780 was SBI-based, with Unibus (and Massbus) hanging off
the SBI. Am I misremembering? It _has_ been quite a while.
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML mo...@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Johnny Billquist
>> ... the KA630 was designed so that MP systems can be done, it is not
>> in general with shared memory. Instead, each processor has its own
>> memory, and you can share a small window of it.
>
> Not all that small a window; the Qbus space is 4M (well, 4M minus 8K),
> and the most RAM a KA6[23]0 can have is 16M.
Yes, but with 4 CPUs, you'll have (potentially) 4*16MB, which is 64MB.
4MB in that perspective is a little less.
>> NUMA system, it would seem... :-)
>
> Very NUMA! I suppose for DEC's designed-for use case it was fine.
Yes. It would also appear to be the only reasonable way to do it within
the restrictions of the Qbus.
>> I wonder if NetBSD could be tricked into handling such a machine.
>
> That would be interesting. It would probably be easiest to do it as
> one NetBSD instance per CPU, with non-primary CPUs having no Qbus
> devices, just a fast network interface built atop the doorbell
> interrupt and shared memory (and perforce running diskless). But
> that's not really using it as a multiprocessor machine; it's just a way
> to get the hardware running something.
Guess so. But if you decide to do it with shared memory, you'll limit
the machine to less than 4MB. More interesting to go with the NUMA, and
have each machine have it's own memory space. Consider each processors
memory as just a part of the total physical memory that exist. Costly
with cross processor memory access, but that's what NUMA is all about.
And while only one processor will deal with interrupts, DMA could
potentially go to any memory, and you could even potentially let any
processor initiate I/O. All you need is proper synchronization. :-)
And then implement a way for memory contents to migrate from one CPU to
another, if needed. But I don't know if/how well NetBSD can deal with
NUMA architectures?
Johnny
Johnny Billquist
> Johnny Billquist wrote:
>>>
>>> I'm proposing a simple fix to make that code (which once worked) work
>>> again.
>>
>> Ok. I had to check things through a little more to refresh my memory.
>> Yes, there is a cpu_attach function for all cpus. But as far as I can
>> tell, it is also called for all CPUs.
> yes, good. But that function is never called on a 7xx. Those lines of
> code never execute, which is the point
> of my original bug report. cpu_mainbus_attach is only called if the
> config is set up correctly.
Ok. I must start by saying that I have not tested anything in the last
five months, so things might have become broken since then. But I have
been peeking around some, and I can't see any commits of relevance to
the various files under VAX being done this spring/summer, so I would
think that it still works as well(?) as it did this winter.
First question is what config file you use to build your kernel?
Second: How do you know that they are not called?
It will take some setting up for me to try this with simh, so I'm a bit
reluctant to do that. I'm normally trying on real hardware, and don't
have any 11/780. :-)
However, I've run NetBSD-current about 5 months ago on a 8650, and the
ka86_attach_cpu() function was called there, which is absolutely through
the same mechanism as for the 11/780.
So I find it a bit surprising that you claim that it is not called.
> If you build 5.1 and run it on a 7xx you'll find that cpu probe
> functions are never called because of two
> reasons (outlined in my original email).
I never said that any probe function is called. There is no need to,
since we already know exactly what CPU exists, so the attach function is
called directly without any probing. The "probing" that is done, is in
locore.c.
In fact, I can't even find any cpu probe function for the VAX-11/780.
>> It's done in autoconf.c at line 186:
>>
>> if (dep_call->cpu_attach_cpu != NULL)
>> (*dep_call->cpu_attach_cpu)(self);
> that code never executes on an 11/7xx. I propose to fix that.
>
>> Sos it's not really called through the normal autoconf paths using the
>> config file.
> well, yes and no. In fact, it is, as that code will not get executed
> unless you "fix" the
> config file and the cpu match code (see my original email).
Reading through your original mail again, I'm a bit confused. You say
that you added "cpu* at mainbus0" to the config file. But such a line
already exists in the GENERIC config. Where did you add it? (In GENERIC,
it is at line 177)
Second, the patch you posted to autoconf.c is for revision 1.90, which
is dated 11-Mar-2008. The current version is 1.92.
While it's NetBSD 5, it is not current. Might be better to work against
current, when you do work (just a suggestion).
>> True. locore sets up the CPU information. autoconf then calls the
>> actual attach function. Not through some probing, but directly, as the
>> CPU is already known to exist, and we already know which type.
> Sort of true, but that cpu_attach_cpu code is only called if the config
> is set up correctly.
> So yes, not as the result of probing, but not directly.
>
> build for a 780 and try it on simh and you'll see.
I might have to do that, but it will take some time and work, which I'm
short on right now.
> I'd rather talk about the proper fix, to be honest.
I'm still not convinced it isn't working.
Johnny
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: b...@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
--
Isildur
with multiple KA630's in the same backplane. Don't know if it was in the
DTJ or something else, i recall coming across it on paper in a cabinet in
the library once..
so it has definitely been done already.
On Sun, 26 Jun 2011, der Mouse wrote:
> True enough. But that would require a Qbus backplane with at least
> eight PMI slots (four for CPUs and four for RAM), which as far as I
> know does not exist. Of course one could be built.
>
>>> Very NUMA! I suppose for DEC's designed-for use case it was fine.
>> Yes. It would also appear to be the only reasonable way to do it
>> within the restrictions of the Qbus.
>
> Well...the uV2 already ignores the QBus's restrictions in at least one
> respect, with its private memory interconnect. There's no inherent
> reason the interconnect couldn't've been designed to accept multiple
> CPUs instead of just one, in which case the only limit on memory
> sharing would be the 16MB addressing limit.
>
>> More interesting to go with the NUMA, and have each machine have
>> [its] own memory space. Consider each processors memory as just a
>> part of the total physical memory that exist. Costly with cross
>> processor memory access, but that's what NUMA is all about.
>
> Indeed. It might make a good NUMA research platform _because_
> interprocessor access is so costly.
>
>> And while only one processor will deal with interrupts, DMA could
>> potentially go to any memory, and you could even potentially let any
>> processor initiate I/O. All you need is proper synchronization. :-)
>
> Just a SMP...er, SMOP. :-)
>
>> And then implement a way for memory contents to migrate from one CPU
>> to another, if needed.
>
> I've been toying with the idea of experimenting with that already, but
> with Ethernet as the interconnect rather than Qbus. Perhaps the uV2,
> where you don't have to worry about CPUs (and associated resources)
> appearing and disappearing at runtime, would be a good first step.
>
>> But I don't know if/how well NetBSD can deal with NUMA architectures?
>
> Neither do I. But then, I know very little about how NetBSD deals with
> multiprocessing of an sort.
>
> /~\ The ASCII Mouse
> \ / Ribbon Campaign
> X Against HTML mo...@rodents-montreal.org
> / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
>
--
der Mouse
>> 8K), and the most RAM a KA6[23]0 can have is 16M.
> Yes, but with 4 CPUs, you'll have (potentially) 4*16MB, which is
> 64MB. 4MB in that perspective is a little less.
True enough. But that would require a Qbus backplane with at least
eight PMI slots (four for CPUs and four for RAM), which as far as I
know does not exist. Of course one could be built.
>> Very NUMA! I suppose for DEC's designed-for use case it was fine.
> Yes. It would also appear to be the only reasonable way to do it
> within the restrictions of the Qbus.
Well...the uV2 already ignores the QBus's restrictions in at least one
respect, with its private memory interconnect. There's no inherent
reason the interconnect couldn't've been designed to accept multiple
CPUs instead of just one, in which case the only limit on memory
sharing would be the 16MB addressing limit.
> More interesting to go with the NUMA, and have each machine have
> [its] own memory space. Consider each processors memory as just a
> part of the total physical memory that exist. Costly with cross
> processor memory access, but that's what NUMA is all about.
Indeed. It might make a good NUMA research platform _because_
interprocessor access is so costly.
> And while only one processor will deal with interrupts, DMA could
> potentially go to any memory, and you could even potentially let any
> processor initiate I/O. All you need is proper synchronization. :-)
Just a SMP...er, SMOP. :-)
> And then implement a way for memory contents to migrate from one CPU
> to another, if needed.
I've been toying with the idea of experimenting with that already, but
with Ethernet as the interconnect rather than Qbus. Perhaps the uV2,
where you don't have to worry about CPUs (and associated resources)
appearing and disappearing at runtime, would be a good first step.
> But I don't know if/how well NetBSD can deal with NUMA architectures?
Neither do I. But then, I know very little about how NetBSD deals with
multiprocessing of an sort.
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML mo...@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
--
Johnny Billquist
>> Older VAXen don't really have a CPU attach function. There can only
>> be one CPU, and it is implicitly already attached.
>
> hack-job would be dreaming?
Right. The 782 is not supported. It is in fact rather tricky, if we were
ever to support it, since it's not an SMP machine.
The second CPU acts as a slave to the first one. They are not equal. The
second CPU can only do computations, and have no I/O attached to it.
>> So, there is no cpu probe code being done (or needed) for an 11/780,
>> nor any other Unibus VAX.
>
> I thought the 780 was SBI-based, with Unibus (and Massbus) hanging off
> the SBI. Am I misremembering? It _has_ been quite a while.
He. Of course someone should catch me in that lie... :-)
Yeah, the 11/780 have an SBO as the system bus, and then one or more
Unibus adapters (and massbus adapters) attached to the SBI, along with
memory.
There is in fact no VAX which have a Unibus as the native bus. The
closest would be the uVAX I, which have the Qbus as the system bus. But
that's the only one.
But I don't really know a good way to describe all the early VAXen with
a common name, since they are so different. But it's all machines before
VAXBI came, when actual MP VAXen first showed up.
VAXBI machines should have a CPU attach function somewhere, although I
have not checked the code for those machines out.
Johnny
--
Johnny Billquist || "I'm on a bus
|| on a psychedelic trip
email: b...@softjar.se || Reading murder books
pdp is alive! || tryin' to stay hip" - B. Idol
--
Brad Parker
> First question is what config file you use to build your kernel?
I used both "GENERIC" and a stripped down 7xx only config I made.
You are right, the GENERIC does have "cpu* at mainbus0"; that was my
mistake - I didn't originally have it my custom config.
But, even if that is there, the match in cpu_mainbus_attach will fail.
> Second: How do you know that they are not called?
I added printf and booted the kernel.
> However, I've run NetBSD-current about 5 months ago on a 8650, and the
> ka86_attach_cpu() function was called there, which is absolutely
> through the same mechanism as for the 11/780.
> So I find it a bit surprising that you claim that it is not called.
I think the ka86 follows a different path because of NMI.
> In fact, I can't even find any cpu probe function for the VAX-11/780.
semantics, sorry. I would claim the "cpu_attach_cpu" function is a
probe function.
> I'm still not convinced it isn't working.
ok. I have special versions of simh which claim to be an 11/730, and
11/750 and an 11/780 and when I put
printf's in the "cpu_attach_cpu" functions for each they don't print.
after I fix the match function, they do.
Johnny Billquist
>> But for VMS [on the -11/782], the secondary CPU was scheduled by the
>> primary CPU. It did not run the OS at all. Instead, [...]
>
> Of course, the real question is, how much of this was imposed by the
> hardware and how much is just how VMS happened to use it; just because
> the hardware is (almost) symmetric does not necessarily mean software
> will use it symmetrically.
Exactly. I have the vague idea that the possibility to interrupt the
other processor is not symmetric, but I can't find any documentation of
the 11/782 right now, so I don't know where I got this notion from.
>>> Certainly the MicroVAX-II supported relatively symmetric
>>> multiprocessing; as far as I can recall the only asymmetries there
>>> are that the CPU at ID 0 (a) terminates the Qbus, (b) runs at
>>> power-up instead of halting and waiting for another CPU to boot it,
>>> and (c) gets Qbus device interrupts.
>> Well, not really that simple. There are no IDs for devices on a
>> Qbus.
>
> No, but there are for MicroVAX-II CPUs. There can be up to four of
> them per Qbus; for the KA630 and KA620, the ID bits are set by the
> connector that on a main CPU runs to the back-panel plate with the
> console DE9, speed switch, halt switch, etc. The usual back-panel
> board sets those bits to ID 0, the distinguished CPU, but alternative
> boards can set it to any of the other three (as can an ordinary board
> with minimal surgery).
[...]
Ah! Thanks. I read through the user guide for the KA630. Fun stuff.
While it generally means that the KA630 was designed so that MP systems
can be done, it is not in general with shared memory. Instead, each
processor has its own memory, and you can share a small window of it.
Mostly thought to be used for message passing.
But you do indeed have four different CSRs reserved for the four CPUs,
so that they can knock on each other. And only the primary CPU gets
interrupts from the Qbus, although all CPUs gets other interrupts.
>> It also becomes even more weird, since memory access by one CPU does
>> not even go out on the Qbus, so you can only get access to another
>> CPUs memory by faking that you are accessing the I/O page of the
>> Qbus, which is a 8Kbyte memory in the high end.
>
> I'm fairly sure there was a way to access more than 8K at a time, but
> that is getting into things lost to the mists of memory.
After looking, you are right. As the CPU appears to be able to present
the private memory as Qbus memory. That sets the limit to 4MB. I wonder
if that is done through the same map as for DMA. Would seem logical.
>> And this has not at all solved the interrupt issues. If a device
>> field an interrupt, all CPUs would see it. How would you prevent all
>> but one CPU to ignore this?
>
> I *think* this too is driven off the ID bits - an ID setting other than
> 00 disables the Qbus interrupt response circuitry. Again, I'd have to
> check the doc to be sure.
No need. You remember right. It's controlled by those ID bits. They
select if arbitration is done, interrupt handling, and also the CSR
address for the CPU.
NUMA system, it would seem... :-) I wonder if NetBSD could be tricked
into handling such a machine.
Johnny
Johnny Billquist
>>>> Older VAXen don't really have a CPU attach function. There can
>>>> only be one CPU, and it is implicitly already attached.
>> were ever to support it, since it's not an SMP machine. The second
>> CPU acts as a slave to the first one. They are not equal. The
>> second CPU can only do computations, and have no I/O attached to it.
>
> For example, only one CPU can field interrupts?
Hmm. Well, I can't really say. The MP machines I've worked on have been
much more SMP than the 11/782 is. But it's true that few machines are
truly symmetric. There is usually some CPU which is a little special.
I must admit that I'm not totally aware of how the hardware works in the
11/782. I only know how VMS ran on that machine, and as far as I know,
only VMS supported it. But for VMS, the secondary CPU was scheduled by
the primary CPU. It did not run the OS at all. Instead, the primary CPU
scheduled the secondary, and whenever something significant happened for
the process controlled by the secondary CPU, it just interrupted the
primary, who had to deal with it. And the secondary CPU then waited for
some new work to be scheduled for it by the primary CPU. The machine did
have shared memory, but apart from that, I'm not sure how the
communication between the CPUs were done. Some sort of way for the
secondary to interrupt the primary existed, and some way for the primary
to feed information to the secondary for it to start work, but I lack
more detail. (I bet it exists somewhere though, if we really want to
figure this out...)
> Certainly the MicroVAX-II supported relatively symmetric
> multiprocessing; as far as I can recall the only asymmetries there are
> that the CPU at ID 0 (a) terminates the Qbus, (b) runs at power-up
> instead of halting and waiting for another CPU to boot it, and (c) gets
> Qbus device interrupts. (Not all device interrupts; some devices, such
> as console serial ports, exist on each CPU board and interrupt the CPU
> whose board they're on.)
Well, not really that simple. There are no IDs for devices on a Qbus. If
you wanted to have a second CPU on the Qbus, you need to remove the bus
termination and bus arbitration from that CPU, since you cannot have two
bus arbitrators. And as far as I know, the KA630 don't have a way to
actually do that disabling.
Also, the uVAX II have the memory on PMI, which is separate from the
Qbus, so a second CPU cannot access the memory of the primary CPU. This
is the difference from the uVAX I, which actually had memory on the Qbus
(that also limited the memory to 4MB on the uVAX I).
> It was difficult to get a KA-630 without a MicroVAX around it, becaus
> DEC didn't want knock-off Qbus backplanes furnished with KA630s and
> sold as cheap MicroVAXen. (The KA620 was their answer to that; I don't
> recall whether NetBSD supports the KA620. It would probably be easy to
> add if not; it was a minor tweak.) I know a bit about this because a
> reseacher I was supporting once wrote a robot control system where the
> control law ran on an auxiliary KA620 - and I wrote the kernel for the
> auxiliary CPU.
I have actually never seen or heard of a MP KA630, but if you have some
real references, it would be interesting. I looked up the KA620, and it
might be that this model was designed with the possibility to have
several on a single Qbus, but it is a bit unclear. I'm also not sure how
you dealt with interprocessor communication in this case. I suspect the
KA620 used normal Qbus memory, but how would one CPU interrupt another one?
KA630 cpu boards exists in ridiculous numbers nowadays, you know... ;-)
>> There is in fact no VAX which have a Unibus as the native bus. The
>> closest would be the uVAX I, which have the Qbus as the system bus.
>> But that's the only one.
>
> What did the uV2 have? Certainly from a user's perspective the Qbus
> was the system bus there; are you considering there to be a memory bus
> on the CPU board which the CPU, RAM, and Qbus all hang off of?
RAM does not sit on the Qbus. Remember, the Qbus can only address 4MB.
You have a Unibus map (well, I guess it should be called a Qbus map in
this case) between the Qbus and memory, while the CPU talks to memory
through PMI.
There is no official name for whatever bus is closest to the CPU on the
uVAX II, and since that bus is internal to the CPU card, it's easy to
regard the machine as a Qbus machine, but looking closer, it is pretty
obvious that it's not really true that the Qbus is the system bus. The
Qbus is just an I/O bus for the uVAX II.
I think that in NetBSD we call it "VSBUS" for lack of a better name.
> Certainly from a multiprocessing perspective, the Qbus was the main
> interconnect; for one CPU to get the attention of another required a
> Qbus cycle. (Each CPU had its own memory interconnect; the only way to
> share memory between CPUs, as far as I know, was over the Qbus. This
> was a bit of a pain, because it meant that for every byte of memory,
> one CPU had fast access to it and everybody else had slow, or no,
> access.)
That would also imply DMA access to get to another CPUs memory, running
through that CPUs mapping registers, and so on. It also becomes even
more weird, since memory access by one CPU does not even go out on the
Qbus, so you can only get access to another CPUs memory by faking that
you are accessing the I/O page of the Qbus, which is a 8Kbyte memory in
the high end. The fiddling to pull this off is more than I want to think
of, if at all possible. Such an I/O page access on the Qbus is not the
same as a DMA memory access by some other device on the Qbus, so I very
much doubt the other CPUs memory mapping and response would be
activated. And you'd still need to avoid your own CPU to react to those
accesses. Besides, you then also need to make the CPU respond to a whole
range of addresses up high. And this has not at all solved the interrupt
issues. If a device field an interrupt, all CPUs would see it. How would
you prevent all but one CPU to ignore this?
Johnny
der Mouse
>>> only be one CPU, and it is implicitly already attached.
> were ever to support it, since it's not an SMP machine. The second
> CPU acts as a slave to the first one. They are not equal. The
> second CPU can only do computations, and have no I/O attached to it.
Aren't a lot of multiprocessor machines asymmetric in that sort of way?
For example, only one CPU can field interrupts?
Certainly the MicroVAX-II supported relatively symmetric
multiprocessing; as far as I can recall the only asymmetries there are
that the CPU at ID 0 (a) terminates the Qbus, (b) runs at power-up
instead of halting and waiting for another CPU to boot it, and (c) gets
Qbus device interrupts. (Not all device interrupts; some devices, such
as console serial ports, exist on each CPU board and interrupt the CPU
whose board they're on.)
It was difficult to get a KA-630 without a MicroVAX around it, becaus
DEC didn't want knock-off Qbus backplanes furnished with KA630s and
sold as cheap MicroVAXen. (The KA620 was their answer to that; I don't
recall whether NetBSD supports the KA620. It would probably be easy to
add if not; it was a minor tweak.) I know a bit about this because a
reseacher I was supporting once wrote a robot control system where the
control law ran on an auxiliary KA620 - and I wrote the kernel for the
auxiliary CPU.
>>> [...] for an 11/780, nor any other Unibus VAX.
>> I thought the 780 was SBI-based, [...]
> He. Of course someone should catch me in that lie... :-)
:-)
> There is in fact no VAX which have a Unibus as the native bus. The
> closest would be the uVAX I, which have the Qbus as the system bus.
> But that's the only one.
What did the uV2 have? Certainly from a user's perspective the Qbus
was the system bus there; are you considering there to be a memory bus
on the CPU board which the CPU, RAM, and Qbus all hang off of?
Certainly from a multiprocessing perspective, the Qbus was the main
interconnect; for one CPU to get the attention of another required a
Qbus cycle. (Each CPU had its own memory interconnect; the only way to
share memory between CPUs, as far as I know, was over the Qbus. This
was a bit of a pain, because it meant that for every byte of memory,
one CPU had fast access to it and everybody else had slow, or no,
access.)
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML mo...@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
--
der Mouse
> primary CPU. It did not run the OS at all. Instead, [...]
Of course, the real question is, how much of this was imposed by the
hardware and how much is just how VMS happened to use it; just because
the hardware is (almost) symmetric does not necessarily mean software
will use it symmetrically.
>> Certainly the MicroVAX-II supported relatively symmetric
>> multiprocessing; as far as I can recall the only asymmetries there
>> are that the CPU at ID 0 (a) terminates the Qbus, (b) runs at
>> power-up instead of halting and waiting for another CPU to boot it,
>> and (c) gets Qbus device interrupts.
> Well, not really that simple. There are no IDs for devices on a
> Qbus.
No, but there are for MicroVAX-II CPUs. There can be up to four of
them per Qbus; for the KA630 and KA620, the ID bits are set by the
connector that on a main CPU runs to the back-panel plate with the
console DE9, speed switch, halt switch, etc. The usual back-panel
board sets those bits to ID 0, the distinguished CPU, but alternative
boards can set it to any of the other three (as can an ordinary board
with minimal surgery).
> If you wanted to have a second CPU on the Qbus, you need to remove
> the bus termination and bus arbitration from that CPU, since you
> cannot have two bus arbitrators.
I believe using non-0 ID bits does that. I'd have to check the
KA630/KA620 docs to be sure.
> And as far as I know, the KA630 don't have a way to actually do that
> disabling.
I'm writing from memory here. I have a scanned copy of
EK-KA630-UG-001, but thanks to monitor trouble, it's difficult for me
to look at it right now; see
ftp.rodents-montreal.org:/mouse/misc/EK-KA630-UG-001/ if you want to
check it out. But I'm fairly sure just feeding it ID bits other than
00 does what's necessary. (If you have trouble finding it, I should be
able to look at the files on Monday, or I might even be able to find
the paper original.)
> Also, the uVAX II have the memory on PMI, which is separate from the
> Qbus, so a second CPU cannot access the memory of the primary CPU.
Right. That's what I mentioned later. Every CPU has its own pool of
private-interconnect memory, which it may choose to let other CPUs
access - but access by other than the native CPU will involve Qbus
cycles (it just looks like Qbus memory to other CPUs).
> I have actually never seen or heard of a MP KA630,
Well...the KA630 is just the processor. An MP KA630 does not,
strictly, exist, but, an MP machine may have multiple KA630s as CPUs.
> but if you have some real references, it would be interesting. I
> looked up the KA620, and it might be that this model was designed
> with the possibility to have several on a single Qbus, but it is a
> bit unclear.
The KA620 is identical to the KA630 except that on the KA630, P0 and P1
page tables live in system virtual space, whereas on the KA620, they
live in physical space. The only other difference I know of is the way
DEC marketed them - and I worked with them fairly extensively when I
was at McGill.
> I'm also not sure how you dealt with interprocessor communication in
> this case. I suspect the KA620 used normal Qbus memory, but how
> would one CPU interrupt another one?
The KA620 can use Qbus memory, just as the KA630 can. But it can also
use private-interconnect memory, just as the KA630 can. And it has 1MB
onboard, same as the KA630.
Any Qbus initiator can interrupt any KA6[23]0 by a suitable write to
that CPU's IPCR (InterProcessor Communication Register), colloquially
called the doorbell interrupt. (The IPCR's address depends on the
CPU's ID bits; there are addresses specified for four of them.)
On most CPUs, doorbell interrupts are unreliable; a fraction on the
order of several hundred PPM of interprocessor interrupts get lost.
This is because of a hardware bug. There's a particular etch run that
crosses the whole board; the resulting capacitance, in conjunction with
the driving impedance, should be insufficient. That it works even two
or three nines of the time is because DEC routinely overdesigned. DEC,
to their credit, was able to find and fix this once my colleague was
able to produce convincing evidence of the misbehaviour with a simple
test program, and, as long as a DEC field service organization existed,
you could get the fix (for free, IIRC) if you knew exactly which FCO to
ask for.
When running DEC's multiprocessor thing (VAXELN is the name that comes
to mind), interprocessor doorbells don't need to be reliable; losing a
tiny fraction of them doesn't hurt. But it did hurt the robot control
system my colleague built.
> KA630 cpu boards exists in ridiculous numbers nowadays, you know... ;-)
I know. I have at least twice as many KA630s as I have Qbus
backplanes. If I had two 16MB single-board memory cards I'd be
extremely tempted to put together a two-CPU machine in a BA123.
>> (Each CPU had its own memory interconnect; the only way to share
>> memory between CPUs, as far as I know, was over the Qbus. [...])
> That would also imply DMA access to get to another CPUs memory,
> running through that CPUs mapping registers, and so on.
Right. That's why you might not have access to all of another CPU's
memory: that CPU's mapping registers might not be configured to.
> It also becomes even more weird, since memory access by one CPU does
> not even go out on the Qbus, so you can only get access to another
> CPUs memory by faking that you are accessing the I/O page of the
> Qbus, which is a 8Kbyte memory in the high end.
I'm fairly sure there was a way to access more than 8K at a time, but
that is getting into things lost to the mists of memory.
> And this has not at all solved the interrupt issues. If a device
> field an interrupt, all CPUs would see it. How would you prevent all
> but one CPU to ignore this?
I *think* this too is driven off the ID bits - an ID setting other than
00 disables the Qbus interrupt response circuitry. Again, I'd have to
check the doc to be sure.
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML mo...@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
--
der Mouse
> in general with shared memory. Instead, each processor has its own
> memory, and you can share a small window of it.
Not all that small a window; the Qbus space is 4M (well, 4M minus 8K),
and the most RAM a KA6[23]0 can have is 16M.
> NUMA system, it would seem... :-)
Very NUMA! I suppose for DEC's designed-for use case it was fine.
> I wonder if NetBSD could be tricked into handling such a machine.
That would be interesting. It would probably be easiest to do it as
one NetBSD instance per CPU, with non-primary CPUs having no Qbus
devices, just a fast network interface built atop the doorbell
interrupt and shared memory (and perforce running diskless). But
that's not really using it as a multiprocessor machine; it's just a way
to get the hardware running something.
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTML mo...@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
--