I'll admit your responses were surprising. The point of my original post was that there were existing projects underway with similar functionality that the QubesOS team could have built on to prevent redundant work. You said you weren't aware of any projects producing systems with similar features, especially that your team could have built on. When I give examples, you give the red herring "Where can I got an install-able & usable distribution/OS based on this concept?" The point was your "install-able & usable distro" could have been built on top of those to reduce redundant work and produced an "install-able & usable distro" with better assurance and features. I'll give you some links anyway:
"yeah, and we all see how secure mobile phones are these days..."
Sarcasm doesn't an argument make. The vendors aren't concerned with security. They mainly use OKL4 to isolate their baseband stacks from the main, insecure OS to prevent DOS attacks on their infrastructure & increase reliability of baseband stack. Separating GPL code & better multicore management are another big use case. This has worked very well. This doesn't mean the microkernel's security is poor. They've recently deployed a mobile security platform with Sirrix, but I haven't been able to evaluate it yet.
"I'm surprised you never mentioned the secure-by-design microkernel based Mac OS X!"
That statement sounds ignorant. Mach is a first generation microkernel renowned for trying to do too much, being too complex, and having horrible performance. By contrast, the third generation L4 kernels fit in a processors L1 cache and only support a few primitives. Every vendor making safety- and security-critical RTOS's takes a microkernel approach with a very successful track record (see INTEGRITY, DEOS, & PikeOS). Apple took the worst microkernel ever designed, added BSD's networking, added graphics, named the oversized kernel-mode heap Darwin, and then proclaimed it secure because Mac's don't matter to virus writers. So, of course I didn't mention a bogus marketing claim about a hybrid kernel design in my discussion of >micro<-kernels and truly secure systems. Why would I?
"Please support this statement [about microkernel superiority for virtualization with stand-alone app capability] with concrete arguments"
Good papers and benchmarks have already been made. Links:
On more isolated nature of drivers in microkernel systems and Linux driver reuse, you said I clearly don't know what I'm talking about. Specific criticism would be easier to respond to than dismissive sarcasm, but I'll use a general response. For readers who are unaware, the Trusted Computing Base (TCB) is what any piece of software depends on to maintain its security policy. The more code, the more bugs (Daniel J Bernstein, "QMail Lessons Learned" paper). The more trusted code, the more vulnerabilities. This is why the TCB, especially privileged code, should be as small as possible. Layering and abstraction in the code itself is also good for verificaiton. These principles have been standard practice in *every* certified safety- or security-critical high assurance project for the last thirty years.
Monolithic systems, including Linux and Linux-reliant Xen platform, inevitably turned up serious flaws due to their size and complexity. Several projects, like Xenon, are *still* trying to make a version of Xen that can be certified to medium or high robustness. Although they still haven't finished, during the same time period, several microkernels were designed, rigorously analyzed, pen tested, DO-178B/EAL6+ certified to high assurance and/o fielded in jets, medical devices, industrial machinery, phones, and PC's with flawless operation. Thirty years of success at achieving reliability and security goals with failure being exceptionally rare proves the microkernel & modular design approaches are superior to monolithic approaches. There's not one monolothic OS on the planet running on COTS hardware that hasn't had a serious vulnerability caused by their own software. There are quite a few microkernels-based platforms that can make that claim, though.
So, running a driver in user-mode on a 15KB kernel with capability-based access control (microkernel approach) isn't more isolated/secure/reliable than running them in Xen's Dom0 on top of Xen, a paravirtualized Linux and side-by-side with a bunch of other software in that Linux VM and little access restrictions on the driver? You would be the first to claim that. I haven't looked into Qubes internals to determine whether or not your team has isolated the drivers directly on Xen to reduce the TCB size or something like that, but drivers are already restricted, lightweight, user-mode processes by default on a microkernel. One only needs to add multiplexing like Xen did for disk and networking IO, although many already can do this due to the client-server programming model.
Reusing Linux drivers in L4 kernels adds a wrapper around the driver, but it's still isolated from others in its own capability-protected, address space. Finally, an even greater compromise, OKL4 platforms can allow drivers to be used in any VM by software in any other VM, as displayed in Heiser's posts on the subject. VM's can possess actual drivers, virtual drivers, or just capability-protected communication channels with other, more privileged VM's. This flexibility allows developers using the microkernel approach to virtualization to carefully select the right tradeoff between security, development effort, and performance.
"oh and don't forget that we want features such as power management in Qubes! and we want to support x86"
Strawman argument + false claim. Every piece of software I mentioned runs on x86. Were you implying they didn't? And you could have added power management with all the effort you spent duplicating other projects accomplishments. Might have saved time. And, again, Xen+Dom0Linux doesn't provide the assurance or performance of Microkernel+User-ModeLinux, so the results were less favorable in key areas for a security-focused project.
"take an academic paper from the 80's, and rewrite it adding funny jokes, publish. you got us!"
Never claimed that. Many people online talk like Invisible Things was the first to find that x86 architecture posed inherent vulnerabilities, especially virtualization and access control mediation issues. I claimed that plenty of papers have done that for over a decade and that invisible things did "awesome work" demonstrating it in practice.
"separation kernels have nothing to do with trusted boot -- these are orthogonal things"
I agree and mentioned more than just trusted boot in that statement. Everything else depends on a kernel's ability to properly isolate every task, enforce access control, etc. Trusted boot is useless without a high assurance kernel or firmware to prevent sabotage of the boot process or of the system after boot. So, they go hand-in-hand, but aren't the same.
"that's mostly covered by trusted boot technologies (which itself have bugs, of course, as we already domed twice"
Yeah, that was the point. Your teams exposed the vulnerabilities, yet few teams have build good secure hardware approaches. I'd love to see Invisible Things Lab apply their expertise to ongoing projects like Secure Core if any spare time crops up.
And feel free to offer *specific* criticisms about any points that *I actually made* in this post. Because counters like these (and I'm paraphrasing some of your replies)...
"I wasn't up to date on what other people were doing."
"I have little knowledge on high assurance design or what that field has accomplished in the past thirty years." "I have some vague problem with what you just said but I won't elaborate." "I couldn't get them on the phone one time last year so they're vaporware."
"Smaller TCB's don't improve driver reliability or security!" "I can't understand how a user-mode app on 15KB kernel is more trustworthy than an app depending on Xen and Dom0 Linux."
"Reliable kernel mode code with strong isolation properties isn't part of a trusted boot process!" "Why didn't you mention Apple's bogus marketing claims as a supporting point!?"
...don't constitute effective or knowledgeable arguments. Fortunately, some parts of your reply didn't follow this pattern. Please, lets stick to objective, specific claims and avoid sarcasm. I'd say we're even now on the latter.
"Nick seemed rather knowledgeable" (PH.T.)
"please, for Christ's sake, don't refer to blog *comments*!" (Joanna)
I would hope so, PH.T. Clive Robinson ("main of many brains") and I have
been hammering at this problem for about a year or two now on Schneier's
blog (Google our names and Schneier's for thorough discussions) and I have
over 200 papers on high assurane designs I've skimmed so far. I'm always
looking for new and old approaches to bulletproof systems. And quoting
comments is a good idea if they contain high quality information, which I'll
leave others to decide with regard to mine. I don't maintain a blog at the
moment. I comment mainly on Krebs on Security and Schneier on Security about
the security of OS's, crypto, applications and systems in general. I would
[sadly] say that the comments on Schneier's blog are usually of higher
quality than many IT blogs... (Although Joanna's advice is appropriate for
*those* blogs ;)
"It's a mistake to compare Qubes to some hypervisor projects..." (Joanna)
I could certainly have worded my post better, as it was hastily typed on
someone else's PC. My comparison was to other projects with the same goals
as I read of QubesOS in various press releases. These projects not only had
similar goals, but also often had similar functionality already finished (or
deployed in real world). I'll post examples for clarification.
"I'm not aware of any project that could be even compared to Qubes -- if you
know, please let me know..." (Joanna)
"Qubes project focuses on things such as: secure GUI isolation,
driver/backend sandboxing, trusted boot, and hiding all the inconveniences
above from the user..." (Joanna)
A number of projects I mentioned have similar focus, aspirations and
sometimes existing feature set. Here's a few.
Green Hill's INTEGRITY Desktop: military-grade secure kernel, trusted path
GUI with secure attenuation sequence, high assurance networking stack,
driver isolation, deprivileged virtual machine monitor, Windows/Linux/POSIX
support, ability to run native programs beside virtual machines, TPM/TXT
support, and high assurance cross domain transfers between VM's. It's been
available since 2005 and is my gold standard for measuring open source
TU Dresden's L4-related work in 2006 produced Nizza Architecture, allowing
security-critical apps to run side-by-side with apps in a legacy OS VM.
Demoed by GPLed TUD:OS, it had an minimal-TCB GUI, deprivileged Linux VM,
scheme to run Linux-compatible device drivers isolated as apps on
microkernel, a virtual filesystem, Qt3 toolkit, libSDL toolkit, and an
ecommerce transaction demo that shows the simplicity of the system. Nizza
was later used to create Micro-SINA, a low TCB IPSec VPN.
Perseus Security Architecture, designed in 2001 and implemented mostly by
2005, was Europe's scheme. The resulting Turaya Security Kernel platform was
released under GPL and provides: microkernel; secure storage for keys; TPM
booting; isolated apps; isolated drivers; VM for legacy Linux apps &
untrusted code. The Turaya Security Platform is being applied in a secure
phone by OK Labs & Sirrix, VPN, and drive encryption. First application was
Turaya Desktop w/ TPM/kernel-protected disk encryption, VPN, and DLP
OK Labs and NICTA. They created L4 & OKL4 kernels, deprivileged Linux layer,
isolated Linux-compatible device driver scheme, ultra-fast isolated apps on
microkernel, and capability based security. It was licensed open-source for
non-commercial use and pretty cheap for commercial use. Recently, a formally
verified microkernel (seL4/OKL4 Verified) with the above features was
released. Field tested in 1 billion phones.
These are my main comparisons and I contend they are quite similar to what
the QubesOS team built in many respects, but their base platform is
superior. The main superiority is that microkernel's can run isolated apps
much better than the Xen kernel, a derivitive of the Nemesis microkernel
designed primarily to run VM's, clunky for isolated app execution, and not
designed with security baked-in from start. The more isolated nature of the
drivers in these other projects, along with Linux driver reuse, already made
one of your goals easy w/out having to contend with the dom0 problem Xen
Now, all that said, I definitely like your project. You've accomplished a
lot and developed plenty of great features. My criticism was that applying
the same amount of effort to one of the above existing projects would have
resulted in a platform with better features, more assurance, a base more
amenable to high assurance methods, and probably higher performance if a
microkernel like OKL4 was used. Again, though, even if researchers disagree
where to put their efforts, I'm glad to see another useful way to improve
the security of the average user's desktop and I love the quality of
Invisible Things' work. My mind has even been turning over the idea of
integrating your team's work with one of these other systems.
To be honest, my biggest disappointment was personal and unrelated to the
results of the QubesOS project. I had been following Invisible Things work
on vulnerabilities in hardware, firmware, etc. for a while. You all simply
did awesome work. The types of vulnerabilities found were well-known in
academic circles: papers written at the time of the Orange Book showed x86
for the insecure, hard to virtualize and covert channel-ridden mess it was.
Your team demonstrated these theoretical findings perfectly & made the
So, here we were, working on designs building on top of projects like
Perseus. Trusted boot, trusted path, secure storage, isolation, etc. were
mostly solved problems, esp. with the separation kernels. The remaining
problem was untrustworthy hardware/firmware/BIOS undermining all of the
above (including Qubes to a degree). I was hoping Invisible Things' next
move was solving that problem somehow so that our existing software could
work w/out worries. Then, we got another secure virtualized desktop platform
running on buggy x86, admittedly with some nice new features.
Idk. I guess I was just hoping the best hardware hackers would give us
better hardware assurance profile, if only a proposed design. Fortunately,
Rockwell-Collins AAMP7G processor, the VAMP processor project, Aegis secure
processor, and Secure Core project gave us (or are giving us) something to
work with. Again, it was just a personal hope that the lab's expertise would
solve the one remaining problem for even the most basic secure x86 system:
untrustworthy hardware, microcode, firmware, BIOS, etc. The problem still
exists, effort was duplicated and only Secure Core provides hope that x86
SOC assurance can be improved. Until then, it's the weakest link in even the
best kernelized system.
Your team was the only well-known team that I thought could do the job since
you lived and breathed hardware flaws. Now we just have a few obscure groups
with less pentesting experience trying to solve the problem. It's the most
neglected area of INFOSEC. With each security layer depending on the one
before it, this is also the most important part of a secure system design.
Few designs provide high assurance of trustworthy hardware and firmware
operation. We need more efforts in this area. More teams with Invisible Labs
expertise hitting the oldest unsolved problem in secure computing: the layer
below the OS.
On 03/21/11 06:48, Nick wrote:
> support, and high assurance cross domain transfers between VM's. It's been
> available since 2005 and is my gold standard for measuring open source
If you could get me a running copy of GH's OS onto my desktop, then I
could comment on this product. Or, at the very least, the architecture
documentation for their mysterious product. Interestingly I have
requested a copy of the documentation through their website more than a
year ago and they never got back to me... (Hey GH!) So, I consider this
product essentially a vaporware. I will change my mind when I will be
able to download at least a trial ISO from their website and try by
> microkernel, a virtual filesystem, Qt3 toolkit, libSDL toolkit, and an
> ecommerce transaction demo that shows the simplicity of the system. Nizza
> was later used to create Micro-SINA, a low TCB IPSec VPN.
Where can I got an install-able & usable distribution/OS based on this
> phone by OK Labs & Sirrix, VPN, and drive encryption. First application was
> Turaya Desktop w/ TPM/kernel-protected disk encryption, VPN, and DLP
Where can I got an install-able & usable distribution/OS based on this
> non-commercial use and pretty cheap for commercial use. Recently, a formally
> verified microkernel (seL4/OKL4 Verified) with the above features was
> released. Field tested in 1 billion phones.
Yeah, and we all see how secure the mobile phones are these days...
Superior security indeed ;)
<even more ironic>
I'm surprised you never mentioned the secure-by-design
_micro-kernel-based_ Max OS X?! ;)
> the QubesOS team built in many respects, but their base platform is
> superior. The main superiority is that microkernel's can run isolated apps
> much better than the Xen kernel, /.../
Please support this statement with concrete arguments.
> The more isolated nature of the
> drivers in these other projects,
You clearly don't know what are you talking about here...
> along with Linux driver reuse, already made
> one of your goals easy w/out having to contend with the dom0 problem Xen
You clearly don't know what are you talking about here...
> resulted in a platform with better features, more assurance, a base more
> amenable to high assurance methods, and probably higher performance if a
> microkernel like OKL4 was used.
Why? Give concrete arguments, pls. Oh, and don't forget that we want
features such as power management in Qubes! And we want to support x86.
> did awesome work. The types of vulnerabilities found were well-known in
> academic circles: papers written at the time of the Orange Book showed x86
> for the insecure, hard to virtualize and covert channel-ridden mess it was.
Yup, our algorithm for coming up with new research: take an academic
paper from the 80's, and rewrite it adding funny jokes, publish. You got us!
> So, here we were, working on designs building on top of projects like
> Perseus. Trusted boot, trusted path, secure storage, isolation, etc. were
> mostly solved problems, esp. with the separation kernels.
You clearly confuse terms -- separation kernels have nothing to do with
trusted boot -- these are orthogonal things.
> The remaining
> problem was untrustworthy hardware/firmware/BIOS undermining all of the
> above (including Qubes to a degree).
That's mostly covered by trusted boot technologies (which itself have
bugs, of course, as we already domed twice).