Critique of the Xen Security Process

[Joanna Rutkowska]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

Recently Xen has released the XSA-148 advisory [1] addressing a fatal bug in the
hypervisor. The bug has been lurking there for the last 7 years! We, the Qubes
OS Project, have commented on this in our Security Bulletin #22 [2]. And far
from enthusiastic commentary that was (FWIW, it was me who wrote this QSB, as
evidenced in the commits log, in case some from the Xen community would like to
direct their rage towards a particular human being ;) Ian Jackson then wrote a
response on the Xen blog [3]. I was then asked to share some more thoughts about
how I thought Xen could actually improve its security process [4]. So, I share
some these below:

1. First of all, I wish Xen was somehow more defensively coded. To provide some
examples:

a. In XSA-109 [5] there was a problem with the hypervisor dereferencing a NULL
pointer. The problem was fixed by the Xen Security Team by applying a patch
which (hopefully) made sure the execution path that lead to this NULL pointer
dereferencing code was never taken. Back then I suggested (on the Xen
pre-disclosure list) to make this patch more explicit though:

> On Wed, Jan 21, 2015 at 02:31:51PM +0100, Joanna Rutkowska wrote:
> (...)
>>
>> Wouldn't it be prudent to also check if:
>>
>> (v->arch.paging.mode>{write_guest_entry,cmpxchg_guest_entry} != NULL)
>>
>> ... in the two affected functions, just before derefing these function
>> pointers?
>>
>> Going even a step further: how about replacing all
>> function-pointer-based calls with macros that first validates the
>> pointer before derefing it? At least when the system doesn't have SMEP?
>>

...to which I got a reply from one of the Xen Security Team engineers that the
above might perhaps be justified in debug builds only, followed by a standard:
"feel free to contribute a patch".

b. The XSA-123 [6] was another critical security bug in Xen, this time resulting
from one of the hypervisor developer's fetish to use an absolutely confusing
construct in order to save a few modest bytes in a structure which might have
been allocated by the system maybe a few tens of times at best. Even more
worrying was the way how Xen Security Team decided to fix the bug: again by
modifying some condition in the code further up the execution path, with the
hopes that this time they would ensure this puzzling construct would always be
used properly. We wrote more about this in our QSB #18 [7].

c. Finally, the way how Xen fixed the recent XSA-148 looks also very reactive,
IMHO. With a bug of this calibre, I would expect Xen to carefully review and
augment all its PV memory virtualization code with additional checks (ASSERTs),
ensuring certain invariants are always satisfied. Such as e.g. that none of the
pages containing PDEs or PTEs are becoming writeable by the VM.

I can't help but have a feeling that some of the Xen developers seem to be
overconfident in their belief they can fully understand all the possible
execution paths in their code. Well, the XSAs quoted above are an indisputable
prove that this is not quite always the case. Realizing that, each developer by
themselves, might be a great step towards a more secure hypervisor...

2. Another security-related aspect of the Xen project is how it totally ignores
problems related to the build process security. Those who don't believe me
should grep the sources for wget, which is now disguised as "FETCHER" shell
variable... (so grep for "FETCHER" string)

I feel embarrassed that I need to explain, at the end of 2015, why the build
process of any serious software project should not blindly download unsigned
components (sources) from the Internet, especially if it is about to execute
Makefiles from these components a moment later... Come on, guys!

(Of course we have been forced to get around this gapping security whole in
Qubes OS [8] ourselves, sadly with a method that is not well suited for
upstreaming).

3. Another thing is, of course: stop adding features to the core hypervisor. We
really need Xen to finally mature, stabilize, and for its development process to
be slowing down over time (just the bug fixes). We need a long-term-supported
hypervisor, which doesn't change with subsonic speed. This would allow this core
code to be widely audited by many experts. If some users want features, these
should perhaps be maintained as additional modules (no, I don't mean dynamically
loaded modules, just compile-time included), preferably in separate repos.

Perhaps also to move all the non-hypervisor code, such as all the toolstacks,
stubdom, etc, into separate repos also. For hygiene, if for nothing else.

Admittedly, some of the features are a result of hardware evolution, such as
e.g. UEFI support. But many are not. Again, maintaining these as optional code
(in separate repos) would be a great step into getting the hypervisor maturing,
finally.

I have already written about it years ago [9], as a matter of fact.

4. Finally, I've been really surprised by the line of reasoning Ian expressed in
the above-mentioned blog post. TL;DR: "we're still doing pretty great, compared
to other projects, because: 1) we have smaller number of publicly disclosed
bugs, and 2) we actually publicly disclose these bugs which we are aware of".

The attitude presented in the blog post is so wrong, that I'm not even sure
where to start commenting on this...

With a single bug like the XSA-148 which, let me repeat that one more time: had
been present in the hypervisor for the last 7 years, so with a bug like this it
really doesn't matter how many (i.e. what number) of critical bugs does the
competition have. Because only one bug of this calibre is enough for the
attacker to never really bother to find another one. The mere fact that
competing hypervisors might got 12 bugs during the same period, really doesn't
make Xen look any better, sorry.

Also, there is really nothing to be proud that you disclose the bugs. It would
be a problem if you didn't.

Hope the above comments might help improve the Xen security. Perhaps some would
perceive them as arrogant or rude. Too bad. Remember the actual attackers will
not be arrogant or rude -- they will just come and exploit bugs, silently.
Admittedly this might not hurt some of the developers ego, not in the short time
at least.

Can we, the Qubes OS project, or myself personally, help with implementing the
above suggestions? Sadly, no. While some of us do contribute occasional patches
to Xen (specifically Marek Marczykowski-Górecki), we really work for a different
project and have different tasks and responsibilities.

Regards,
joanna.

[1] http://xenbits.xen.org/xsa/advisory-148.html
[2] https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-022-2015.txt
[3] https://blog.xenproject.org/2015/10/30/security-vs-features/
[4] https://twitter.com/xen_org/status/660151720463482880
[5] http://xenbits.xen.org/xsa/advisory-109.html
[6] http://xenbits.xen.org/xsa/advisory-123.html
[7] https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-018-2015.txt
[8] https://github.com/QubesOS/qubes-vmm-xen/commit/dcd6c0a4f2c6226a9b706e62469d420579c86975
[9] http://lists.xen.org/archives/html/xen-devel/2013-09/msg01815.html
-----BEGIN PGP SIGNATURE-----

iQIcBAEBAgAGBQJWPOHUAAoJEDOT2L8N3GcYzpEP/An/PTnKDOaC4tPKw3+Y8VIL
n/xIfRPRnPRy18Tbx6EnrKzgohtVvtigtmd/FIxjYVuZ3Luw2B4RFSqUENg758Aa
ANLs4kUD+yaUO82Jfg1nq/6PXBNZlKFovQuYV20LEW9JV6DvMCbzYJ2evZ6t0XS/
EAhUOP1OqY4vb0kah4dwhQKepqwPcD5Tm5LLZn/qbO30e2zN9MkKB851vguQtVIz
k5I8pv+MSQp1efRG2eg470onGtU36IIYFsY1OLihJA9MYh+74FpIA1xoURenJg6+
NJhXEDnxxlz78BJaGOiSwHwB59yd2DXDJKAaUNV/H1LqQu3o1ED+8IZWUARkc0Wl
ckTfQz/++exDhyRcWVHF5GnxEHWdyu/gNZOCNjl4o4HiYS4SQrhTRn7rWwalbyBB
/jG3bAnU8m/Gtp95FtuWXCwuKeeOeBSfnxKMrksxu3JFSNevsYPZu5lrdtUEGLZA
97SwLj70GLesvMqEV3k7XmrQyt8LwyBzLCm+cCocaPEmOQAymeuslrs/RehjGSCQ
L34Ipjvg85GoND64N8X56NuvD+/LrteRhp8hS+aEWv2YpNVJB9tmcOTzLzf7faPK
KPyqa2lW7XKwm7n0WCbtPnrVHRbFPbKvkJRDnfPDwEAEiZcj0SjJ8h7fIDSQ4qac
vgYBTJr/2cRrtC4y1An5
=zQmw
-----END PGP SIGNATURE-----

[James Bulpin]

On Fri, Nov 06, 2015, Joanna Rutkowska wrote
> [snip] I was then asked to share some more

> thoughts about how I thought Xen could actually improve its security
> process [4].

Thanks Joanna for taking the time to put these thoughts into writing.
I think there are a number of actionable things here we should be taking
a look at as a community. Some of the key things from my perspective:

1. Fixing the use of the FETCHER in the build process (I know there has
already been some discussion around this as part of the Raisin work). In
the commercial Xen distribution I'm responsible for we actually replace
this mechanism and run in an isolated build environment; I'd like to see
this be the standard model.

2. "Disaggregating" the hypervisor makes a lot of sense. With Xen seeing
growth in embedded and client use-cases as well as the traditional server
and cloud ones it will increasingly be the case that some hypervisor
features are necessary in some cases and not in others. I certainly only
want my TCB to include stuff I really need.

3. Do we need to revisit entire old subsystems/features that have been
subject to years of incremental development? Perhaps a line item for Xen
4.7 could be a holistic review of the PV MM code. HVMLite may help in this
specific area longer term but PV (and some of our other older features)
are going to be around for a while yet.

4. Development/coding guidance and standards. This is something we need
anyway as we grow the community. We should consider more specific
guidance on defensive coding practices.

To maintainers and committers: perhaps some of these architectural
direction topics would be good discussions for an in-person dev meeting.

Cheers,
James

[Low Eel]

Well exposed, thank you Johanna.

Just one question: anybody knows which EAL level has Xen being certified for?

The question may seem stupid, but I wonder if the approach of closing execution path can survive to semiformal or formal verification.

Thanks

L.

[w.peter...@gmail.com]

This is scary, but unfortunately, it's commonplace.  There is a sad mentality of not including basic checks because it will generate extra instructions that will take more microseconds to execute.  That may have been a concern when we were running 1MHz 8-bit processors, but now that code takes such an insignificant amount of time to execute that it doesn't matter.  Coding should no longer be about making it as fast and efficient as possible -- it needs to be about making it stable and secure.

Maybe it's time to split off the code into a new branch specifically for Qubes, and pick and choose what gets pulled over from their code?
 

3. Another thing is, of course: stop adding features to the core hypervisor. We
really need Xen to finally mature, stabilize, and for its development process to
be slowing down over time (just the bug fixes). We need a long-term-supported
hypervisor, which doesn't change with subsonic speed. This would allow this core
code to be widely audited by many experts. If some users want features, these
should perhaps be maintained as additional modules (no, I don't mean dynamically
loaded modules, just compile-time included), preferably in separate repos.

Perhaps also to move all the non-hypervisor code, such as all the toolstacks,
stubdom, etc, into separate repos also. For hygiene, if for nothing else.

Admittedly, some of the features are a result of hardware evolution, such as
e.g. UEFI support. But many are not. Again, maintaining these as optional code
(in separate repos) would be a great step into getting the hypervisor maturing,
finally.

I have already written about it years ago [9], as a matter of fact.

4. Finally, I've been really surprised by the line of reasoning Ian expressed in
the above-mentioned blog post. TL;DR: "we're still doing pretty great, compared
to other projects, because: 1) we have smaller number of publicly disclosed
bugs, and 2) we actually publicly disclose these bugs which we are aware of".

The attitude presented in the blog post is so wrong, that I'm not even sure
where to start commenting on this...

The bigger question is how much testing is really being done to find vulnerabilities in the first place.  Xen certainly isn't Windows with a base of hundreds of millions and thousands trying to rip it apart and break it.  It sounds like we'll be in for more surprises to come.
 

[Jan Beulich]

And you know what - I can only repeat this now. Indeed, the points
you make are all quite reasonable from a technical perspective (and
especially the code quality aspect is something that I also have been
complaining _but also trying to do something_ about repeatedly), but
the attitude behind the way you present them bothers me quite a lot:
I get the impression that you (a) expect others to do work that you
neither pay for, nor are at the slightest willing to help with and (b)
de-value all our efforts during the last so many years to move the
code base from a research project to a production usable one.

Yes, the work item you mention above has been on our virtual,
combined todo list, but with all the other work that needs to be done
it seems like no-one actually found the time to actually carry this out.
A good example that just pointing out issues is nice and easy, but
actually finding time to do the work is quite a bit harder.

> b. The XSA-123 [6] was another critical security bug in Xen, this time
> resulting
> from one of the hypervisor developer's fetish to use an absolutely confusing
> construct in order to save a few modest bytes in a structure which might
> have
> been allocated by the system maybe a few tens of times at best. Even more
> worrying was the way how Xen Security Team decided to fix the bug: again by
> modifying some condition in the code further up the execution path, with the
> hopes that this time they would ensure this puzzling construct would always
> be
> used properly. We wrote more about this in our QSB #18 [7].

I suppose you already know what my response here would be?

> c. Finally, the way how Xen fixed the recent XSA-148 looks also very reactive,
> IMHO. With a bug of this calibre, I would expect Xen to carefully review and
> augment all its PV memory virtualization code with additional checks (ASSERTs),
> ensuring certain invariants are always satisfied. Such as e.g. that none of the
> pages containing PDEs or PTEs are becoming writeable by the VM.

Perhaps I'm overlooking something, but I cannot see what kinds of
ASSERT()s could have helped avoid that problem. The only things I
can see that would have helped here are
- better thinking through of all cases by the original developer
- better review before things going in
- regular code audits
None of which, according to what you write further down, you're
willing to help with.

> I can't help but have a feeling that some of the Xen developers seem to be
> overconfident in their belief they can fully understand all the possible
> execution paths in their code. Well, the XSAs quoted above are an
> indisputable
> prove that this is not quite always the case. Realizing that, each developer
> by
> themselves, might be a great step towards a more secure hypervisor...

I think we need to find a reasonable balance here. Taking your
statement to the extreme, we might end up with code having more
ASSERT()s in it than anything else. Surely that can't be the goal,
as this would severely hamper maintainability. It simply doesn't
make sense for every function to mistrust its callers in all aspects.

> 2. Another security-related aspect of the Xen project is how it totally
> ignores
> problems related to the build process security. Those who don't believe me
> should grep the sources for wget, which is now disguised as "FETCHER" shell
> variable... (so grep for "FETCHER" string)
>
> I feel embarrassed that I need to explain, at the end of 2015, why the build
> process of any serious software project should not blindly download unsigned
> components (sources) from the Internet, especially if it is about to execute
> Makefiles from these components a moment later... Come on, guys!
>
> (Of course we have been forced to get around this gapping security whole in
> Qubes OS [8] ourselves, sadly with a method that is not well suited for
> upstreaming).

To be honest, for a purely development oriented build environment
this might by fine. No-one is required to use that, or even run their
builds outside of sandboxes. Even in my local build environment
(used for just my own purposes) I avoid any such downloads, albeit
perhaps more for performance than security reasons. But what this
tells us is that avoiding to build on this "convenience" is quite simple.

> 3. Another thing is, of course: stop adding features to the core hypervisor. We
> really need Xen to finally mature, stabilize, and for its development process to
> be slowing down over time (just the bug fixes). We need a long-term-supported
> hypervisor, which doesn't change with subsonic speed. This would allow this core
> code to be widely audited by many experts. If some users want features, these
> should perhaps be maintained as additional modules (no, I don't mean dynamically
> loaded modules, just compile-time included), preferably in separate repos.

I admit that, especially during the last couple of months, I have
wished the ratio between feature addition patches and bug fixes
would shift towards the latter (and that without hoping for there
to be more bugs to be fixed - in fact the opposite has seemed to
be the case in the more recent past), albeit for other reasons than
yours.

But is not adding new features realistic? I don't think so. People
want a secure _and_ functionally complete _and_ performant
hypervisor, and the latter two require making use of new
hardware features as they become available (and occasionally
introduce pure software features too, see the xSplice work
underway).

> Admittedly, some of the features are a result of hardware evolution, such as
> e.g. UEFI support. But many are not. Again, maintaining these as optional code
> (in separate repos) would be a great step into getting the hypervisor maturing,
> finally.

This, afaict, would quickly become unwieldy and unmaintainable.

> 4. Finally, I've been really surprised by the line of reasoning Ian
> expressed in
> the above-mentioned blog post. TL;DR: "we're still doing pretty great,
> compared
> to other projects, because: 1) we have smaller number of publicly disclosed
> bugs, and 2) we actually publicly disclose these bugs which we are aware
> of".
>
> The attitude presented in the blog post is so wrong, that I'm not even sure
> where to start commenting on this...
>
> With a single bug like the XSA-148 which, let me repeat that one more time: had
> been present in the hypervisor for the last 7 years, so with a bug like this it
> really doesn't matter how many (i.e. what number) of critical bugs does the
> competition have.

And let me repeat - it doesn't matter for how long the bug has been
there without anyone noticing. Had you noticed it 7 years ago, and
had we both rejected your fixes and not taken steps to fix it
ourselves, I could see you being upset. But you having done nothing
(Out of the now around 150 security issues in Xen, how many did you
discover, btw?), what gives you the right to complain about a bug (no
matter how severe) having been there for an admittedly rather long
time? If you weren't able to spot it, how can you demand someone
else to be able to? It is the nature of bugs that at times they hide
rather well.

> Can we, the Qubes OS project, or myself personally, help with implementing the
> above suggestions? Sadly, no. While some of us do contribute occasional patches
> to Xen (specifically Marek Marczykowski-Górecki), we really work for a different
> project and have different tasks and responsibilities.

But you seem to forget that many if not most of us too are not
employed by their employers to look after Xen as their sole or
primary task. I.e. if everyone reasoned this way, we could simply
bin the entire project.

So bottom line - _constructive_ criticism is welcome, but
accusations, especially without any intent to help changing things
(other than demanding others to do the necessary work), aren't.

(Please note that all of the above are my personal views, i.e. I'm
explicitly not speaking on behalf of the project.)

Jan

[Franz]

It seems positions are antithetic with no possible compromise in view. Sadly this is a general problem of FOSS software: developers  tend to do what they like and not what users request. And who can blame developers for that? After all they are working as volunteers so they deserve to do what they like. No possible compromise.

Perhaps a way out of this impasse is to put bounties on Xen security tasks identified by Joanna and properly advertise these bounties to Xen users. I would give particular attention to the companies that use Xen. I imagine there are many of them. Some of them may already be giving money to Xen. If some of you knows who are these companies and how is the proper way to set up a bounty, I offer to write the copywriting letter, already have some clear ideas how to do that.

Best

Fran

 

Jan

--
You received this message because you are subscribed to the Google Groups "qubes-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qubes-devel...@googlegroups.com.
To post to this group, send email to qubes...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-devel/56409B7702000078000B3018%40prv-mh.provo.novell.com.

For more options, visit https://groups.google.com/d/optout.

[Wojtek Porczyk]

On Mon, Nov 09, 2015 at 04:31:58PM +0000, Franz wrote:
> Perhaps a way out of this impasse is to put bounties on Xen security tasks
> identified by Joanna and properly advertise these bounties to Xen users.

> [snip]

This is fundamentaly wrong idea. Security isn't something you can
"apply" or put bounty on. It's a state of the mind, especcialy
developer's. Joanna wrote in her mail:

> > > I can't help but have a feeling that some of the Xen developers seem to be
> > > overconfident in their belief they can fully understand all the possible
> > > execution paths in their code. Well, the XSAs quoted above are an indisputable
> > > prove that this is not quite always the case. Realizing that, each developer by
> > > themselves, might be a great step towards a more secure hypervisor...

And that's why we can't just "submit a patch" to "contribute security".
There is something wrong with Xen as a whole project, but that something
isn't the code. There is a mindset to be fixed.

--
regards, _.-._
Wojtek Porczyk .-^' '^-.
Invisible Things Lab |'-.-^-.-'|
| | | |
I do not fear computers, | '-.-' |
I fear lack of them. '-._ : ,-'
-- Isaac Asimov `^-^-_>

[chris]

+1... so many great points here that ive thought many times its almost as if i could have written it

great post!

chris

_______________________________________________
Xen-devel mailing list
Xen-...@lists.xen.org
http://lists.xen.org/xen-devel

[Richard H]

Having read the conversation thus far, I can see and sympathize with both sides.  It does not appear to me that the situation at Xen is something that can be very easily "fixed" to the satisfaction of ITL and the Qubes community.  This is especially evident if you consider it to be "a mindset" that needs to be "fixed" and the fact that Xen doesn't see eye-to-eye on this matter.  As much as I would like and expect Xen to be secure and fit into Qubes' needs, I don't think it is fair to demand this of a project that is coded by volunteers.  They are providing free software that they want to provide. In my opinion, if one is not happy with it, then they should either become active in the project to facilitate a change or use an alternative product / fork.

I imagine that using an alternative is easier said than done.  However, if ITL and the Qubes community believes that "there is something wrong with Xen as a whole project", and Xen is not, and will not for the foreseeable future, be providing the security that is critical to the security of Qubes, are you able to share your thoughts on what is being planned to rectify this situation?

Note: This post is directed to ITL and the Qubes Community, accordingly, I have not cc'd the xen-devel list on it.

Kind regards,
Richard 

[Radoslaw Szkodzinski]

As usual. Security, performance, convenience, price. Pick any mixture.

As is usual for most software, developer convenience trumps most other
considerations. I include ease of generating nice papers and jobs
under developer convenience.
Big players are much more concerned about performance, which saves
money on machines you need to buy. (Note the resources poured into
tmem, which is very iffy from security standpoint.)
Even if there was a sudden security drive for Xen, it's a huge and
probably fruitless task. Xen is not designed top down so you get no
nice overview to check main assumptions. This is also why Xen is
written in C and assembly, not in anything easier to use.

Likewise security-oriented microkernels such as seL4 push drivers to
"userspace" making it convenient for their developers, at the cost of
performance and security, since they are now Somebody Else's Problem.
Surprisingly, seL4 seems to be the only active and working
non-research project that happens to have security as main aim, and
they don't even implement many of features that are "required" for
modern secure design, say, IOMMU...
They want you to write your drivers as automated proofs instead.
Meaning cheap developers will not be able to even lift a finger in
writing those, since you need to learn Isabelle DSL and understand
what is required in order to write a working driver with a good API,
then also understand and model features of underlying hardware. This
requires good mathematical background, as in actual CS, not just
writing code.

What seL4 folks are actually doing (have done, even) is writing a
potent compiler between both C and theorem proving language, plus a
set of theorems describing assumptions made on underlying hardware and
security features. Very simple, just some 250k lines of theorems to
prove correctness of 7k lines of C code.

Good luck trying to port Qubes to their architecture. It's not
impossible, but it is quite a task. So many theorems to wade through
and understand, not to mention a whole new programming language.
You'd get to add well defined isolation primitives to their theorem
prover, then write a parallel proof to capDL kernel (ouch) or extend
capDL with such isolation features and requisite APIs.
Again, developer convenience suffers and have too high barrier of
entry, and you won't get any code written or it will get too expensive
in terms of time, skill and therefore money.

As to other attempts...
Python code? Forget about it, Python is thoroughly undefined behavior
with no truly defined memory model. :) Even C++11 is better there.

Rust? Rust mechanisms are undefined in terms of timing, which will
result in unbounded execution time (hangs) when actual hardware is
involved - there is no way to describe timing behavior of hardware in
it in such a way that these properties are verified at higher layers.
They attempt concurrency but the primitives are not bound to actual
real time and hardware. There is no library for bare metal
programming. Memory allocations are assumed to never fail - and many
other things as well. (Does this sound like some other "never fails"
software we're depending upon?)

Trying to write a secure kernel in an unsafe language is akin to
trying to bail a sinking ship with a spoon. It can theoretically be
done, just not in practice. And actual low-level and safe languages
are surprisingly rare.

Best regards,
R.

[Lars Kurth]

Hi everyone, 

firstly I wanted to thank everyone for raising this issue. I wanted to point out that we are not talking about a security process here, but the development process. Or more accurately the cost of writing more secure code and the relative importance of security compared to features. And of course the recent increase of relative importance of "built-in security as a feature" since Snowden. 

On 9 Nov 2015, at 16:31, Franz <169...@gmail.com> wrote:

(Please note that all of the above are my personal views, i.e. I'm
explicitly not speaking on behalf of the project.)

It seems positions are antithetic with no possible compromise in view. Sadly this is a general problem of FOSS software: developers  tend to do what they like and not what users request. And who can blame developers for that? After all they are working as volunteers so they deserve to do what they like. No possible compromise.

I don't think this is a fair statement: more than 95% of developers working on Xen are employees of large organisations. And they follow the priorities that their employers set. The same is true in Linux and for comparable projects and of course for proprietary software developers. Blaming open source developers, is simply wrong and not constructive. 

In fact, throughout 2014 and 2015, the project has received complaints from several large vendors that the Xen Project today is too rigorous with code reviews compared to Linux, KVM and QEMU and that it is now harder to get code into Xen compared to similar projects. To understand the root causes, the Xen Project Advisory Board has commissioned a study to look at the root causes  (see http://lists.xenproject.org/archives/html/xen-devel/2015-10/msg01635.html). The result of the first phase of the study showed, that a primary factor for longer review times are significantly higher quality expectations on new code compared to the past. Of course this does not help with code that has been added in the past.

In many ways, the problem which was stated comes down to different expectations, by different classes of contributors. And a solution, will necessarily have to involve compromise. 

Perhaps a way out of this impasse is to put bounties on Xen security tasks identified by Joanna and properly advertise these bounties to Xen users. I would give particular attention to the companies that use Xen.

I can certainly raise this suggestion with the Advisory Board and see whether we can make some funds available. However, the board has already invested nearly 50% of its entire budget in Test Infrastructure and is planning to continue to spend in this area at roughly this proportion of the projects budget. However, we do not have huge amounts of funds: thus, what we could do with bounties would necessarily be limited. 

I personally also looked at other ways to change the cost-benefit equation. One example is the Feature Maturity Lifecycle (see http://lists.xen.org/archives/html/xen-devel/2015-11/msg00609.html & http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01992.html) which aims to use better classification to change behaviour of contributors. 

I do hope, that this discussion can remain constructive. De-motivating the good work many of our developers (and in particular code reviewers) are doing, is really not helpful in this context. 

Best Regards

Lars

[Lars Kurth]

> On 9 Nov 2015, at 18:15, Wojtek Porczyk <wo...@invisiblethingslab.com> wrote:
>
> On Mon, Nov 09, 2015 at 04:31:58PM +0000, Franz wrote:
>> Perhaps a way out of this impasse is to put bounties on Xen security tasks
>> identified by Joanna and properly advertise these bounties to Xen users.
>> [snip]
>
> This is fundamentaly wrong idea. Security isn't something you can
> "apply" or put bounty on. It's a state of the mind, especcialy
> developer's.

I don't think this was what was proposed. What was proposed, if I understood correctly, was to use funds to sponsor solutions to specific problems in specific areas of code that have been developed in the past and where we may have issues due to past mistakes.

> Joanna wrote in her mail:
>
>>>> I can't help but have a feeling that some of the Xen developers seem to be
>>>> overconfident in their belief they can fully understand all the possible
>>>> execution paths in their code. Well, the XSAs quoted above are an indisputable
>>>> prove that this is not quite always the case. Realizing that, each developer by
>>>> themselves, might be a great step towards a more secure hypervisor...
>
> And that's why we can't just "submit a patch" to "contribute security".

Although I agree that you can't submit a patch to fix security, it is possible to focus on some areas you care about and lead by example. In particular, if you believe that there is a mindset problem today. I think, you do also need to distinguish between what was happening in the past and what the project does today. There is little point to complain about what developers, who now don't work on Xen any more, did in the past. In particular, as the vast majority of active developers in the project were not around 6-7 years ago.

> There is something wrong with Xen as a whole project, but that something
> isn't the code. There is a mindset to be fixed.

I think this is a rather generalising statement, and maybe it would be more fruitful to focus on some concrete areas that can be improved. I also believe, that there has been a significant shift in mindset in many areas in the project in the last few years.

Lars

[Franz]

On Tue, Nov 10, 2015 at 1:09 PM, Lars Kurth <lars.ku...@gmail.com> wrote:

> On 9 Nov 2015, at 18:15, Wojtek Porczyk <wo...@invisiblethingslab.com> wrote:
>
> On Mon, Nov 09, 2015 at 04:31:58PM +0000, Franz wrote:
>> Perhaps a way out of this impasse is to put bounties on Xen security tasks
>> identified by Joanna and properly advertise these bounties to Xen users.
>> [snip]
>
> This is fundamentaly wrong idea. Security isn't something you can
> "apply" or put bounty on. It's a state of the mind, especcialy
> developer's.

I don't think this was what was proposed. What was proposed, if I understood correctly, was to use funds to sponsor solutions to specific problems in specific areas of code that have been developed in the past and where we may have issues due to past mistakes.

Yes Lars of course, and also changing mindset has real costs, because a more security oriented mindset involves some additional work, some hard thinking, some additional study, something already done that needs recoding etc.  I mean it is not simply an act of will, rather a targeted effort. A security mindset is slowing developing in the whole society and I see Qubes developers as the more advanced front. The reality is changing fast. A serious  bug in Xen practically means that all my passwords can be stolen from my no-network VM or even more seriously that the life of an activist can be compromised. These facts are dramatic or tragic and the situation is constantly and fast worsening. So nobody is guilty for 7 years old code developed in a different situation, but the same Joanna words are totally justified and welcome. We have to face reality and take action. From my point of view It is a matter of educating those who pay, for that I offered to write some copywriting.

Best

Fran

[Chris Laprise]

Hello...

On 11/10/2015 05:52 AM, Lars Kurth wrote:
> Hi everyone,
>
> firstly I wanted to thank everyone for raising this issue. I wanted to
> point out that we are not talking about a security process here, but
> the development process. Or more accurately the cost of writing more
> secure code and the relative importance of security compared to
> features. And of course the recent increase of relative importance of
> "built-in security as a feature" since Snowden.
>
>> On 9 Nov 2015, at 16:31, Franz <169...@gmail.com

>> <mailto:169...@gmail.com>> wrote:
>>
>> (Please note that all of the above are my personal views, i.e. I'm
>> explicitly not speaking on behalf of the project.)
>>
>>
>> It seems positions are antithetic with no possible compromise in
>> view. Sadly this is a general problem of FOSS software: developers
>> tend to do what they like and not what users request. And who can
>> blame developers for that? After all they are working as volunteers
>> so they deserve to do what they like. No possible compromise.
>
> I don't think this is a fair statement: more than 95% of developers
> working on Xen are employees of large organisations. And they follow
> the priorities that their employers set. The same is true in Linux and
> for comparable projects and of course for proprietary software
> developers. Blaming open source developers, is simply wrong and not
> constructive.

Nevertheless, Xen is a creature of interfaces that purport to uphold
contracts. These (software engineering, not legal) contracts are
explicitly security-themed... the software is promising to isolate
processes from each other and protect against privilege escalation. The
Xen project itself asserts that it is focused on the security benefits
of the hypervisor (to the point of invoking "microkernel" in Xen's
description), not mere administrative convenience.

What seems to be missing from the defense of Xen project so far in this
thread is a level of acknowledgement of this very important aspect of
twenty-first century software development: Can you honor what your
interfaces communicate to your audience? Its true that some FOSS
projects do not care for modern methodologies (which are greatly about
concept and mindset), but should Xen be that way?

Should Xen project also be a blanket absolution of "blame"? And is the
corporate status of some of its contributors a proper justification for
being blame-less (perhaps we can think of them like members of 'The
Borg')? I think not.

This is the trap that FOSS projects most commonly fall into: 'We are
free to publish because of liberty, but you are not free to criticize.'
Its odd when you think about it. This is the squandering of user
motivation and valuable feedback.

>
> In fact, throughout 2014 and 2015, the project has received complaints
> from several large vendors that the Xen Project today is too rigorous

> with code reviews compared to Linux, KVM...


Famous last words before a Heartbleed-scale media circus ensues?

>
> I can certainly raise this suggestion with the Advisory Board and see
> whether we can make some funds available. However, the board has
> already invested nearly 50% of its entire budget in Test
> Infrastructure and is planning to continue to spend in this area at
> roughly this proportion of the projects budget. However, we do not
> have huge amounts of funds: thus, what we could do with bounties would
> necessarily be limited.
>
> I personally also looked at other ways to change the cost-benefit
> equation. One example is the Feature Maturity Lifecycle (see
> http://lists.xen.org/archives/html/xen-devel/2015-11/msg00609.html &
> http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01992.html)
> which aims to use better classification to change behaviour of
> contributors.
>
> I do hope, that this discussion can remain constructive. De-motivating
> the good work many of our developers (and in particular code
> reviewers) are doing, is really not helpful in this context.
>

It does seem to me that the suggestion of a Long-Term Support (LTS)
release is a constructive one, among the other suggestions that Joanna
made. The FML you cite is interesting, but seems to be aimed squarely at
developers. You are bound to get better results if the expectations of
users change along with developers, so LTS releases may be the better idea.

> Best Regards
> Lars
>
>

[Stefano Stabellini]

On Fri, 6 Nov 2015, Joanna Rutkowska wrote:
> Can we, the Qubes OS project, or myself personally, help with implementing the
> above suggestions? Sadly, no. While some of us do contribute occasional patches
> to Xen (specifically Marek Marczykowski-Górecki), we really work for a different
> project and have different tasks and responsibilities.

Ian wrote in his blog post: "Ultimately, of course, a Free Software
project like Xen is what the whole community makes it. In the project as
a whole we get a lot more submissions of new functionality than we get
submissions aimed at improving the security".

If invisiblethingslab, or other security groups, were among of the top
three contributors or reviewers in Xen Project, the community and the
code would be different.

If somebody submitted a clean patch to disable or remove a functionality
because it is not secure, we would consider it. Even better, if somebody
reviewed a patch and found security issues with it, she would prevent
the patch from getting in in the first place. What doesn't this happen
more often?

In general, if security is important to many people, why don't we see
more security oriented submissions and reviews? Anybody can do reviews!
He or she doesn't need to be a maintainer to do them.


Open Source software communities are a funnel:

https://talesfromthecommunity.wordpress.com/2012/06/16/viewing-communities-as-funnels/

and in particular:

https://talesfromthecommunity.files.wordpress.com/2012/06/basicfunnel1.png

It seems to me that not enough security aware people are getting past
the "customize without contributing back" or "occasionally contributing"
stages. Why is that?

Don't underestimate the impact that even a single very committed
individual can have on a software project.

[Lars Kurth]

> On 11 Nov 2015, at 11:36, Chris Laprise <tas...@openmailbox.org> wrote:
>
> Hello...
>
> On 11/10/2015 05:52 AM, Lars Kurth wrote:
>> Hi everyone,
>>
>> firstly I wanted to thank everyone for raising this issue. I wanted to point out that we are not talking about a security process here, but the development process. Or more accurately the cost of writing more secure code and the relative importance of security compared to features. And of course the recent increase of relative importance of "built-in security as a feature" since Snowden.
>>
>>> On 9 Nov 2015, at 16:31, Franz <169...@gmail.com<mailto:169...@gmail.com>> wrote:
>>>
>>> (Please note that all of the above are my personal views, i.e. I'm
>>> explicitly not speaking on behalf of the project.)
>>>
>>>
>>> It seems positions are antithetic with no possible compromise in view. Sadly this is a general problem of FOSS software: developers tend to do what they like and not what users request. And who can blame developers for that? After all they are working as volunteers so they deserve to do what they like. No possible compromise.
>>
>> I don't think this is a fair statement: more than 95% of developers working on Xen are employees of large organisations. And they follow the priorities that their employers set. The same is true in Linux and for comparable projects and of course for proprietary software developers. Blaming open source developers, is simply wrong and not constructive.
>
> Nevertheless, Xen is a creature of interfaces that purport to uphold contracts. These (software engineering, not legal) contracts are explicitly security-themed... the software is promising to isolate processes from each other and protect against privilege escalation. The Xen project itself asserts that it is focused on the security benefits of the hypervisor (to the point of invoking "microkernel" in Xen's description), not mere administrative convenience.

All of this is true from an architectural viewpoint and I don't think that has been disputed. And it also related to how we handle security issues, which we have proven to handle more effectively than most other projects and even commercial vendors.

> What seems to be missing from the defense of Xen project so far in this thread is a level of acknowledgement of this very important aspect of twenty-first century software development: Can you honor what your interfaces communicate to your audience? Its true that some FOSS projects do not care for modern methodologies (which are greatly about concept and mindset), but should Xen be that way?

There has been some discussion around this in the past. The problem is that the cost of doing this retrospectively (at least in the context of certification, which would be one way to achieve this) is prohibitively expensive. And the cost of doing this for new code, is also very high. You can find some cost calculations in: http://www.xenproject.org/presentations-and-videos/video/xpds14v-certification.html

You seem to be arguing that Xen Project should be like a security hardened Linux distro. The project never claimed that this was the case. Moving towards something better, may be achievable through some better working practice. But not without significant investment, which will affect other members of the community.

The fundamental questions is, whether for the majority of Xen users, what we do today is good enough or not. And whether we do better than competing projects and products. And I believe that compared to others we are doing better. Maybe the solution is for some of the security conscious vendors and users of Xen to collaborate and create a security hardened version of Xen, or to work with the community in other ways, rather than to ask others to pay them a free lunch.

> Should Xen project also be a blanket absolution of "blame"? And is the corporate status of some of its contributors a proper justification for being blame-less (perhaps we can think of them like members of 'The Borg')? I think not.

That was not what I argued against: what I argued against the statement that "this is a general problem of FOSS software: developers tend to do what they like and not what users request". Of course vendors who contribute to open source do so primarily because their customers are asking for features and other functionality. And IMHO, the pressure to constantly deliver new things has actually become worse in the last 2 years. You just need to look at the Container and Docker hype, to see the majority of the industry appear to value features and the latest "shiny new thing" a lot more than security (or slower development velocity as a consequence of security).

> This is the trap that FOSS projects most commonly fall into: 'We are free to publish because of liberty, but you are not free to criticize.' Its odd when you think about it. This is the squandering of user motivation and valuable feedback.
>
>
>>
>> In fact, throughout 2014 and 2015, the project has received complaints from several large vendors that the Xen Project today is too rigorous with code reviews compared to Linux, KVM...
>
>
> Famous last words before a Heartbleed-scale media circus ensues?

This is already happening: see http://www.washingtonpost.com/sf/business/2015/11/05/net-of-insecurity-the-kernel-of-the-argument/

>> I can certainly raise this suggestion with the Advisory Board and see whether we can make some funds available. However, the board has already invested nearly 50% of its entire budget in Test Infrastructure and is planning to continue to spend in this area at roughly this proportion of the projects budget. However, we do not have huge amounts of funds: thus, what we could do with bounties would necessarily be limited.
>>
>> I personally also looked at other ways to change the cost-benefit equation. One example is the Feature Maturity Lifecycle (see http://lists.xen.org/archives/html/xen-devel/2015-11/msg00609.html & http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01992.html) which aims to use better classification to change behaviour of contributors.
>>
>> I do hope, that this discussion can remain constructive. De-motivating the good work many of our developers (and in particular code reviewers) are doing, is really not helpful in this context.
>>
>
> It does seem to me that the suggestion of a Long-Term Support (LTS) release is a constructive one, among the other suggestions that Joanna made. The FML you cite is interesting, but seems to be aimed squarely at developers. You are bound to get better results if the expectations of users change along with developers, so LTS releases may be the better idea.

The FML is clearly targeted at developers - actually more at vendors who contribute to the project.

As for a LTS release, I assume what you meant is to allow security conscious people to have a long-term baseline of Xen, which does not include new features. We do have mechanisms for this in the community today: any vendor, individual or group of vendors are very welcome to own and maintain such a LTS release with support and backing from the project. In fact, this is not that hard and does not require that much effort. And I would also be willing to support such a proposal if some people stepped up and see whether more interested parties can be pulled in. To create a security hardened LTS branch however - which it seems you and others are arguing for - would of course be significantly more expensive.

That is not to say that we are unwilling to improve: I believe the project has a track record of improving working practices. But in the end trade-offs have to be made somewhere. And a realistic compromise will most likely fall short of a security hardened Xen LTS.

I don't think it is acceptable though to ask others to foot the bill for it, without contributing something. In particular if what we do today is good enough for those who put the effort in. That approach very much feels like "getting a free lunch and complaining that it isn't what you were hoping for".

Regards
Lars

[Lars Kurth]

> On 11 Nov 2015, at 09:59, Lars Kurth <lars.ku...@gmail.com> wrote:
>
>>> but it doesn't seem
>>> to have a lot of community effort behind it and it too attempts to
>>> install dependencies on my machine and wants to be run with sudo.
>>
>> I believe it has a mode where it simply checks for dependencies and tells
>> you what is required and thereby avoids the need for sudo, but I'm not
>> sure.
>
> It seems that raisin may provide a good baseline the for "build process security", but it would of course be good to hear this from others who have raised this issue. Assuming it is (we probably need a few ACKs for this), would it make sense to take this into a separate thread then (with an appropriate CC list), and refer to it from here?


The details of this are discussed at http://lists.xenproject.org/archives/html/xen-devel/2015-11/msg01045.html now
Lars

[Lars Kurth]

> On 11 Nov 2015, at 09:43, Ian Campbell <Ian.Ca...@citrix.com> wrote:
>
>> Project Raisin is aiming to help with this
>
> Indeed, and it might also allow us to make some of the above options the
> default in the future.
>
> Maybe in the meantime perhaps a ./configure --ensure-offline or --disable-
> downloads which:
> * either disables stubdoms automatically or checks you've passed --
> disable-stubdom as well
> * either disables all the other things which might be cloned or requires
> the corresponding --with-system-foo=, or has a guess at a default system
> version
> * sets FETCHER to /bin/false
>
> would be useful? (essentially as a guard against new options being required
> to turn stuff off).

>
>> but it doesn't seem
>> to have a lot of community effort behind it and it too attempts to
>> install dependencies on my machine and wants to be run with sudo.
>
> I believe it has a mode where it simply checks for dependencies and tells
> you what is required and thereby avoids the need for sudo, but I'm not
> sure.

It seems that raisin may provide a good baseline the for "build process security", but it would of course be good to hear this from others who have raised this issue. Assuming it is (we probably need a few ACKs for this), would it make sense to take this into a separate thread then (with an appropriate CC list), and refer to it from here?

Regards
Lars

[chris]

This is true but I can see her point of view as someone with a product that is based on the open source core. There have been a lot of very drastic changes that would cause someone with a product to have to do a significant amount of re-engineering. Also how many different toolstacks have their been?

I also see both sides but Joanna definitely has some very good points

chris