A system for rebootless kernel security updates
Jeff Arnold
I've put together an automatic system for applying kernel security patches
to the Linux kernel without rebooting it, and I wanted to share this
system with the community in case others find it useful or interesting.
Here's the summary: The system takes as input a kernel security patch
(which can be a unified diff taken directly from Linus' GIT tree) and the
source code corresponding to the running kernel, and it automatically
creates a set of kernel modules to perform the update. The running kernel
does not need to have been customized in advance in any way. To be fully
automatic, the system cannot be used to apply patches that introduce
semantic changes to data structures, but most Linux kernel security
patches don't make these kinds of changes. I've evaluated the system
against various kernel versions and security vulnerabilities, and the
system can automatically apply 84% of the significant kernel security
patches from May 2005 through December 2007.
I've been pursuing this project because I don't like dealing with reboots
whenever a new local kernel security vulnerability is discovered. The
rebootless update practices/systems that are already out there require
manually constructing an update (through a process that can be tricky and
error-prone), and they tend to have other disadvantages as well (such as
requiring a custom kernel, not handling inline functions properly, etc).
This new system works on existing kernels, and it simply takes a unified
diff as input and does the rest on its own.
The system's website is http://web.mit.edu/ksplice.
The GIT repository, code tarball, and binary tarballs are available here:
http://web.mit.edu/ksplice/ksplice.git
http://web.mit.edu/ksplice/dist/ksplice-src.tar.gz
http://web.mit.edu/ksplice/dist/ksplice-bin-i386.tar.gz
http://web.mit.edu/ksplice/dist/ksplice-bin-x86_64.tar.gz
A document describing how the system works is available here:
http://web.mit.edu/ksplice/doc/ksplice.pdf
Any feedback would be appreciated.
Jeff Arnold
jbar...@mit.edu
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
FD Cami
Jeff Arnold <jbar...@MIT.EDU> wrote:
> Hello,
Hi Jeff,
> I've put together an automatic system for applying kernel security patches
> to the Linux kernel without rebooting it, and I wanted to share this
> system with the community in case others find it useful or interesting.
(reading on)
> Here's the summary: The system takes as input a kernel security patch
> (which can be a unified diff taken directly from Linus' GIT tree) and the
> source code corresponding to the running kernel, and it automatically
> creates a set of kernel modules to perform the update. The running kernel
> does not need to have been customized in advance in any way. To be fully
> automatic, the system cannot be used to apply patches that introduce
> semantic changes to data structures, but most Linux kernel security
> patches don't make these kinds of changes. I've evaluated the system
> against various kernel versions and security vulnerabilities, and the
> system can automatically apply 84% of the significant kernel security
> patches from May 2005 through December 2007.
Awesome. Please note that reading this, I thought at first that the set of
kernel modules were in fact, updated kernel modules (i.e. necessary unloading/
loading of modules) which I understood was not the case after reading your
PDF. After checking with a friend of mine, he understood it like I did.
Perhaps :
- it automatically creates a set of kernel modules to perform the update.
+ it automatically creates a set of kernel modules containing the kernel
+ functions touched by the update, and arranges for the running kernel to
+ use the new functions from now on.
would be better.
> I've been pursuing this project because I don't like dealing with reboots
> whenever a new local kernel security vulnerability is discovered. The
> rebootless update practices/systems that are already out there require
> manually constructing an update (through a process that can be tricky and
> error-prone), and they tend to have other disadvantages as well (such as
> requiring a custom kernel, not handling inline functions properly, etc).
> This new system works on existing kernels, and it simply takes a unified
> diff as input and does the rest on its own.
It really looks like a non intrusive way of achieving superior uptime.
Congrats !
Best,
Francois
Andi Kleen
> I've put together an automatic system for applying kernel security
> patches to the Linux kernel without rebooting it, and I wanted to
> share this system with the community in case others find it useful or
> interesting.
Sounds interesting. You should consider bringing the kernel module
up to kernel coding standards and submitting it for mainline. That
would be the best way to get more users and testers.
Have you also considered to turn it into a generic incremental binary
linker too? I guess most of the infrastructure needed for it is
already there. Even though gold looks very promising long link times
can be still a problem in development.
-Andi
Christian Hesse
> Hello,
>
> I've put together an automatic system for applying kernel security patches
> to the Linux kernel without rebooting it, and I wanted to share this
> system with the community in case others find it useful or interesting.
This sounds really interesting. I just wanted to test on my Gentoo system with
kernel 2.6.25. The tarball creation works, but applying the change brings an
error to syslog:
ksplice_242e9pfc_helper: Unknown symbol init_mm
Any chance to get a fix for that?
--
Regards,
Chris
Tomasz Chmielewski
> I've put together an automatic system for applying kernel security patches
> to the Linux kernel without rebooting it, and I wanted to share this
> system with the community in case others find it useful or interesting.
Hmm, the idea seem to be patented by Microsoft, i.e. this patent from
December 2002:
http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
(and other patents by Microsoft if you search for "hotpatching").
And those patent descriptions, by the way, remind the way kexec works
("A software module is hotpatched by loading a patch into memory and
modifying an instruction in the original module to jump to the patch"),
which was released much earlier... In essence, they patented kexec ;)
--
Tomasz Chmielewski
http://wpkg.org
Andi Kleen
> And those patent descriptions, by the way, remind the way kexec works
> ("A software module is hotpatched by loading a patch into memory and
> modifying an instruction in the original module to jump to the patch"),
> which was released much earlier... In essence, they patented kexec ;)
The basic patching idea is old and has been used many times, long
predating kexec. e.g. it's a common way to implement incremental linkers
too.
-Andi
Jeff Arnold
> tarball creation works, but applying the change brings an error to
> syslog
Sorry. (In 2.6.25, init_mm is marked as an unused symbol, and so whether
it is exported depends on whether CONFIG_UNUSED_SYMBOLS is enabled).
I've committed a change which should fix this problem. You can get get
the new version (0.8.1) from http://web.mit.edu/ksplice/dist/ or from Git.
(I haven't yet made the new version the default, so you won't receive it
from the main Ksplice page).
Jeff Arnold
jbar...@mit.edu
Christian Hesse
> > I just wanted to test on my Gentoo system with kernel 2.6.25. The
> > tarball creation works, but applying the change brings an error to
> > syslog
>
> Sorry. (In 2.6.25, init_mm is marked as an unused symbol, and so whether
> it is exported depends on whether CONFIG_UNUSED_SYMBOLS is enabled).
>
> I've committed a change which should fix this problem. You can get get
> the new version (0.8.1) from http://web.mit.edu/ksplice/dist/ or from Git.
> (I haven't yet made the new version the default, so you won't receive it
> from the main Ksplice page).
This time I get the following in syslog:
ksplice_h: Preparing and checking ksplice_oq7ne7l6
ksplice: Aborted. (stack check: to-be-replaced code is busy)
I tried the example from your website (modifying printk).
Pavel Machek
> Jeff Arnold wrote:
>
> >I've put together an automatic system for applying
> >kernel security patches to the Linux kernel without
> >rebooting it, and I wanted to share this system with
> >the community in case others find it useful or
> >interesting.
>
> Hmm, the idea seem to be patented by Microsoft, i.e.
> this patent from December 2002:
>
> http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
>
> (and other patents by Microsoft if you search for
> "hotpatching").
..so US will not be able to fix security holes without reboot, good.
Perhaps they fix their stupid laws after next worm outbreak...
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Gilles Espinasse
----- Original Message -----
From: "Pavel Machek" <pa...@ucw.cz>
To: "Tomasz Chmielewski" <man...@wpkg.org>
Cc: "LKML" <linux-...@vger.kernel.org>; <jbar...@MIT.EDU>;
<franco...@free.fr>; "Andi Kleen" <an...@firstfloor.org>;
<ma...@earthworm.de>
Sent: Sunday, April 27, 2008 12:17 PM
Subject: Re: A system for rebootless kernel security updates
> On Thu 2008-04-24 16:26:44, Tomasz Chmielewski wrote:
> > Jeff Arnold wrote:
> >
> > >I've put together an automatic system for applying
> > >kernel security patches to the Linux kernel without
> > >rebooting it, and I wanted to share this system with
> > >the community in case others find it useful or
> > >interesting.
> >
> > Hmm, the idea seem to be patented by Microsoft, i.e.
> > this patent from December 2002:
> >
> > http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
> >
> > (and other patents by Microsoft if you search for
> > "hotpatching").
>
> ...so US will not be able to fix security holes without reboot, good.
> Perhaps they fix their stupid laws after next worm outbreak...
>
I have hot-patched on PDP-11 without reboot over the phone line many years
before Microsoft patent.
In the time of MS patent, the machines I care were no more running.
Just another stupid patent.
Gilles
Willy Tarreau
> On Thu 2008-04-24 16:26:44, Tomasz Chmielewski wrote:
> > Jeff Arnold wrote:
> >
> > >I've put together an automatic system for applying
> > >kernel security patches to the Linux kernel without
> > >rebooting it, and I wanted to share this system with
> > >the community in case others find it useful or
> > >interesting.
> >
> > Hmm, the idea seem to be patented by Microsoft, i.e.
> > this patent from December 2002:
> >
> > http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
> >
> > (and other patents by Microsoft if you search for
> > "hotpatching").
>
> Perhaps they fix their stupid laws after next worm outbreak...
Sounds like a bullshit patent. I remember having loaded a lot of NLM
patches under netware 4.0 in 96-97 without ever rebooting. I think
that the patches only redefined the faulty symbol(s) they wanted to
patch. That was pretty convenient because when in doubt, you could
simply unload the modules and get back to previous situation.
> Pavel
Willy
Oliver Pinter
--
Thanks,
Oliver
Jesper Juhl
> On Sun, Apr 27, 2008 at 12:17:00PM +0200, Pavel Machek wrote:
> > On Thu 2008-04-24 16:26:44, Tomasz Chmielewski wrote:
> > > Jeff Arnold wrote:
> > >
> > > >I've put together an automatic system for applying
> > > >kernel security patches to the Linux kernel without
> > > >rebooting it, and I wanted to share this system with
> > > >the community in case others find it useful or
> > > >interesting.
> > >
> > > Hmm, the idea seem to be patented by Microsoft, i.e.
> > > this patent from December 2002:
> > >
> > > http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
> > >
> > > (and other patents by Microsoft if you search for
> > > "hotpatching").
> >
> > ...so US will not be able to fix security holes without reboot, good.
> > Perhaps they fix their stupid laws after next worm outbreak...
>
> Sounds like a bullshit patent. I remember having loaded a lot of NLM
> patches under netware 4.0 in 96-97 without ever rebooting. I think
> that the patches only redefined the faulty symbol(s) they wanted to
> patch. That was pretty convenient because when in doubt, you could
> simply unload the modules and get back to previous situation.
>
And then there's 'alternatives' that patch running code, there's kexec
and I guess you could even say that various root kits that patch the
running kernel get prior art on that patent ;)
--
Jesper Juhl <jespe...@gmail.com>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
Jeff Arnold
This problem has now been resolved (a broken wireless card apparently left
a kernel thread in an unusual state, which led to Ksplice aborting the
upgrade). Christian now has Ksplice working on his system.
If anyone else tries the software and encounters trouble, please let me
know.
Thanks,
Jeff Arnold
jbar...@mit.edu
Jeff Arnold
> standards and submitting it for mainline. That would be the best way to
> get more users and testers.
I'm certainly interested in bringing the code up to kernel coding
standards (for example, I'd be happy to address any issues with the code
that are brought to my attention). I'm not sure whether submitting it for
mainline makes sense since the software doesn't significantly benefit from
being bundled with the kernel.
Instead, it might be more important to 1) package the userspace
update-construction software for common Linux distributions to make it
easily available to interested users, and 2) to provide binary kernel
updates for common distribution kernels so that users can simply sign up
and get fewer "your machine needs to be rebooted now for an update to take
effect" notifications.
I'm open to suggestions about how I should proceed, however.
> Have you also considered to turn it into a generic incremental binary
> linker too?
I've considered modifying the system for use in other contexts, but I
don't think that I'll get around to such a project anytime soon; my
current focus is on making it easier for people to perform rebootless
kernel security updates.
Jeff Arnold
jbar...@mit.edu
Andi Kleen
> I'm certainly interested in bringing the code up to kernel coding
> standards (for example, I'd be happy to address any issues with the code
> that are brought to my attention). I'm not sure whether submitting it
> for mainline makes sense since the software doesn't significantly
> benefit from being bundled with the kernel.
To be honest you weren't the first to come up with something like this
(although you're the first to post to l-k as far as I know). But the
usual problem of something that is kept out of tree is that it
eventually bitrots and gets forgotten. The only sane way to make such
extensions a generically usable linux feature is to merge them to mainline.
> Instead, it might be more important to 1) package the userspace
> update-construction software for common Linux distributions to make it
> easily available to interested users, and 2) to provide binary kernel
> updates for common distribution kernels so that users can simply sign up
> and get fewer "your machine needs to be rebooted now for an update to
> take effect" notifications.
(2) is a incredibly large amount of work longer time. And when
distributions merge your feature they become committed to it so even if
you go away they would still need to maintain it on their own. Since
they understand how much work this is they likely won't do it in the
first place. Really it's far better to just merge if you want it to
make it out of the "toy" stage.
-Andi
Bill Davidsen
> Jeff Arnold wrote:
>
>> I've put together an automatic system for applying kernel security
>> patches to the Linux kernel without rebooting it, and I wanted to
>> share this system with the community in case others find it useful or
>> interesting.
>
> Hmm, the idea seem to be patented by Microsoft, i.e. this patent from
> December 2002:
>
> http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
>
> (and other patents by Microsoft if you search for "hotpatching").
>
>
> And those patent descriptions, by the way, remind the way kexec works
> ("A software module is hotpatched by loading a patch into memory and
> modifying an instruction in the original module to jump to the patch"),
> which was released much earlier... In essence, they patented kexec ;)
>
and if memory serves DTSS, which was in the 1960's. I think MULTICS
allowed that as well, by patching the library dispatch table (sort of
like TLB) to map a virtual address of the entry point to a new location.
All of which doesn't matter, of course, because no one has the money to
challenge MSFT, and it would be in court until the heat death of the
universe anyway.
--
Bill Davidsen <davi...@tmr.com>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot
Jeff Arnold
> feature is to merge them to mainline.
> Really it's far better to just merge if you want it to make it out of
> the "toy" stage.
You're probably right.
I'm willing to undertake the project of bringing the code up to kernel
coding standards so that it can eventually be considered for mainline.
I'll plan on undertaking this project if I don't receive feedback that I
shouldn't do so.
If people have concerns about the high-level design of the system, it
would be useful for me to know that information sooner rather than later.
If there's anything that I can do to facilitate feedback at this stage,
please let me know.
Thanks for the advice.
Jeff Arnold
jbar...@mit.edu
Dan Noe
> If there's anything that I can do to facilitate feedback at this stage,
> please let me know.
Is there a ksplice mailing list? I did not see mention of one on the
web page.
Cheers,
Dan
--
/--------------- - - - - - -
| Dan Noe
| http://isomerica.net/~dpn/
Jeff Arnold
>> please let me know.
> Is there a ksplice mailing list?
I've now created one:
http://mailman.mit.edu/mailman/listinfo/ksplice-devel
Jeff Arnold
jbar...@mit.edu
Jeff Arnold
That's a patent application, not a patent. That application has received
a "final rejection" from the U.S. patent office (Microsoft is appealing,
but so far their attempts at arguing the merits of the application have
not been successful). You can browse the relevant documents at
http://portal.uspto.gov/external/portal/pair
(The application number is 10/307,902).
Jeff Arnold
jbar...@mit.edu
Enrico Weigelt
Hi,
> I'm willing to undertake the project of bringing the code up to kernel
> coding standards so that it can eventually be considered for mainline.
> I'll plan on undertaking this project if I don't receive feedback that I
> shouldn't do so.
Great think :)
I'd actually like to see it mainline tree (I prefer vanilla kernel
instead of distro specific).
> If people have concerns about the high-level design of the system, it
> would be useful for me to know that information sooner rather than later.
I didn't have the time for an deeper study yet, but as you already
mentioned, there're lots of limitations which can make it harmful:
as soon as interfaces chance, you're in *big* trouble. There should
be a way for finding them (automatically). Maybe extract the
interface signatures (including structs!) so some appropriate place
next to the kernel, so they can be checked before (re)loading the
module.
Ah, of course you can't change code that's not an dynamic module :(
Even this goes OT now - I'd really prefer more things in userland,
eg. network- or synthetic filesystems, crypt stuff, etc - so
there would be less to update within the kernel ;-o
cu
--
---------------------------------------------------------------------
Enrico Weigelt == metux IT service - http://www.metux.de/
---------------------------------------------------------------------
Please visit the OpenSource QM Taskforce:
http://wiki.metux.de/public/OpenSource_QM_Taskforce
Patches / Fixes for a lot dozens of packages in dozens of versions:
http://patches.metux.de/
---------------------------------------------------------------------
David Collier-Brown
> I didn't have the time for an deeper study yet, but as you already
> mentioned, there're lots of limitations which can make it harmful:
> as soon as interfaces chance, you're in *big* trouble. There should
> be a way for finding them (automatically). Maybe extract the
> interface signatures (including structs!) so some appropriate place
> next to the kernel, so they can be checked before (re)loading the
> module.
Actually there is a mature discipline around data changes that
dates back to before Unix v7, but it's off-topic for this
discussion: it would make the changes too big (;-))
If anyone is interested, see http://multicians.org/stachour.html
and/or send me mail
--dave (reading LKML when he should be working) c-b
--
David Collier-Brown | Always do right. This will gratify
Sun Microsystems, Toronto | some people and astonish the rest
dav...@sun.com | -- Mark Twain
(905) 943-1983, cell: (647) 833-9377, (800) 555-9786 x56583
bridge: (877) 385-4099 code: 506 9191#