Security Patches and Rails 2.3.11/3.0.4

[fowlduck]

The release of Rails 2.3.11 and 3.0.4 included a critical security fix
for a CSRF protection bypass vulnerability. It's great that this was
caught and, other than a couple things which I'll explain in a moment,
I think it's being handled very well.

This release, however, highlights a flaw in the release process with
regard to security fixes, one which I think it fairly easy to fix. The
release that seemed like merely a security fix to many also possibly
includes some bugs. This could prevent those on 3.0.3 (or earlier)
from upgrading to 3.0.4:

http://groups.google.com/group/rubyonrails-core/browse_thread/thread/f013d01fa561abe5

Other issues arise, however, if you're on a version well before the
latest release. We have an application on Rails 2.3.5 still and we
were tasked with figuring out how to apply this security fix to the
application without the pain of upgrading to 2.3.11. Other than this
security fix, we don't have a reason to upgrade just yet.

"Just apply the patch," you might say. While this patch is handy, it
assumes a vendored rails that you are comfortable with modifying in
this. For example, suppose you need to upgrade from 2.3.5 to 2.3.7 and
you re-vendor. Make sure you apply that patch again! And suppose you
avoid vendoring rails as I, and likely many others, am in the practice
of doing, how are you to apply this patch? It's obvious, right? Just
monkey-patch the appropriate methods in rails.

Except, since the code to monkey-patch isn't provided in a centralized
way (like, oh, say the REXML vulnerability in 2008:
http://weblog.rubyonrails.org/2008/8/23/dos-vulnerabilities-in-rexml
), the labor of this task has been multiplied to everyone in a similar
situation to the one I find myself in.

So, I submit this code to you. The tests work on Rails 2.3.5 - 2.3.10:

https://gist.github.com/818691

I'd encourage anyone who's using rails 3 and is in a similar situation
to provide the same.

[fowlduck]

All that and I forgot to mention that the flaw in the release process
was that this wasn't provided along with the patches. I propose that
security releases in the future include a file to monkey-patch the fix
into a reasonable number of previous patch releases. Maybe for
releases back to patch 0 of the current minor release?

[José Valim]

1) I agree with the commented flaw in this release. Next time, we
should do a security release from the previous tag, this is even
easier because we won't need to release a RC.

2) In any case, we have released an RC that reduces the changes of
having regressions. The e-mail you pointed is not really a regression,
but the result of fixing an html safety issue in Rails.

3) I disagree that Rails Core should provide monkey patches for
security releases, except the cases it is the only option. Rails
provides the ability to be vendored exactly so you can apply important
patches, like the ones in security releases. If someone wishes for a
reason completely external to Rails to not vendor it, we cannot and
should not be responsible to support it. This also applies for those
who decidedly do not upgrade their Rails versions when tiny versions
are released. We cannot and should not support all Rails versions
released.

As you already said, there is *a lot* of effort in doing a Rails
Security Release. We provide patches not only for 3.0.x, but also
2.1.x, 2.2.x and 2.3.x. Asking for monkey patches because a small
niche prefers to not vendor or update their Rails versions is missing
the big picture and the amount of effort that is already put into a
security release.

Finally, I appreciate your effort on proposing suggestions to the
release process. I also want to thank Koz for doing those extremely
important security releases.

[fowlduck]

1) I have a hard time believing that the majority of shops upgrade
every when every patch release comes out despite not being affected by
a bug, not getting a desirable feature, or not having a security
issue. I do not think we're a niche at all. Certainly if there's a
compelling reason to upgrade, I upgrade.

2) The patch provided by rails core doesn't work on 2.3.2-2.3.4 due to
form_authenticity_param being missing and doesn't work on 2.3.5 due to
the lack of the html_safe method. Applying the patch to vendored
rails, in this case, would have resulted in a broken app (which even
if fixed may not work as expected).

Which versions of rails are considered supported with regard to
security fixes, then? That's a compelling reason to upgrade with every
patch release.

[fowlduck]

Also, it would be great to include which versions are supported on the
security page: http://rubyonrails.org/security

[José Valim]

The tags on 3-0-stable, 2-3-stable, 2-2-stable and 2-1-stable are
currently patched when there is a security release. This means the
latest tiny versions of these branches are the ones supported when
there is a security release. It is always recommended to stay up-to-
date to the latest tiny version in your minor release.

It is likely that when 3.1 is released, we will drop support to 2.2.x
and 2.1.x versions.

[Jason King]

On Thu, Feb 10, 2011 at 9:28 AM, fowlduck <nathan...@gmail.com> wrote: 

2) The patch provided by rails core doesn't work on 2.3.2-2.3.4 due to
form_authenticity_param being missing and doesn't work on 2.3.5 due to
the lack of the html_safe method. Applying the patch to vendored
rails, in this case, would have resulted in a broken app (which even
if fixed may not work as expected).

I think you're misunderstanding what the last number in 2.3.2, 2.3.4 etc. means.  And everyone is using the word "version" to mean two different things here.

 

Which versions of rails are considered supported with regard to
security fixes, then? That's a compelling reason to upgrade with every
patch release.

Yes it is.

The versions of Rails supported are 2.3 and 3.0 (although José says 2.1 and 2.2 as well) - which is why both had a patch release with the security patches.  In the proper sense of the word, "upper-case V" Version if you like, 2.3.5 is not a version of Rails, it's patch release 5 of version 2.3.

2 = Major version

3. = Minor version

5 = Patch number

Cheers,

Jason

[José Valim]

Jason, you are actually right. 2-2-stable and 2-1-stable are not
supported anymore even though we provided patches for the CSRF fix:

"Please note that only the 2.3.x and 3.0.x series are supported at
present. Users of earlier unsupported releases are advised to
upgrade
as soon as possible as we cannot guarantee continued security fixes
indefinitely."

http://groups.google.com/group/rubyonrails-security/browse_thread/thread/2d95a3cc23e03665

So it is likely that we will drop 2-3-stable support once 3.1 is out.

On Feb 10, 7:00 pm, Jason King <j...@handle.it> wrote:

> On Thu, Feb 10, 2011 at 9:28 AM, fowlduck <nathan.sut...@gmail.com> wrote:
>
> > 2) The patch provided by rails core doesn't work on 2.3.2-2.3.4 due to
> > form_authenticity_param being missing and doesn't work on 2.3.5 due to
> > the lack of the html_safe method. Applying the patch to vendored
> > rails, in this case, would have resulted in a broken app (which even
> > if fixed may not work as expected).
>
> I think you're misunderstanding what the last number in 2.3.2, 2.3.4 etc.
> means.  And everyone is using the word "version" to mean two different
> things here.
>
> > Which versions of rails are considered supported with regard to
> > security fixes, then? That's a compelling reason to upgrade with every
> > patch release.
>
> Yes it is.
>

> The *versions* of Rails supported are 2.3 and 3.0 (although José says 2.1

> and 2.2 as well) - which is why both had a patch release with the security
> patches.  In the proper sense of the word, "upper-case V" Version if you

> like, 2.3.5 is not a *version* of Rails, it's patch release 5 of version

[fowlduck]

I screwed up and did "Reply to author" instead of "Reply". I fully
admit it, I'm a google groups noob. Perhaps Jason will be kind enough
to return my post or post it for me? :)

[Jason King]

Nathan's message....

---------- Forwarded message ----------
From: fowlduck <nathan...@gmail.com>
Date: Thu, Feb 10, 2011 at 10:55 AM
Subject: Re: Security Patches and Rails 2.3.11/3.0.4
To: Jason King <j...@handle.it>

> I think you're misunderstanding what the last number in 2.3.2, 2.3.4 etc.
> means.  And everyone is using the word "version" to mean two different
> things here.
>

> The *versions* of Rails supported are 2.3 and 3.0 (although José says 2.1

> and 2.2 as well) - which is why both had a patch release with the security
> patches.  In the proper sense of the word, "upper-case V" Version if you

> like, 2.3.5 is not a *version* of Rails, it's patch release 5 of version

> 2.3.
>
> 2 = Major version
> 3. = Minor version
> 5 = Patch number

1) Semantic Versioning calls them versions and it's part of the
version number.

2) The post on the new releases disagrees with your usage of the term:
http://weblog.rubyonrails.org/2011/2/8/new-releases-2-3-11-and-3-0-4

3) http://weblog.rubyonrails.org/2011/2/8/csrf-protection-bypass-in-ruby-on-rails
calls what you're calling "versions," the 2.3 and 3.0 "release
series." This makes it unclear which patches are supported.

4) On top of it all, rails patch versions often introduce backwards
incompatible changes, so they're closer to minor versions than
patches.

So if you want to get into semantics, I think it's clear that they're
versions or at least called versions. The treatment of them as simple
patches that are trivial to upgrade (meaning no changes to your app)
is unfounded.

[Jason King]

---------- Forwarded message ----------
From: fowlduck <nathan...@gmail.com>
Date: Thu, Feb 10, 2011 at 10:55 AM
Subject: Re: Security Patches and Rails 2.3.11/3.0.4
To: Jason King <j...@handle.it>

> I think you're misunderstanding what the last number in 2.3.2, 2.3.4 etc.
> means.  And everyone is using the word "version" to mean two different
> things here.
>

> The *versions* of Rails supported are 2.3 and 3.0 (although José says 2.1
> and 2.2 as well) - which is why both had a patch release with the security
> patches.  In the proper sense of the word, "upper-case V" Version if you
> like, 2.3.5 is not a *version* of Rails, it's patch release 5 of version
> 2.3.
>
> 2 = Major version
> 3. = Minor version
> 5 = Patch number

1) Semantic Versioning calls them versions and it's part of the
version number.

2) The post on the new releases disagrees with your usage of the term:
http://weblog.rubyonrails.org/2011/2/8/new-releases-2-3-11-and-3-0-4

3) http://weblog.rubyonrails.org/2011/2/8/csrf-protection-bypass-in-ruby-on-rails
calls what you're calling "versions," the 2.3 and 3.0 "release
series." This makes it unclear which patches are supported.

All of that is what I meant when I said "everyone is using the word "version" to mean two different things here".

 

4) On top of it all, rails patch versions often introduce backwards
incompatible changes, so they're closer to minor versions than
patches.

Agreed, the Rails project doesn't always attain perfection.  The aim though is to make patch releases backwardly compatible.

 

So if you want to get into semantics, I think it's clear that they're
versions or at least called versions. The treatment of them as simple
patches that are trivial to upgrade (meaning no changes to your app)
is unfounded.

For the most part, and (more importantly) barring unintentional bugs, this is precisely the intention of patch releases.

Anyway, what are you suggesting?  To maintain a security patch release on top of each of the releases?  That will just shift the problem to the next level - ie. there'll *still* be people who want the 2.3.5.2 patch but don't want 2.3.5.1 because it breaks something they rely on (less likely than now, yes, but impossible to eliminate altogether).  People will still need to resort to just pulling the patch and applying it to a vendored Rails.

[Michael Koziarski]

Rather than getting mired in some semantic argument about versions
versus patches, I'll just outline the security policy (and update the
website once I figure out how to log in).

We support the current release branch (3-0-stable) for bugfixes and
the release branch before that (2-3-stable) for severe bug fixes and
security issues. *all other branches* are unsupported.

Now in reality it's often quite straightforward for us to backport
security fixes to 2-2-stable and 2-1-stable so depending on the
severity and the work involved, I'll usually do that. However that's
a 'best effort' thing, there's absolutely no promises there. Once 3.1
ships officially we wouldn't be supporting 2-3-stable, however given
the magnitude of the upgrade I'll personally commit to keeping
2-3-stable support throughout the 3.1 lifetime. If I'm hit by a bus
though, that may not be something the others can do.

What this means is that you should always be running the latest point
release of your release series, for 2.3 users that means you should be
on 2.3.11. I realise that sometimes we fuck up a release or ... 4...
and it can be a bit of a rough ride, however 2.3.11 should have
finally resolved all the backwards incompatible changes we were
notified about.

I don't want to sound like a matyr but to put things in perspective,
those two security releases took about 60 hours of my time.
backporting the patches, verifying the fixes, making sure other
versions weren't affected etc. Doing that for every point release in
a support series is infeasible.

If there's something blocking you from running 2.3.11 please make sure
there's a lighthouse ticket describing it so we can fix it. Otherwise
you'll be stuck backporting your own patches.

--
Cheers

Koz

[fowlduck]

> All of that is what I meant when I said "everyone is using the word
> "version" to mean two different things here".

Then you said I was using it wrong, which isn't the case.

> Agreed, the Rails project doesn't always attain perfection.  The aim though
> is to make patch releases backwardly compatible.

Sure, but that's not the problem, it's 1) treating patches as trivial
to upgrade and 2) not being clear as to which version or patch is
supported by security releases.

> For the most part, and (more importantly) barring unintentional bugs, this
> is precisely the intention of patch releases.

Intention vs reality. Brian Cardarella's blog post is apt:
http://bcardarella.com/post/628870702/a-cry-for-standardized-versioning-in-ruby-pretty

This is all an aside, of course.

> Anyway, what are you suggesting?  To maintain a security patch release on
> top of each of the releases?  That will just shift the problem to the next
> level - ie. there'll *still* be people who want the 2.3.5.2 patch but don't
> want 2.3.5.1 because it breaks something they rely on (less likely than now,
> yes, but impossible to eliminate altogether).  People will still need to
> resort to just pulling the patch and applying it to a vendored Rails.

Right now all I'm suggesting, if monkey-patches aren't provided for
those who aren't at the latest patch level, is that the version that
will be supported with security patches should be made clear.

Before this I didn't have a reason to upgrade patch versions unless a
bug affected me. If only the latest patch is supported, then I do.

[Michael Koziarski]

> Before this I didn't have a reason to upgrade patch versions unless a
> bug affected me. If only the latest patch is supported, then I do.

This has always been the case. we have *never* released bug fixes for
specific earlier versions. You should basically always upgrade unless
you have a reason not to, and your reason had better be a good one :)

--
Cheers

Koz

[fowlduck]

> Rather than getting mired in some semantic argument about versions
> versus patches, I'll just outline the security policy (and update the
> website once I figure out how to log in).
>
> We support the current release branch (3-0-stable) for bugfixes and
> the release branch before that (2-3-stable) for severe bug fixes and
> security issues.  *all other branches* are unsupported.

The current release branch maps to the latest tag? Or what?

> Now in reality it's often quite straightforward for us to backport
> security fixes to 2-2-stable and 2-1-stable so depending on the
> severity and the work involved, I'll usually do that.  However that's
> a 'best effort' thing, there's absolutely no promises there.  Once 3.1
> ships officially we wouldn't be supporting 2-3-stable, however given
> the magnitude of the upgrade I'll personally commit to keeping
> 2-3-stable support throughout the 3.1 lifetime.  If I'm hit by a bus
> though, that may not be something the others can do.

That's awesome, I really appreciate it. Upgrading to 3 is on the list,
but our app is huge. :)

> What this means is that you should always be running the latest point
> release of your release series, for 2.3 users that means you should be
> on 2.3.11.  I realise that sometimes we fuck up a release or ... 4...
> and it can be a bit of a rough ride, however 2.3.11 should have
> finally resolved all the backwards incompatible changes we were
> notified about.

Cool, glad this is clear.

> I don't want to sound like a matyr but to put things in perspective,
> those two security releases took about 60 hours of my time.
> backporting the patches, verifying the fixes, making sure other
> versions weren't affected etc.  Doing that for every point release in
> a support series is infeasible.

I didn't realize it took that long, ouch.

> If there's something blocking you from running 2.3.11 please make sure
> there's a lighthouse ticket describing it so we can fix it.  Otherwise
> you'll be stuck backporting your own patches.

I started a branch and am working on it. I'll be sure to report
anything I find.

[fowlduck]

> This has always been the case.  we have *never* released bug fixes for
> specific earlier versions.  You should basically always upgrade unless
> you have a reason not to, and your reason had better be a good one :)

The only one that was for earlier versions that jumped out at me at
the time of writing this was the REXML bug, but I realized that this
was actually a ruby bug, not a rails bug, so it's a bad example.
Actually looking back through the releases, now, and I realize you're
correct.

We'll definitely upgrade patch versions from now on....or wait a few
days, and then upgrade. ;)

[Michael Koziarski]

> The only one that was for earlier versions that jumped out at me at
> the time of writing this was the REXML bug, but I realized that this
> was actually a ruby bug, not a rails bug, so it's a bad example.
> Actually looking back through the releases, now, and I realize you're
> correct.

Right, in that case people needed to wait for upstream packagers etc,
and the monkeypatch was like 5 lines ;)

> We'll definitely upgrade patch versions from now on....or wait a few
> days, and then upgrade. ;)

Even better, track 2-3-stable and 3-0-stable with your app and let us
know when we break something :)

--
Cheers

Koz

[Nathan Sutton]

> Even better, track 2-3-stable and 3-0-stable with your app and let us
> know when we break something :)

Ha, well, I'm sure our users who depend on us would appreciate that. ;)