mod_jk + tomcat 8.0 / 8.5 + ajp connectors

[Austin]

Hello Sakai-Devs,

This is probably basic info to most of you, but I thought I'd share our experience.

Since Tomcat 8.0 will be EOL this summer (http://tomcat.apache.org/tomcat-80-eol.html) we've been preparing to upgrade to Tomcat 8.5 for our Sakai 11.4 installation, but I have a question and a couple of observations.

1) Question: Tomcat 8.5 no longer supports the BIO connector, so we must choose NIO, NIO2, or the APR/native connector.

However, reading through this forum thread:

https://groups.google.com/a/apereo.org/forum/#!search/wrong$20page/sakai-dev/uv12kOG1QH8/IaXh7vYmDwAJ

it sounds like there were some problems with the NIO connector and user sessions getting mixed up and users were able to see other user's sessions.  One of the bug reports in that thread (https://marc.info/?l=tomcat-user&m=148153558926396&w=2) mentions using NIO2 or APR/native connectors might work around the issue (however, that bug report doesn't look like an 'official' tomcat bug report?)

Does this problem only affect Load Balanced instances?  Either HW load balanced in front of sakai servers (this is what we use) or load balancing multiple tomcats on the same server with a proxy?  

Is it still a problem with the latest Tomcat 8.5.28?

2) Observation:  We're fronting tomcat with mod_jk, but it seems like there is a problem with mod_jk and the default NIO connector.  With Tomcat 8.0 we ended up going back to the BIO connector, but since Tomcat 8.5 no longer supports BIO and we still couldn't get mod_jk to work with NIO in tomcat 8.5... using NIO2 worked!  Which according to the 'unofficial' bug report above, "might" also work around the user session problem above... but I'd still be worried about it.  We considered switching to mod_proxy_ajp, but since NIO2 worked, we'll probably stick with that.

3) Observation:  When Tomcat 8.5 extracts war files in webapps, it no longer gives those folders o+rx access, which is good, but since we're using mod_jk to let apache serve files in /webapps/library, /webapps/portal/styles, and /webapps/portal/scripts those folders are no longer accessible by the apache user so we're probably going to give the apache user group access to the group that runs our tomcat.

Thanks,

Austin

[Sam Ottenhoff]

>   One of the bug reports in that thread (https://marc.info/?l=tomcat-user&m=148153558926396&w=2) mentions using NIO2 or APR/native connectors might work around the issue (however, that bug report doesn't look like an 'official' tomcat bug report?)

That looks pretty official to me. It even has a CVE: https://nvd.nist.gov/vuln/detail/CVE-2016-8745

> Does this problem only affect Load Balanced instances?

No, it doesn't seem related to load balancing.

 
>   Either HW load balanced in front of sakai servers (this is what we use) or load balancing multiple tomcats on the same server with a proxy?  
>
> Is it still a problem with the latest Tomcat 8.5.28?

You'd have to ask the Tomcat team, but the post you reference above says it was fixed in 8.5.9.

 
> 2) Observation:  We're fronting tomcat with mod_jk, but it seems like there is a problem with mod_jk and the default NIO connector.

Why are you using two load balancers (hardware LB and HTTPD/mod_jk)?  If you're using mod_jk, then aren't you using the AJP protocol and not NIO (HTTP)?

 
>   With Tomcat 8.0 we ended up going back to the BIO connector, but since Tomcat 8.5 no longer supports BIO and we still couldn't get mod_jk to work with NIO in tomcat 8.5... using NIO2 worked!  Which according to the 'unofficial' bug report above, "might" also work around the user session problem above... but I'd still be worried about it.  We considered switching to mod_proxy_ajp, but since NIO2 worked, we'll probably stick with that.

I recommend APR and the load balancer of your choice in front. It seems like you are using two load balancers. If you have a HW load balancer, connect directly to the Tomcats via HTTP (APR).

--Sam

[Sanghyun Jeon]

We are encountering the similar experiences when we are experimenting tomcat 8.5.28 with Sakai 11.4

S

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+unsubscribe@apereo.org.
To post to this group, send email to saka...@apereo.org.
Visit this group at https://groups.google.com/a/apereo.org/group/sakai-dev/.

[Austin]

Hello Sam,

> You'd have to ask the Tomcat team, but the post you reference above says it was fixed in 8.5.9.

Thanks for the reply.  Good eye, I didn't catch that they recommend 8.5.9 or later.   However,  was a bit confused because it's not super clear if problem described in the sakai-dev forum, where user sessions are getting crossed, is being caused by the issue in the CVE (and the original bugzilla https://bz.apache.org/bugzilla/show_bug.cgi?id=60409) where those had to do with file uploads, however, I guess it also does say "Sharing a Processor can result in information leakage between requests including, not not limited to, session ID and the response body", which sounds like user sessions could get mixed up?

> Why are you using two load balancers (hardware LB and HTTPD/mod_jk)?  If you're using mod_jk, then aren't you using the AJP protocol and not NIO (HTTP)?

We're only using mod_jk for serving static pages with apache.   Yes, we are using AJP protocol with mod_jk.  Even though that CVE bug only mentions the http connectors, I'd still be paranoid that there could be a problem with the NIO (AJP) connector as well... but hopefully not!

> I recommend APR and the load balancer of your choice in front. It seems like you are using two load balancers. If you have a HW load balancer, connect directly to the Tomcats via HTTP (APR).

Again, we're only using mod_jk for serving static pages, so there's only one load balancer.  Would you also recommend APR for AJP protocol?  Or was that recommendation only for HW Load balancer with HTTP?  Also, are the nightly servers Load Balanced? I noticed that in the nightly logs, it looks like they're using NIO.

Thanks,

Austin

[Sam Ottenhoff]

>  Yes, we are using AJP protocol with mod_jk.  Even though that CVE bug only mentions the http connectors, I'd still be paranoid that there could be a problem with the NIO (AJP) connector as well... but hopefully not!

The Tomcat team is very experienced and Apache is a great organization. I trust their published docs and advisories.

> Would you also recommend APR for AJP protocol?  

I would recommend APR, but I wouldn't necessarily recommend AJP. I recommend using Tomcat APR/HTTP with your preferred load balancer. I prefer Nginx but Apache HTTPD and HW load balancers can also do a great job.

> Also, are the nightly servers Load Balanced? I noticed that in the nightly logs, it looks like they're using NIO.

Nightly uses Nginx as the frontend/load balancer with one Tomcat node behind it. Yes, it uses NIO because the Tomcat native APR lib has not been updated recently and HTTP reverse proxying.