Patching for the "Ghost" (CVE-2015-0235) Remote Code Execution Vulnerability is causing problems with the Zelig data analysis functionality on DVN v3.6.2

[Joerg Messer]

Greetings,

Patching our DVN server yesterday evening to plug the recently discovered Ghost glibc vulnerability seems to be causing problems with Zelig.  Following the "Access Analysis + Subsetting" link gives the errors below.  Any thoughts on how we might resolve this would be most welcome.  Thanks.

//Joerg

Browser error:
-------------------

"XML Parsing Error: no element found Location: http://dvn.library.ubc.ca/dvn/dv/UBC_RD/faces/subsetting/SubsettingPage.xhtml?dtId=1414&versionNumber=1&studyListingIndex=2_71c309fdf71c71a5ee0e143d1c4d Line Number 1, Column 1:"

Corresponding error cascade in server.log:
---------------------------------------------------------

[#|2015-01-29T11:47:51.623-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=33;_ThreadName=Thread-2;|org.rosuda.REngine.Rserve.RserveException: voidEval failed, request status: error code: 127
...

[#|2015-01-29T11:47:51.624-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=33;_ThreadName=Thread-2;|.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
...

[#|2015-01-29T11:47:51.625-0800|SEVERE|glassfish3.1.2|edu.harvard.iq.dvn.core.web.subsetting|_ThreadID=33;_ThreadName=Thread-2;|AnalysisPage Initialization Failure|#]
...

[#|2015-01-29T11:47:51.626-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=33;_ThreadName=Thread-2;|com.sun.faces.mgbean.ManagedBeanCreationException: Cant instantiate class: edu.harvard.iq.dvn.core.web.subsetting.AnalysisApplicationBean.
...

[#|2015-01-29T11:47:51.627-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=33;_ThreadName=Thread-2;|nstructor.newInstance(Constructor.java:513)
...

[#|2015-01-29T11:47:51.634-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=33;_ThreadName=Thread-2;|javax.servlet.ServletException: Cant instantiate class: edu.harvard.iq.dvn.core.web.subsetting.AnalysisApplicationBean.
...

[#|2015-01-29T11:47:51.634-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=33;_ThreadName=Thread-2;|a:86)
...

[#|2015-01-29T11:47:51.635-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=33;_ThreadName=Thread-2;|javax.servlet.ServletException: Cant instantiate class: edu.harvard.iq.dvn.core.web.subsetting.AnalysisApplicationBean.
..

[#|2015-01-29T11:47:51.636-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=33;_ThreadName=Thread-2;|a:86)
...

[Joerg Messer]

In the interests of full disclosure, we're hosting our DVN service on a Red Hat Enterprise v5.11 system.  A number of other patches where also applied although I did not keep a list.  The server is now fully up to date with all necessary patches.  A colleague also made some changes to our templates around the same time as the Ghost patches were applied. 

[Joerg Messer]

I'm starting to suspect that a change to our default DVN template is the real culprit and not the recent patches.  Not only do we get an error when trying to follow the
"Access Analysis + Subsetting" link but also the "Edit Metadata" link.  To make matters worse, we also get an error when trying to edit the default template.  I'm tempted to promote one of the other templates to default status using "Make Default Template" but I'm worried about making the problem worse.  Has anyone seen anything like this before?  Resetting the default template to "default" may help but that might entail some patching in the database.  Another possibility is to create a clone of one the working templates and make it the default using "Make Default Selection".  Could that work?

[Condon, Kevin]

Hi Joerg,

The error code is very generic and kind of the only thing Rserve sends back. Here are some troubleshooting steps you might try:

1. Just restart rserve. Sometimes this process becomes unresponsive and a simple restart resolves it.
2. Can you telnet to the rserve port, 6311 from the web server?
3. Have you upgraded R at some point recently? That would cause an issue.
4. Does the files, dvn_data_functions.R exist on the web server in<glassfish install>/glassfish/domains/domain1/config?
5. Try loading dvn_data_functions.R into the R on your Rserver and see whether it reports any errors, indicating an issue with the installed packages. Eg.

setwd('/usr/local/glassfish3/glassfish/domains/domain1/config')
source('dvn_data_functions.R')

6. Verify vdc,dsb.rserve options are still in the domain.xml file and the username and password specified in /etc/Rserv.conf and Rserv.pwd on the rserve side and vdc.dsb.host points to the right host.

7. On the Rserve machine, verify /tmp/Rserv and /tmp/VDC exist and are owned by the user rserve and the rserv files in the /etc directory are owned and readble by user rserve.

8. Do not suppress the output from starting rserve:

We normally recommend starting rserve with something like this:
   R CMD Rserve>/dev/null 2>&1
- which hides whatever std output rserve may be producing... So it could be a useful experiment to kill the rserve process; then start it without the /dev/null redirect - simply as

 R CMD Rserve

 on the command line; then try to go to the Analysis page again and see if there are any error messages/anything interesting coming out of rserve.

Let us know if you find anything.

Regards,

Kevin

[Joerg Messer]

Hi Kevin,

I did poke around R a bit and didn't see anything obvious.  Fortunately your comprehensive check list lead to a least one of the problems that we're having.  I worked through #5 and when I tried to load the 'dvn_data_functions.R' file there was an indication that the modules were compiled before the install date of our R version.  Running 'rpm -qi R.x86_64' indicated that the build date was Oct 31/14 so I'm still confused.  Anyways, I ran the 'installModules.sh' script to recompile all the modules and that seemed to clear things up on the R/Zelig front.  I suspect that R must have somehow been updated during our patching in spite of the reported date.  Many thanks for helping us fix this problem.

I might be pushing things a bit but do you have any recommendations on how one might reset the root default template to a default state?  That seems to be our other, likely unrelated, problem. 

//Joerg

[Condon, Kevin]

Joerg,

Good news then on the R front. With respect to the template issue and errors editing metadata and the default template, I would need to know what the errors are and get the server.log from when the problem happens. This should never happen so I am not sure what could cause it. Maybe you can open a support ticket at sup...@dataverse.org so we can discuss offline?

As for resetting the default template, I would need to look at which db tables need to be updated. In theory, you would just overwrite the contents with something else, such as from a newly created and unmodified template or if you know what change you made you can try undoing it.

Does the error editing metadata happen if you create a new study with a different template?

Changing which template is the default should not cause a problem but I would do some testing with various templates, perhaps a new, unmodified one in a test dataverse before making that a new default.

That's all I can think of for now.

Kevin

---

From: dataverse...@googlegroups.com [dataverse...@googlegroups.com] on behalf of Joerg Messer [joerg....@gmail.com]

Sent: Friday, January 30, 2015 7:08 PM
To: dataverse...@googlegroups.com
Subject: Re: [Dataverse-Users] Re: Patching for the "Ghost" (CVE-2015-0235) Remote Code Execution Vulnerability is causing problems with the Zelig data analysis functionality on DVN v3.6.2

Hi Kevin,

I did poke around R a bit and didn't see anything obvious.  Fortunately your comprehensive check list lead to a least one of the problems that we're having.  I worked through #5 and when I tried to load the 'dvn_data_functions.R' file there was an indication that the modules were compiled before the install date of our R version.  Running 'rpm -qi R.x86_64' indicated that the build date was Oct 31/14 so I'm still confused.  Anyways, I ran the 'installModules.sh' script to recompile all the modules and that seemed to clear things up on the R/Zelig front.  I suspect that R must have somehow been updated during our patching in spite of the reported date.  Many thanks for helping us fix this problem.

I might be pushing things a bit but do you have any recommendations on how one might reset the root default template to a default state?  That seems to be our other, likely unrelated, problem. 

//Joerg

[Joerg Messer]

Hi Kevin,

Thanks for your comments.  I'll open a ticket on your support channel describing the template problem more completely.

//Joerg

[Joerg Messer]

Greetings,

Just a quick word of thanks to Kevin and Co for helping to resolve our two DVN problems. 

//Joerg