well this is interesting - a security vulnerability in mod_pagespeed (also web1 and web3 no longer accessible directly)

[habeanf]

hey guys.

for a while now we've been having issues with our web servers.. and today we may have finally figured out why.

as most of you know, for rendering performance oknesset.org uses google's mod_pagespeed apache module.

we used to serve directly from apache with mod_wsgi but a few weeks ago we moved to gunicorn running through apache with mod_proxy and pagespeed

anyhow - we've had a number of servers get tied up in very high cpu usage without any explanation.

in the beginning, we thought it was just mod_wsgi, and gunicorn did help but still - web1 wasn't working until this afternoon, again due to high cpu.

so after some digging it turns out mod_pagespeed has/had a serious vulnerability: it could be used to cause the web server to retrieve files from other servers.

and indeed, we saw a large amount of requests in our log files to domains other than oknesset.org

apparently there is an attack, specifically on web1.oknesset.org that accesses the server directly but then sends requests to http://<some other domain>/<for something else>

mod_pagespeed has/had a vulnerability that makes it actually go get that file -> many such requests kill the server

anyhow - how does all this affect you?

1. as an initial defensive measure we've blocked access on port 80 directly to any web server.

all port 80 requests go through the load balancer (which should only answer to requests for oknesset.org)

2. api.oknesset.org might be unresponsive while the dns caches update the remapping from web1.oknesset.org to the load balancer

hopefully this will solve our high cpu problem

cheerio mates,

amir and meir