Error 502 when running PulledPork rule-update

[Jeronimo L. Cabral]

Dear, I have installed Security Onion 12.04.4 and I've setup the proxy variables in /etc/environment.

When I run manually the /usr/bin/rule-update command, I get the following error:

Running PulledPork.

        Error 502 when fetching https://rules.emergingthreatspro.com/open/snort-2.9.5/emerging.rules.tar.gz.md5 at /usr/bin/pulledpork.pl line 458

        main::md5file('open', 'emerging.rules.tar.gz', '/tmp/', 'https://rules.emergingthreatspro.com/open/snort-2.9.5/') called at /usr/bin/pulledpork.pl line 1782

Can you help me in solve the problem please ???

Special thanks,

JeLo

    

[Doug Burks]

Hi JeLo,

Did you follow our Proxy page?
https://code.google.com/p/security-onion/wiki/Proxy

Have you tried running "sudo -i rule-update" to see if that makes any
difference?

Please attach your /etc/environment file (redacted if necessary).

What kind of proxy is it?

Does the proxy do any HTTPS MITM?

Does the proxy allow you to connect to https://rules.emergingthreatspro.com?

Are you able to access that URL from a standard browser?

> --
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks

[Jeronimo L. Cabral]

Dear Doug, my response is inline below...I continue with the problem....

Did you follow our Proxy page?
https://code.google.com/p/security-onion/wiki/Proxy

Yes

 

Have you tried running "sudo -i rule-update" to see if that makes any
difference?

Yes but it's the same:

Running PulledPork.

        Error 502 when fetching https://rules.emergingthreatspro.com/open/snort-2.9.5/emerging.rules.tar.gz.md5 at /usr/bin/pulledpork.pl line 458

        main::md5file('open', 'emerging.rules.tar.gz', '/tmp/', 'https://rules.emergingthreatspro.com/open/snort-2.9.5/') called at /usr/bin/pulledpork.pl line 1782

    http://code.google.com/p/pulledpork/

 

Please attach your /etc/environment file (redacted if necessary).

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games"

export http_proxy=http://10.4.133.1:4800

export https_proxy=http://10.4.133.1:4800

export ftp_proxy=http://10.4.133.1:4800

export PERL_LWP_ENV_PROXY=http://10.4.133.1:4800

export no_proxy="localhost,127.0.0.1"

 What kind of proxy is it?

It's a Squid HTTP proxy 

Does the proxy do any HTTPS MITM?

No

 

Does the proxy allow you to connect to https://rules.emergingthreatspro.com?

Yes

 

Are you able to access that URL from a standard browser?

Yes, perfectly.

[Doug Burks]

Have you checked your Squid proxy logs for any related clues?

[Jeronimo L. Cabral]

Dear Doug, my Squid access log when I try to update the ET rules is:

TCP_MISS/502 1483 GET https://rules.emergingthreatspro.com/open/snort-2.9.5/emerging.rules.tar.gz.md5 - DIRECT/96.43.137.98 text/    html

But when I go there with my browser I can do it perfectly....maybe the rule-update script add something that Squid doesn't like.....what do you think ???

Thanks again !!!

JeLo

[Brian]

Sounds like the issue I have/had. Everything else configured correctly and as mentioned in other troubleshooting threads, but no dice. If you notice in the squid log, the request uses the GET method for an HTTPS request, which is... strange. It should be using CONNECT. I know others are successfully using proxies, but perhaps squid (or the version we are both using) is more "sensitive" to that?

Anyway, there seems to have been a fundamental change in SSL behavior in the LWP 6+ perl module (which is what PP uses). Only way I was able to get it working was install a back-level version - 5.837, in my case.

Might want to test that in a VM first in your environment, though.

[Mike Pilkington]

I've had similar problems too.  I think same as Brian.  Specifically with a 500 error on Blue Coat.  I added it to this Pulled Pork issue:  

https://code.google.com/p/pulledpork/issues/detail?id=154

As a workaround, I ended up changing to http in /etc/nsm/pulledpork/pulledpork.conf.

-Mike

[Jeronimo L. Cabral]

Dear Mike, with the change as you mentioned: http in place of https in /etc/nsm/pulledpork/pulledpork.conf....did you success in downloading the ET rules ???

Please tell me because now I'm not at office and I can't test this change.

Special thanks,

JeLo

[Wayne Veilleux]

Mike,
I just had the same problem with pulledpork with our new MS ISA proxy and changing https to http in pulledpork.conf as you mentioned make it work properly. (http://rules.emergingthreatspro.com/|emerging.rules.tar.gz|open)
Thanks :)

[Bruno Savioli]

Hi All,

I had problems with proxy for a while and changing the rule_url to http had fixed it, however, it looks like snort is not making the rules available over http anymore
curl -v http://www.snort.org/rules/snortrules-snapshot-2956.tar.gz?oinkcode=111111111111111111
* About to connect() to proxy proxy1 port 3128 (#0)
* Trying 10.5.13.181... connected
> GET http://www.snort.org/rules/snortrules-snapshot-2956.tar.gz?oinkcode=111111111111111 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: www.snort.org
> Accept: */*
> Proxy-Connection: Keep-Alive
>
* HTTP 1.0, assume close after body
< HTTP/1.0 301 Moved Permanently
< Server: Cowboy
< Location: https://www.snort.org/rules/snortrules-snapshot-2956.tar.gz?oinkcode=1111111111111111
< Content-Type: text/html
< Status: 301 Moved Permanently
< Date: Thu, 17 Jul 2014 10:16:01 GMT
< X-Cache: MISS from localhost
< X-Cache-Lookup: MISS from localhost:3128
< Via: 1.1 vegur, 1.0 localhost (squid/3.1.19)
< Connection: close

From what I have been reading, it's todo with the perl library used by pulledpork.

Anyone has a workaround?

Thanks,

Bruno

[Doug Burks]

Hi Bruno,

This is most likely due to the recent snort.org changes:
http://blog.snort.org/2014/07/the-new-snortorg-is-here.html
http://blog.snort.org/2014/07/snort-subscriber-rule-set-update.html

You might want to check with the VRT folks and see if they have an
option for keeping the connection http instead of redirecting to
https.

You might also want to check with the PulledPork folks to see if they
have any workarounds for better https proxy behavior.

http://securityonionsolutions.com

[Matt Peterschlingmann]

Confirming this issue is present in environments where the proxy server does not support GET requests for HTTPS.

The same request to wget the snort rules via the proxy works fine.

Quick packet captures demonstrates that wget uses CONNECT rather than pulledpork which uses GET method.

Fastest solution would be to permit your snorby box out to the internet for updates bypassing the proxy, until pulledpork supports HTTPS via CONNECT method.