e2g 3.0.4 urlregexplist not working

[Mike K]

In order for proper filtering of searches with google, we found we need to redirect the https request to a non-https request. Using e2guardian -v shows 3.0.4. Squid version is 3.5.5 with ssl-bump enabled and working.

So this was what we did in urlregexplist:
"^https://www.google.[a-z]{2,6}(.*)"->"http://www.google.com/?nord=1"

It works fine the first day, but at some point it stops working (I haven't pinpointed when it stopped working). When attempting to go to google.com or use any kind of toolbar or search box in the browser, it fails, just displaying an error message.

The e2guardian log shows:
2015.6.29 15:41:26 - 192.168.2.110 https://www.google.com:443 *URLMOD*  CONNECT 3845 0  1 200 -     - -

The squid log shows:
1435610486.999     54 127.0.0.1 TAG_NONE/503 0 CONNECT http:443 - HIER_NONE/- -


So e2guardian does not seem to be passing along the redirect to squid properly, so all squid sees is empty request to http:443 which of course is not a valid website.
We even tried using urlredirectregexplist as 'https://www.google.*'->'http://www.google.com/?nord=1' but that does not work at all. All requests no matter what rules I've tried all have the browser trying to go back to "https://www.google.com/?gws_rd=ssl" with enabled ssl, but it fails every time.

Suggestions or may be a bug?

Mike

[Philip Pearce]

Mike,

The reason that your regexs do not appear to be working is that Google withdrew all http access to search on the 23rd June 2015 (originally scheduled for January 2015).   Google will now always redirect any attempt to access on http to https.  See http://protex.e2bn.org for more information on this.

In order to get proper filtering for Google working you will need to upgrade to e2guardian v3.1.2. or the lastest development commit and enable the SSL interception feature for the Google sites your users access.

So no bug in urlregexp handling.

Regards

Philip Pearce

 

---

--
You received this message because you are subscribed to the Google Groups "e2guardian" group.
To unsubscribe from this group and stop receiving emails from it, send an email to e2guardian+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike K]

I have 3.1.2 now installed but the searches are still not filtered. Typing in the normally blocked search terms and google still displays the search results, videos, and images.


Mike

[Mike K]

I recompiled after finding a new configure option, now google refuses to load at all saying certificate is not valid. The compile setup I used:

./autogen.sh

./configure '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-piddir=/var/run' '--with-logdir=/var/log' '--with-proxyuser=e2guardian' '--with-proxygroup=e2guardian' '--enable-pcre=yes' '--enable-locallists' '--with-filedescriptors=1024' '--with-sysconfsubdir=

e2guardian' '--enable-sslmitm=yes'

make
make install

then I go over the e2guardian.conf and *f1.conf as it has worked before. Most site work properly but google refuses to load due to certificate errors. I had already created and used a certificate on this server but it refuses to accept it.

The /var/log/messages shows "Jun 30 12:02:18 A7750 e2guardian[6981]: error creating certificate sub-directory"

The related cert entries in e2guardian.conf:

sslcertificatepath = '/var/lib/squid_ssl_db/certs/'
cacertificatepath = '/etc/squid/ssl/squid.pem'
caprivatekeypath = '/etc/squid/ssl/squid.key'
certprivatekeypath = '/etc/squid/ssl/squid.key'
generatedcertpath = '/var/lib/squid_ssl_db/certs/'
generatedlinkpath = '/etc/e2guardian/generatedlinks/'


Mike


On Tuesday, June 30, 2015 at 11:50:41 AM UTC-5, Mike K wrote:

I recompiled after finding a new configure option, now google refuses to load at all saying certificate is not valid. The compile setup I used:

./autogen.sh

./configure '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-piddir=/var/run' '--with-logdir=/var/log' '--with-proxyuser=e2guardian' '--with-proxygroup=e2guardian' '--enable-pcre=yes' '--enable-locallists' '--with-filedescriptors=1024' '--with-sysconfsubdir=e2guardian' '--enable-sslmitm=yes'

make
make install

then I go over the e2guardian.conf and *f1.conf as it has worked before. Most site work properly but google refuses to load due to certificate errors. I had already created and used a certificate on this server but it refuses to accept it.

[Mike K]

I also followed the info here: https://github.com/e2guardian/e2guardian/blob/develop/notes/ssl_mitm

The problem using that method (exactly including file names for testing) is the resulting pages show this error (from Firefox):

Secure Connection Failed

An error occurred during a connection to github.com.
SSL received a record that exceeded the maximum permissible length.
(Error code: ssl_error_rx_record_too_long)

Forcing use of the .der file is not an option as there would be no way to implement this among thousands of people that are using existing servers with 3.0.4.

In the end just looking for a way to filter google.... but this certificate stuff is such a pain. They just need to keep the insecure site option active for filter providers (which I suspect is the very reason why they killed the insecure site).

Mike

[Philip Pearce]

Mike,

The generatedcertpath directory must be owned and writable by the e2guardian user and must be a different directory to sslcertificatepath.   Sslcertificatepath should contain the standard CA root certs which are used to check target server certs.   generatedcertpath holds the 'fake' server certs that are sent to the client browser.  

It looks as if you are sharing directories with squid which may explain your issues - the generated directories must be separate and owned by the e2guardian user.

Also it is usual to use a different private key to sign the 'fake' certs with (caprivatekeypath) than is used when talking as a client to the target servers (certprivatekeypath) although I can not see any reason why using the same one should not work.

Also, see the instructions in notes/ssl_mitm

Philip

[Mike K]

At this point we are trying to avoid using the certificate related aspect, since rolling out a new setup that requires the certificate in the individual browsers, adding it to the installers and trying to get thousands of existing customers updated is not an option.... So the issue still remains, urlregexplist is not working. In one simple test, we added a simple entry like this:
"^https://www.google.*/*"->"http://www.bing.com"
yet e2g does not redirect, instead it displays the "Page cannot be displayed" with the google url still in the address line. We even tried the urlredirectregexplist using something like *www.google.*/* -> http://www.bing.com and that is not working either as if it never takes effect, google still loads normally.

So essentially at this point we're realizing there may be no way to filter any searches within google or youtube through e2g. It is only after they get the results and try to click on the link is that new link sent through e2g and is then filtered, but that means any inappropriate word or phrase used also displays the pictures and videos.


Mike

[Philip Pearce]

Mike,

Here at Protex we had a number of discussions with Google regarding this last October-December.  They made very clear to us that if we wanted to continue to provide search term blocking etc on Google search the only way to do it was by using MITM.   As far as they are concerned their forcesafesearch option is good enough for all users, including young children.   This is not our view, and is certainly not the view of our customers (mainly schools).   It was the pressure applied to Google by ourselves and other web filtering providers which resulted in them postponing the end of life date for nosslsearch by 6 months to June, in order to enable web filtering providers to put in place other solutions.

We rolled out our root CA certificate to hundreds of thousands of customer end devices during a period of 3 weeks over the end of December and early January with very few issues.

In the past I was very dubious about MITM, but having implemented and tested it, I can see the huge benefits - full filtering by full url and content, readable block pages on https sites, and increased security on many customer sites as users are not able to override certificate checking and accept self-signed or forged certs and generally better customer happiness!

To recap on the details in your comment:-

urlregexp does work properly in it's intended use to modify on the fly an http url (and only the url part).

It is only possible to rewrite an https site to another https site where the new site is expecting a request for the original site and sends back the original site's cert - this is what Google did with nosslsearch - Google (NOT e2g) then sent a redirect back to the browser to go to http://www.google.com with a special flag so that http://www.google.com did not then redirect back to http://www.google.com and cause a loop.

If you rewrite a https site to a http one it CANNOT work as the browser is expecting an SSL connection and will see http return as an error and/or attack.

urlredirectregexplist will only work with http sites - Why, because it's violating the https protocol for a proxy to redirect the http connect request made by the browser and all modern browsers will reject the redirect.  

However, once you enable MITM with certs you are able to modify the url and do redirection on https sites as e2g is now talking http (via an SSL tunnel) to the browser, and you will find that your original http://www.google.com/..... etc  rewrites will now work once again.

Regards

Philip Pearce
Technical Director
E2BN Protex Ltd
w: protex.e2bn.org