HTTPS Traffic Log IPTABLE Issue

[SK NZ]

Hello,
I've managed to compile squid3 with SSL support in a standalone(squid3 only) server, now I can monitor HTTPS traffic log without issuing any certificate. I got this idea originally from here: http://blog.manty.net/2014/12/squid-proxy-being-transparent-also-for.html

To implement this in a Grase Hotspot Server, I reinstalled squid3 packages with SSL support, also kept all original Grase configurations. Now I modified squid.conf.grase to enable HTTPS. So far it worked perfectly, squid restarted without any error.

http_port 3128
http_port 3129 intercept
https_port 3127 intercept ssl-bump generate-host-certificates=off cert=/etc/squid3/certs/squid.pem
acl ssl-bump_port myportname 3127
always_direct allow ssl-bump_port

For this new squid ports, default IPTABLE rules will not work. So I tried to modify /etc/chilli/ipub.sh

ipt -I PREROUTING -t mangle -p tcp -s $NET/$MASK -d $ADDR --dport 3129 -j DROP
ipt -I PREROUTING -t nat -i $TUNTAP -p tcp -s $NET/$MASK ! -d $ADDR --dport 80 -j REDIRECT --to 3129
ipt -I PREROUTING -t mangle -p tcp -s $NET/$MASK -d $ADDR --dport 3127 -j DROP
ipt -I PREROUTING -t nat -i $TUNTAP -p tcp -s $NET/$MASK ! -d $ADDR --dport 443 -j REDIRECT --to 3127
ipt -I POSTROUTING -t nat -o $HS_WANIF -j MASQUERADE

Now I can't browse after connecting hotspot. IPTABLE is blocking it somewhere!. If anyone can help to figure it out, that will be really a great-great help.


In my standalone server, I've used this IPTABLE rules and it works like a charm!

*nat
:PREROUTING ACCEPT
:POSTROUTING ACCEPT
:OUTPUT ACCEPT
-A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p tcp --dport 443 -j REDIRECT --to-ports 3127
-A PREROUTING -i eth0 -s 192.168.0.0/16 ! -d 192.168.0.0/16 -p tcp --dport 80 -j REDIRECT --to-ports 3129
COMMIT
*filter
:INPUT DROP
:FORWARD DROP
:OUTPUT ACCEPT
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT 
-A INPUT -i eth0 -p tcp --dport http -j ACCEPT
-A INPUT -i eth0 -p tcp --dport 3127:3128 -j ACCEPT
-A INPUT -i eth0 -p udp --dport bootps -j ACCEPT
-A INPUT -i eth0 -p udp --dport domain -j ACCEPT
-A INPUT -i eth0 -p tcp --dport domain -j ACCEPT
COMMIT

Thanks in advance.


 

[Tim]

It looks like you might need to add a firewall rule to allow 3127 to the Grase server.

However, without installing CA certificates on the client, what benefit do you see from "proxying" HTTPS connections through squid?

My long term plan was to remove Squid from future versions of the hotspot, because the logs are becoming useless due to HTTPS traffic. I believe the only thing you can get is the hostname it's connecting to, and even that may not work with HTTPS 2.

Regards

Tim

--
This mailing list is for the Grase Hotspot Project http://grasehotspot.org
---
You received this message because you are subscribed to the Google Groups "Grase Hotspot" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grase-hotspo...@grasehotspot.org.
To view this discussion on the web visit https://groups.google.com/a/grasehotspot.org/d/msgid/grase-hotspot/f6deda79-b57c-4243-aa26-be9952b2eba6%40grasehotspot.org.

[SK NZ]

Hello Tim,

Thanks for the reply. I've edited the /etc/chilli/config to add port 3127, still no luck. I can't browse HTTP or HTTPS.

HS_TCP_PORTS="80 443 22 2812 53 3990 3127 3128" 

This is  iptables -S :

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -i eth1 -j DROP
-A INPUT -d 10.1.0.1/32 -i tun0 -p icmp -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p udp -m udp --dport 67:68 -j ACCEPT
-A INPUT -d 255.255.255.255/32 -i tun0 -p udp -m udp --dport 67:68 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p tcp -m tcp --dport 3128 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p tcp -m tcp --dport 3127 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p tcp -m tcp --dport 3990 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p tcp -m tcp --dport 2812 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p tcp -m tcp --dport 4990 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -p tcp -m tcp --dport 3990 -j ACCEPT
-A INPUT -d 10.1.0.1/32 -i tun0 -j DROP
-A FORWARD -i tun0 -o eth0 -j ACCEPT
-A FORWARD -i tun0 ! -o eth0 -j DROP
-A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o tun0 -j ACCEPT
-A FORWARD -i tun0 -j ACCEPT
-A FORWARD -o eth1 -j DROP
-A FORWARD -i eth1 -j DROP

This is  iptables -vL : https://ibb.co/SVhTZHV

This is  /var/log/squid3/cache.log : https://paste.grasehotspot.org/view/cfa68b9a

I've tested squid3 locally in the terminal using squidclient, it works and logs for both - HTTP and HTTPS. So I guess it's not a squid issue.

192.168.0.103 TCP_MISS/200 909 POST http://ocsp.digicert.com/ - HIER_DIRECT/117.18.237.29 application/ocsp-response
192.168.0.103 TCP_MISS/200 29835 CONNECT github.githubassets.com:443 - HIER_DIRECT/185.199.109.154 -
192.168.0.103 TCP_MISS/200 81152 CONNECT github.githubassets.com:443 - HIER_DIRECT/185.199.109.154 -
192.168.0.103 TCP_MISS/200 22041 CONNECT github.githubassets.com:443 - HIER_DIRECT/185.199.109.154 -
192.168.0.103 TCP_MISS/200 37913 CONNECT customer-stories-feed.github.com:443 - HIER_DIRECT/185.199.110.153 -
192.168.0.103 TCP_MISS/200 571167 CONNECT customer-stories-feed.github.com:443 - HIER_DIRECT/185.199.110.153 -
192.168.0.103 TCP_MISS/200 3741 CONNECT www.google-analytics.com:443 - HIER_DIRECT/74.125.68.139 -

Yes, you're right, for HTTPS without issuing certificate we'll only get hostnames. Above log is for https-github. At least now we can tell which user is connecting to which https sites, it's better than nothing. I can't provide free public wifi without keeping logs, it's our local compliance. So I've to keep at least some form of logs. Please help me to figure it out the issue. Thanks in advance.

To unsubscribe from this group and stop receiving emails from it, send an email to grase-...@grasehotspot.org.

[Daniel Crusoe]

I am in no way knowledgeable in this, but, I have found that unless you give grase dns servers (not just on the outside NIC, but actually in the software) you have issues with connecting to any sites, and it is an intermittent issue.

 

Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to grase-hotspo...@grasehotspot.org.
To view this discussion on the web visit https://groups.google.com/a/grasehotspot.org/d/msgid/grase-hotspot/2277ade4-9429-4821-8d07-d2f3f6dc387f%40grasehotspot.org.

[SK NZ]

Hello Daniel,
Can you please elaborate what do you mean by give grase dns servers? or how to achieve this? Thanks.

To view this discussion on the web visit https://groups.google.com/a/grasehotspot.org/d/msgid/grase-hotspot/2277ade4-9429-4821-8d07-d2f3f6dc387f%40grasehotspot.org.

[christopher]

Hello Daniel,

I have no idea where you got this idea from. 

I have had grase hotspot installed since 2016 and I have ONLY added the ip address of the router connected to the WAN side and google's public nameserver address in the admin panel of grase, and it has NEVER failed. 

I have been a system administrator for decades, and have used a large number of the linux distros out there at one time or another.  If you are running a mail server you need a dns server.  For a hotspot server that is not publicly serving any internet accessible website, no other configuration is needed, other than to have what is needed for grase.

Christopher.

To view this discussion on the web visit https://groups.google.com/a/grasehotspot.org/d/msgid/grase-hotspot/2277ade4-9429-4821-8d07-d2f3f6dc387f%40grasehotspot.org.

[SK NZ]

Hello Tim,
I'm stuck with this issue for the last 3 days. If you kindly give a look at it that will be really life-saving! I've added port 3127 in the firewall rules as you suggested but no luck yet! I cannot browse HTTP or HTTPS sites under AP hotspot. It's a fresh grase hotspot server, minor modifications are listed below.

iptables, squid, chili rules: https://paste.grasehotspot.org/view/raw/80fcc90f

After a fresh boot, I can test squid from ssh(squidclient  https://grasehotspot.org). It works until 1st user login using AP hotspot, after that squidclient also stops responding. I've checked squid3 status, it says active(running). So I guess iptables or chilli is doing something to squid, that's weird!

Exact same squid configuration works in the same version of squid and OS in a standalone server, but it's not working here. I believe the only difference is iptable/nat rules. Another worthy thing to mention, after a fresh grase hotspot server installation it doesn't work out of the box. For default squid.conf.grase, it throws Error: No forward-proxy ports configure. So I have to add a forward port.

http_port 3128
http_port 3127 intercept

ipt -I PREROUTING -t mangle -p tcp -s $NET/$MASK -d $ADDR --dport 3127 -j DROP

ipt -I PREROUTING -t nat -i $TUNTAP -p tcp -s $NET/$MASK ! -d $ADDR --dport 80 -j REDIRECT --to 3127

I'm using stable Squid 3.4.8 @ Debian Jessie, I guess from this version it's a must to specify a forward port. This stops throwing an error in squid cache log, still, HTTP sites are not working. It shows ERROR/Invalid URL squid error page. So I have to modify squid.conf.grase again to make it work!

http_port 3128 accel vhost allow-direct

It's not a transparent proxy anymore, rather a reverse proxy. Now HTTP sites are working and also monitor session works in grase/radmin. So INTERCEPT is not working for only HTTP or both.

intercept means take the origin server details from the NAT system.
vs.
accel means Squid is providing CDN services for the domain being fetched.

I guess something wrong with NAT here when we use one port for intercept another for forwarding? I've done a lot of digging last 3 days, you may look at this( squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Transparent-intercept-Issues-td4681851.html ). I'm not sure this is relevant for my issue or not.

Thanks in advance.

 

On Wednesday, September 11, 2019 at 3:06:23 AM UTC+6, timwhite88 wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to grase-...@grasehotspot.org.

[christopher]

Hello,

I just did a quick look on the net, and the following may give you the solution:

https://serverfault.com/questions/870319/ip-tables-intercept-exception

Regards,

Christopher.

[SK NZ]

Hello Christopher,
Good to see you again as a savior! I'm a bit confused with the provided link's solution. Can you please help me to adjust with my current rules?

sudo nano /etc/chilli/ipup.sh 

ipt -I PREROUTING -t mangle -p tcp -s $NET/$MASK -d $ADDR --dport 3126 -j DROP
ipt -I PREROUTING -t nat -i $TUNTAP -p tcp -s $NET/$MASK ! -d $ADDR --dport 80 -j REDIRECT --to 3126 

ipt -I PREROUTING -t mangle -p tcp -s $NET/$MASK -d $ADDR --dport 3127 -j DROP
ipt -I PREROUTING -t nat -i $TUNTAP -p tcp -s $NET/$MASK ! -d $ADDR --dport 443 -j REDIRECT --to 3127 

ipt -I POSTROUTING -t nat -o $HS_WANIF -j MASQUERADE

Thanks again.

[Tim]

Hi SK

At this stage Debian Jessie isn't a supported distribution. Obviously things have changed with the Squid reverse proxy. Work is currently progressing on V4 of the hotspot, I suggest until then you try installing on Debian 7, or Ubuntu 14.04. Alternatively, you can remove Squid from the config and have everything working without the Squid proxy. Due to the high number of SSL sites in 2019, Squid caching is not going to be very beneficial to you anymore.

Regards

Tim

To unsubscribe from this group and stop receiving emails from it, send an email to grase-hotspo...@grasehotspot.org.
To view this discussion on the web visit https://groups.google.com/a/grasehotspot.org/d/msgid/grase-hotspot/08020c3b-e607-46fc-9f3b-0cc50e26bb41%40grasehotspot.org.