E2guardian interpreted as bot traffic by websites?

[Matthew Lynch]

I receive very frequent captchas from Cloudflare and other websites that significantly slow workflow.  Sometimes they prevent a website from working.  I am running e2guardian v5.5.9r as transparent proxy MITM mode.   Web hosts appear to be interpreting e2guardian MITM as bot traffic.  These problems have been becoming more frequent in the past year.

There are multiple types of problems:

1. Captcha
serverfault.com, stackoverflow.com, quora.com, many others

2. Receive a block page
www.justanswer.com/car/bw062-looking-2017-honda-accord-sport-brake-specs.html
www.pwcgov.org/police

3. Warnings about being rate limited
www.lakeridgeautocare.com
www.google.com - captcha presented with explanation that we have received too many requests from your IP (problem solved if bypass e2g)

4. No response (timeout)
www.thelawnforum.com

5. Receive http 403 error
pubmed.ncbi.nlm.nih.gov


Problem types 2-5 above are worse because I cannot access the page without bypassing e2g.  If I add an exception to MITM or bypass the transparent proxy at the router level, everything works fine for all of these problem websites.  But adding so many exceptions defeats the purpose of having a content filter.


I asked ChatGPT about this and it said it might be a problem with the TLS fingerprint that e2guardian presents to WAN servers.  It also said: "Cloudflare in particular expects modern browsers to use HTTP/2 or 3 — if all your clients downgrade to HTTP/1.1, it looks automated."

It mentioned https://tls.peet.ws/api/all as a means of inspecting the TLS fingerprint and it shows

  "http_version": "HTTP/1.1",

with e2guardian active, and

  "http_version": "h2",
 
with e2guardian bypassed.

Could it be that cloudflare websites see HTTP/1.1 traffic and think it is a bot?  Is there a way to set e2guardian to use HTTP/2 instead of HTTP/1.1?  Is there some other setting in e2guardian that would make it less likely for cloudflare to think it is a bot?  Can someone suggest any other troubleshooting steps?

[Matthew Lynch]

This page can apparently test TLS signatures: https://cloudflare.manfredi.io/test/

It says: "You are using HTTP/1.1...You are not a verified bot and you are 1% human...Your Trust Score is 0." with e2guardian active.

With e2guardian bypassed it says 95-98% human, still trust score 0, also still using HTTP/1.1.  The JA3 and JA4 hashes are different.

[Philip Pearce]

Hi,

 

Using the test pages you have found I done some tests  with 2 versions of openssl libraries (v1.1.1 and v3.0.13).      e2g linked with v3.0.13 gives a human score of 1%.   E2g linked with v1.1.1 gives a score of 87%.

 

Your problematic sites work OK with the e2g linked to v1.1.1.

 

So the issues appear to be  around the openssl versions (and openssl settings) rather than the e2g code itself.

 

I attach this json results from the  https://tls.peet.ws/api/all test.

 

Cloudflare pages are not very helpful hare as to how their bot score is calculated,  so I will do a more detailed analysis of the differences between the two and see if changing openssl v3 settings make any difference to the score when I get the time to do so.

 

 

 

--
E2guardian:
https://groups.google.com/d/forum/e2guardian
Github:
https://github.com/e2guardian/e2guardian
Follow us on twitter:
https://twitter.com/e2guardian
---
You received this message because you are subscribed to the Google Groups "e2guardian" group.
To unsubscribe from this group and stop receiving emails from it, send an email to e2guardian+...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/e2guardian/83be8b73-143d-4c1f-a254-481335ebc4fan%40googlegroups.com.

[Matthew Lynch]

Thank you for looking into this!  I would like to play around with openssl 1.1.1.  I assume I have to:

1. Downgrade my e2guardian server to openssl 1.1.1

2. Recompile e2guardian with options to link to openssl 1.1.1

Is that right?

[Matthew Lynch]

I solved this by adjusting the openssl.conf as you suggested.  I did NOT have to recompile e2guardian or install openssl v1.1.1.  IF ANYONE CAN SEE ANY SECURITY RISK or other inconveniences of this SSL setup, please reply.

I asked AI to generate an openssl.conf file that mimics the openssl 1.1.1 configuration.

This is what it produced:

-------------------------------------------------------

# openssl.cnf — compatibility with OpenSSL 1.1.1-style handshake

openssl_conf = default_conf

[ default_conf ]
ssl_conf = ssl_sect

[ ssl_sect ]
system_default = system_default_sect

[ system_default_sect ]
# --- Protocol Versions ---
MinProtocol = TLSv1.2
MaxProtocol = TLSv1.3 

# --- Cipher Suites ---
# Restrict to ciphers that were commonly available in OpenSSL 1.1.1
CipherString = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:\
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:\
ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305

# --- Supported Groups (Elliptic Curves) ---
# Matches typical 1.1.1 ordering
Groups = X25519:P-256:P-384:P-521

# --- Signature Algorithms ---
# Commonly advertised by OpenSSL 1.1.1
SignatureAlgorithms = RSA+SHA256:RSA-PSS+SHA256:\
ECDSA+SHA256:ECDSA+SHA384:RSA+SHA384:RSA-PSS+SHA384

-------------------------------------------------------

I created a file called openssl.conf with that text.

Move openssl.conf to e2guardian config folder (/etc/e2guardian)

Update the e2guardian.conf to uncomment and set the following options

useopensslconf = on

opensslconffile = '/etc/e2guardian/openssl.conf'

Then restart e2guardian

https://cloudflare.manfredi.io/test/ then showed 98% human, and all the different types of problems with websites listed above (except maybe quora) worked.

I then asked AI to say whether there were any security risks with this openssl.conf file.  It said no.