icap scan with f-secure internet gatekeeper not working

[schr...@gmail.com]

Hey friends,

I just tried e2guardian 5.1 on Ubuntu 18.04 with enabled contentscanners/icapscan.conf.
My goal: Scan all content with http and https (mitm) by FSIGK (F-Secure Internet Gatekeeper) ICAP Service. MITM works, ICAP doesnt.

content of contentscanners/icapscan.conf:

---
plugname = 'icapscan'

# ICAP URL
# Use hostname rather than IP address
# Always specify the port
#
icapurl = 'icap://localhost:1344/'

exceptionvirusmimetypelist = '/etc/e2guardian/lists/contentscanners/exceptionvirusmimetypelist'
exceptionvirusextensionlist = '/etc/e2guardian/lists/contentscanners/exceptionvirusextensionlist'
exceptionvirussitelist = '/etc/e2guardian/lists/contentscanners/exceptionvirussitelist'
exceptionvirusurllist = '/etc/e2guardian/lists/contentscanners/exceptionvirusurllist'
---

But FSIGK logfile is only throwing errors while opening websites (http and http it doesn't change aynthing.

netstat -anp | grep 1344
tcp 0 0 127.0.0.1:1344 0.0.0.0:* LISTEN 40532/fsicapd

### /opt/f-secure/fsigk/log/fsicapd/fsicapd.log
2018-08-03 15:21:56 fsicapd/src/server.c:1264[7] [750:1] ICAP header parsing failed at: 802
2018-08-03 15:21:56 fsicapd/src/server.c:1264[7] [751:1] ICAP header parsing failed at: 802
2018-08-03 15:21:56 fsicapd/src/server.c:1264[7] [752:1] ICAP header parsing failed at: 802
2018-08-03 15:21:57 fsicapd/src/server.c:1264[7] [753:1] ICAP header parsing failed at: 802
2018-08-03 15:21:59 fsicapd/src/server.c:1264[7] [754:1] ICAP header parsing failed at: f2
(...)

Has anyone fun to check out where the problem is? FSIGK is free for downloading, its activating a 30day trial while installing. I can provide a how-to install on ubuntu 18.04 if somebody has the time to trie it out?

by the way: enabling debug in e2guardian does create the debug folder, but nothing inside.

debuglevel = 'ICAP,NET'
debuglevelfile = '/var/log/e2guardian/debug' #debug folder exists after restart, but nothing inside.

[schr...@gmail.com]

Additionally, shouldn't be the ICAP URL something like:

icapurl = 'icap://localhost:1344/response'
icapurl = 'icap://localhost:1344/request'

but i cannot define 2 urls :-)`?

[FredB]

I'm interested by taking a look at this, do you have a configured VM or a container (lxc ?) to share with me ?

About debug, this mode is only about server mode now, I will add client soon

--
Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté.

[schr...@gmail.com]

Good morning FredB,

i've only an ansible playbook for Ubuntu 18.04, it's not flexible yet so it isn't for all OS or Containers yet.

Download FSIGK: https://www.f-secure.com/en/web/business_global/downloads/internet-gatekeeper
ICAP Documentation from F-Secure: https://help.f-secure.com/product.html#business/igk/5.40/en/topic_1C3448E80E42465C8525293E38A04E42-5.40-en

tasks.yml: https://pastebin.com/E4RK8PDx

Manual installation:

---------------------
unarchive fsigk-{{ fsigk_version }}-rtm.tar.gz
dpkg --add-architecture i386
apt install libc6:i386 libncurses5:i386 libstdc++6:i386 zlib1g:i386 libglib2.0-0:i386 libc6-dev linux-libc-dev make gcc

cd (fsigk_unarchived_dir)
make
make install
# fsigk is installed in /opt

a) Enable ICAP in Webui (admin/admin) this way:
WebUI Login to enable ICAP localhost:9012 (admin/admin)

or b) Enable ICAP in fsigk.ini
edit /opt/f-secure/fsigk/conf/fsigk.ini

#fsicapd_service=yes

and

[icap]
bind_addr=127.0.0.1
bind_port=1344
max_conns=500
max_scan_size=2147483648
conn_timeout=600
orsp_file_check=yes
orsp_timeout=5000
block_riskware=no
scan_timeout=90
scan_timeout_block=no
enable_email_services=yes
fsasd_libpath=/opt/f-secure/fsigk/databases/commtouchunix
fsasd_sockpath=/opt/f-secure/fsigk/fsasd-socket

to your recommends.
------------------------------

Thats all. Does this help :-)?

[FredB]

Great thanks I will test soon

[FredB]

icapurl = 'icap://localhost:1344/' where do you see this configuration ? I mean in the f-secure documentation what the client configuration must be ?

[FredB]

For the record:

ICAP wireshark (full in attachment):

    X-FSecure-Versions: F-Secure Corporation Hydra/5.19 build 17/2018-08-10_01 F-Secure Corporation Aquarius/1.0 build 8/2018-08-10_08 fsavd/1.0/0148 fsicapd/1.1.277-263d28a\r\n
    Encapsulated: null-body=0\r\n
    \r\n
    ICAP/1.0 400 Bad Request\r\n
    Server: F-Secure ICAP Server\r\n
    ISTag: "FSAV-2018-08-10_08"\r\n
    Connection: close\r\n
    Expires: Sat, 26 Aug 1922 04:47:06 GMT\r\n
    X-FSecure-Transaction-Duration: 0.000076\r\n
    X-FSecure-Versions: F-Secure Corporation Hydra/5.19 build 17/2018-08-10_01 F-Secure Corporation Aquarius/1.0 build 8/2018-08-10_08 fsavd/1.0/0148 fsicapd/1.1.277-263d28a\r\n
    \r\n

fsecure log:

...
2018-08-10 13:41:51 fsicapd/src/server.c:1264[7] [4:1] ICAP header parsing failed at: 44


ICAP Client debug mode:

hw2: 1533908461 ICAPC debug : hw2: About to send icapheader:
RESPMOD icap://127.0.0.1:1344/respmod ICAP/1.0
Host: 127.0.0.1
Allow: 204
Encapsulated: req-hdr=0, res-hdr=56, res-body=75
Preview: 0

GET http://www.eicar.org/download/eicar.com HTTP/1.0

HTTP/1.0 200 OK

0


hw2: 1533908461 ICAPC debug : hw2: Sending memory date to icap preview first

hw2: 1533908461 ICAPC debug : hw2: reply from icap: ICAP/1.0 100 Continue

hw2: 1533908461 ICAPC debug : hw2: ICAP says continue!

hw2: 1533908461 ICAPC debug : hw2: total sent to icap: 68

hw2: 1533908461 ICAPC debug : hw2: memory was sent to icap

hw2: 1533908461 ICAPC debug : hw2: reply from icap: ICAP/1.0 204 No Modification

hw2: 1533908461 ICAPC debug : hw2: ICAP says clean!

hw3: 1533908511 ICAPC debug : hw3: About to send icapheader:
RESPMOD icap://127.0.0.1:1344/respmod ICAP/1.0
Host: 127.0.0.1
Allow: 204
Encapsulated: req-hdr=0, res-hdr=56, res-body=75
Preview: 0

GET http://www.eicar.org/download/eicar.com HTTP/1.0

HTTP/1.0 200 OK

0


hw3: 1533908511 ICAPC debug : hw3: Sending memory date to icap preview first

hw3: 1533908511 ICAPC debug : hw3: reply from icap: ICAP/1.0 100 Continue

hw3: 1533908511 ICAPC debug : hw3: ICAP says continue!

hw3: 1533908511 ICAPC debug : hw3: total sent to icap: 68

E2guardain dgdebug:

0:  -Content scanners interested in response data: 1 Line: 1120 Function: handleConnection
hw0: mime type: application/octet-stream Line: 217 Function: isContentType
hw0: mimes result : false Line: 244 Function: isContentType
hw0: mime type: application/octet-stream Line: 217 Function: isContentType
hw0: mimes result : false Line: 244 Function: isContentType
hw0:  -Filtering with expectation of a possible csmessage Line: 2880 Function: check_content
hw0:
hw0:  -about to get body from proxy
hw0: Got to final download manager so defaulting to always match.
hw0: Inside default download manager plugin  icap=0
hw0: tranencodeing is
hw0: bytes remaining is 68
hw0: blocksize: 32768
hw0: newsize: 68
hw0: Leaving default download manager plugin
hw0:  -got body
hw0:  -Running scanMemory
hw0: About to send icapheader:
RESPMOD icap://127.0.0.1:1344/respmod ICAP/1.0^M
Host: 127.0.0.1^M
Allow: 204^M
Encapsulated: req-hdr=0, res-hdr=56, res-body=75^M
Preview: 0^M
^M
GET http://www.eicar.org/download/eicar.com HTTP/1.0^M
^M
HTTP/1.0 200 OK^M
^M
0^M

hw0: Sending memory date to icap preview first
hw0: reply from icap: ICAP/1.0 100 Continue^M
hw0: ICAP says continue!
hw0: total sent to icap: 68
hw0: memory was sent to icap
hw0: reply from icap: ICAP/1.0 204 No Modification^M
hw0: ICAP says clean!
hw0:  -AV scan 0 returned: 0
hw0:  -finished running AV
hw0: mime type: application/octet-stream Line: 217 Function: isContentType
hw0: mimes result : false Line: 244 Function: isContentType
hw0: mime type: application/octet-stream Line: 217 Function: isContentType
hw0: mimes result : false Line: 244 Function: isContentType
hw0:  -Skipping content filtering: hw0: mime type: application/octet-stream Line: 217 Function: isContentType
hw0: mimes result : false Line: 244 Function: isContentType
hw0:  -Not texthw0:
hw0: mime type: application/octet-stream Line: 217 Function: isContentType
hw0: mimes result : false Line: 244 Function: isContentType

[FredB]

> icapurl = 'icap://localhost:1344/response'

Only response mode, response from Internet
>
>

[FredB]

With Kaspersky, the file is always good (even eicar ...)

Kaspersky logs:

[11-08-2018 11:17:07 E] PROCESS pid = 30600 SID=w7BBH70000 Call isContinue when 100 continue is not send
[11-08-2018 11:17:52 E] PROCESS pid = 30599 SID=w7BBHqx000 Call isContinue when 100 continue is not send
[11-08-2018 11:18:23 E] PROCESS pid = 30599 SID=w7BBINx000 Call isContinue when 100 continue is not send

E2guardian debug:

ICAP server is 127.0.0.1
ICAP/1.0 OPTIONS response: ICAP/1.0 200 OK
ICAP/1.0 OPTIONS response part: ISTag: "KAVPROXY"
ICAP/1.0 OPTIONS response part: Date: Sat, 11 Aug 2018 09:18:20 GMT
ICAP/1.0 OPTIONS response part: Methods: RESPMOD
ICAP/1.0 OPTIONS response part: Allow: 204
ICAP/1.0 OPTIONS response part: Service: KAV-ICAP-Sever/5.5
ICAP/1.0 OPTIONS response part: Preview: 0
ICAP/1.0 OPTIONS response part: Max-Connections: 5000
ICAP/1.0 OPTIONS response part: Service-ID: KAVIcap
ICAP/1.0 OPTIONS response part: X-Include: X-Client-IP
ICAP/1.0 OPTIONS response part: Transfer-Preview: *
ICAP/1.0 OPTIONS response part: Transfer-Ignore:
ICAP/1.0 OPTIONS response part: Options-TTL: 300
ICAP/1.0 OPTIONS response part: Encapsulated: null-body=0
ICAP/1.0 OPTIONS response part:
Message previews enabled; size: 0
response is TRANSLATION KEY 59 MISSING

hw0: About to send icapheader:

RESPMOD icap://127.0.0.1:1025/av/respmod ICAP/1.0

Host: 127.0.0.1
Allow: 204
Encapsulated: req-hdr=0, res-hdr=56, res-body=75
Preview: 0

GET http://www.eicar.org/download/eicar.com HTTP/1.0

HTTP/1.0 200 OK

0

hw0: Sending memory date to icap preview first
hw0: reply from icap: ICAP/1.0 100 Continue

hw0: ICAP says continue!
hw0: reply from icap: ICAP/1.0 200 OK
hw0: ICAP says maybe not clean!
hw0: Comparing original return code to modified:HTTP/1.1 200 OK
HTTP/1.0 200 OK
hw0: Comparing original body data to modified
hw0: ICAP says clean! (body byte comparison)

In this case comparing original body data to modified seems dangerous and ineffective

https://github.com/e2guardian/e2guardian/blob/v5.1-icapc/src/contentscanners/icapscan.cpp#L750

Fred

[FredB]

David can you try with this branch ? https://github.com/e2guardian/e2guardian/tree/v5.1-icapc

If you are not familiar with compilation and/or github I can provide a binary if needed

[schr...@gmail.com]

Hello Fred, I am just compiling now v5.1-icap, did autoremove+purge the v5.1 on the VM before. (but saved the configs for later)

Sorry I was not able to write earlier here. Will try it out and update you as soon as possible again.

[FredB]

You can also let your configuration and just switch the binary.

The right syntax is 127.0.0.1:1344/respond

Le 13 août 2018 08:48:56 GMT+02:00, schr...@gmail.com a écrit :

Hello Fred, I am just compiling now v5.1-icap, did autoremove+purge the v5.1 on the VM before. (but saved the configs for later)

Sorry I was not able to write earlier here. Will try it out and update you as soon as possible again.

[schr...@gmail.com]

Nice! It definitively works with this branch, very cool!

Before I saw your latest answer, I was using icapurl = 'icap://localhost:1344/ and eicar + some real worms i got by email (.zip) are definitively detected correctly.
I changed the line to icapurl = 'icap://localhost:1344/respond', restarted, its still detecting all the viruses by f-secure icap.

just one cosmetic thing: When trying to download an infected .zip via download button in webmail, its just blocking and logging the request, not throwing an error page. I guess the attachemnt-download-button in my webmail Rainloop is the reason, its doing force download in background or so. beside, the eicar test is giving me the usual error page correctly. "Virus or bad content detected. Unknown. Categories: Content scanning".

Not ICAP related: I have one last issue which is blocking a production use for me, often when I type the URL without www again in a 2nd/3rd/4th tab, its giving me randomly different error pages or messages. Again, this is not ICAP related, it was there also before since 5.0 without SQUID as upstream Best reproducible with 20min.ch, just open that site 10x and 6-8times it wont open. (its one of the most visited swiss website)
See screenshots: https://imgur.com/a/XvDYJzk shall I open a bug report in github for that?

[FredB]

>
> just one cosmetic thing: When trying to download an infected .zip via download button in webmail, its just blocking and logging the request, not throwing an error page. I guess the attachemnt-download-button in my webmail Rainloop is the reason, its doing force download in background or so. beside, the eicar test is giving me the usual error page correctly. "Virus or bad content detected. Unknown. Categories: Content scanning".

Yes there is nothing we can do here

>
> Not ICAP related: I have one last issue which is blocking a production use for me, often when I type the URL without www again in a 2nd/3rd/4th tab, its giving me randomly different error pages or messages. Again, this is not ICAP related, it was there also before since 5.0 without SQUID as upstream Best reproducible with 20min.ch, just open that site 10x and 6-8times it wont open. (its one of the most visited swiss website)
> See screenshots: https://imgur.com/a/XvDYJzk shall I open a bug report in github for that?
>

Please can you open a ticket on github with a wireshark trace ?

Fred

[denis...@gmail.com]

Continuing the topic, I have a question: how I can make that ALL traffic follows via icap server on another machine?

[FredB]

What do you want to do exactly?

Le 28 août 2018 05:49:05 GMT+01:00, denis...@gmail.com a écrit :
>Continuing the topic, I have a question: how I can make that ALL
>traffic follows via icap server on another machine?
>

>--
>E2guardian:
>https://groups.google.com/d/forum/e2guardian
>Github:
>https://github.com/e2guardian/e2guardian
>Follow us on twitter:
>https://twitter.com/e2guardian
>---
>You received this message because you are subscribed to the Google
>Groups "e2guardian" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to e2guardian+...@googlegroups.com.
>To post to this group, send an email to e2gua...@googlegroups.com.
>Visit this group at https://groups.google.com/group/e2guardian.
>To view this discussion on the web, visit
>https://groups.google.com/d/msgid/e2guardian/3e960dd1-4904-40fc-b444-9e4764666ec8%40googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

[Денис Степанов]

I want process all HTTP/HTTPS requests from users of my company. It will be like small DLP system. I don't want to look any content, at this moment - only search requests, mail message and so on.

среда, 29 августа 2018 г., 17:19:30 UTC+7 пользователь FredB написал: