Bug#968365: /usr/bin/freshclam: Ignores DatabaseCustomURL unless --update-db=custom is specified

[Stephan Jänecke]

Package: clamav-freshclam
Version: 0.102.3+dfsg-0~deb9u1
Severity: normal
File: /usr/bin/freshclam

Dear Maintainer,

starting with version 0.102.3 freshclam ignores DatebaseCustomURL
options.

As this option is used to specify custom paths to update databases on
our local mirror, updates fail after upgrading from version 0.101.4.

Let me give you an example. Instead of downloading the database from
`http://update.dfn-cert.de/av-sigs/clamav/db/main.cvd` freshclam will
try `https://update.dfn-cert.de/main.cvd` and fail.

Looking at the code I figured out that freshclam can be motivated to honour
the values in DatabaseCustomURL options. Once I specified the executable
parameter `--update-db=custom` freshclam happily updated the databases
from the custom paths.

Now I'm wondering: is my site incorrectly specifying custom database
paths using `DatabaseCustomURL` and the breakage on update has been
intentionally introduced by upstream? If so, what would be the correct
way to introduce custom paths?

Or did I indeed find a bug?

Best regards,

Stephan Jänecke


-- Package-specific info:
--- configuration ---
Checking configuration files in /etc/clamav

Config file: clamd.conf
-----------------------
AlertExceedsMax disabled
PreludeEnable disabled
PreludeAnalyzerName disabled
LogFile = "/var/log/clamav/clamav.log"
LogFileUnlock disabled
LogFileMaxSize = "4294967295"
LogTime = "yes"
LogClean disabled
LogSyslog = "yes"
LogFacility = "LOG_LOCAL6"
LogVerbose disabled
LogRotate = "yes"
ExtendedDetectionInfo = "yes"
PidFile = "/var/run/clamav/clamd.pid"
TemporaryDirectory = "/var/tmp"
DatabaseDirectory = "/var/lib/clamav"
OfficialDatabaseOnly disabled
LocalSocket = "/var/run/clamav/clamd.ctl"
LocalSocketGroup = "clamav"
LocalSocketMode disabled
FixStaleSocket = "yes"
TCPSocket disabled
TCPAddr disabled
MaxConnectionQueueLength = "200"
StreamMaxLength = "1073741824"
StreamMinPort = "1024"
StreamMaxPort = "2048"
MaxThreads = "100"
ReadTimeout = "120"
CommandReadTimeout = "30"
SendBufTimeout = "500"
MaxQueue = "200"
IdleTimeout = "30"
ExcludePath disabled
MaxDirectoryRecursion = "15"
FollowDirectorySymlinks disabled
FollowFileSymlinks disabled
CrossFilesystems = "yes"
SelfCheck = "3600"
DisableCache disabled
VirusEvent disabled
ExitOnOOM disabled
AllowAllMatchScan = "yes"
Foreground disabled
Debug disabled
LeaveTemporaryFiles disabled
User = "clamav"
Bytecode = "yes"
BytecodeSecurity = "TrustSigned"
BytecodeTimeout = "5000"
BytecodeUnsigned disabled
BytecodeMode = "Auto"
DetectPUA disabled
ExcludePUA disabled
IncludePUA disabled
ScanPE = "yes"
ScanELF = "yes"
ScanMail = "yes"
ScanPartialMessages disabled
PhishingSignatures = "yes"
PhishingScanURLs = "yes"
HeuristicAlerts = "yes"
HeuristicScanPrecedence disabled
StructuredDataDetection disabled
StructuredMinCreditCardCount = "3"
StructuredMinSSNCount = "3"
StructuredSSNFormatNormal = "yes"
StructuredSSNFormatStripped disabled
ScanHTML = "yes"
ScanOLE2 = "yes"
AlertBrokenExecutables disabled
AlertEncrypted disabled
AlertEncryptedArchive disabled
AlertEncryptedDoc disabled
AlertOLE2Macros disabled
AlertPhishingSSLMismatch disabled
AlertPhishingCloak disabled
AlertPartitionIntersection disabled
ScanPDF = "yes"
ScanSWF = "yes"
ScanXMLDOCS = "yes"
ScanHWP3 = "yes"
ScanArchive = "yes"
ForceToDisk disabled
MaxScanTime disabled
MaxScanSize = "10737418240"
MaxFileSize = "10737418240"
MaxRecursion = "16"
MaxFiles disabled
MaxEmbeddedPE = "10485760"
MaxHTMLNormalize = "10485760"
MaxHTMLNoTags = "2097152"
MaxScriptNormalize = "5242880"
MaxZipTypeRcg = "1048576"
MaxPartitions = "50"
MaxIconsPE = "100"
MaxRecHWP3 = "16"
PCREMatchLimit = "100000"
PCRERecMatchLimit = "2000"
PCREMaxFileSize = "26214400"
OnAccessMountPath disabled
OnAccessIncludePath disabled
OnAccessExcludePath disabled
OnAccessExcludeRootUID disabled
OnAccessExcludeUID disabled
OnAccessExcludeUname disabled
OnAccessMaxFileSize = "5242880"
OnAccessDisableDDD disabled
OnAccessPrevention disabled
OnAccessExtraScanning disabled
OnAccessCurlTimeout = "5000"
OnAccessMaxThreads = "5"
OnAccessRetryAttempts disabled
OnAccessDenyOnError disabled
DevACOnly disabled
DevACDepth disabled
DevPerformance disabled
DevLiblog disabled
DisableCertCheck disabled
AlgorithmicDetection = "yes"
BlockMax disabled
PhishingAlwaysBlockSSLMismatch disabled
PhishingAlwaysBlockCloak disabled
PartitionIntersection disabled
OLE2BlockMacros disabled
ArchiveBlockEncrypted disabled

Config file: freshclam.conf
---------------------------
LogFileMaxSize = "10485760"
LogTime = "yes"
LogSyslog = "yes"
LogFacility = "LOG_LOCAL6"
LogVerbose disabled
LogRotate = "yes"
PidFile = "/var/run/clamav/freshclam.pid"
DatabaseDirectory = "/var/lib/clamav"
Foreground disabled
Debug disabled
UpdateLogFile = "/var/log/clamav/freshclam.log"
DatabaseOwner = "clamav"
Checks = "24"
DNSDatabaseInfo = "current.cvd.clamav.net"
DatabaseMirror = "update.dfn-cert.de"
PrivateMirror disabled
MaxAttempts = "3"
ScriptedUpdates disabled
TestDatabases = "yes"
CompressLocalDatabase disabled
ExtraDatabase disabled
ExcludeDatabase disabled
DatabaseCustomURL = "http://update.dfn-cert.de/av-sigs/clamav/db/main.cvd", "http://update.dfn-cert.de/av-sigs/clamav/db/daily.cvd", "http://update.dfn-cert.de/av-sigs/clamav/db/bytecode.cvd"
HTTPProxyServer disabled
HTTPProxyPort disabled
HTTPProxyUsername disabled
HTTPProxyPassword disabled
HTTPUserAgent disabled
NotifyClamd = "/etc/clamav/clamd.conf"
OnUpdateExecute disabled
OnErrorExecute disabled
OnOutdatedExecute disabled
LocalIPAddress disabled
ConnectTimeout = "30"
ReceiveTimeout disabled
SafeBrowsing disabled
Bytecode = "yes"

clamav-milter.conf not found

Software settings
-----------------
Version: 0.102.3
Optional features supported: MEMPOOL IPv6 FRESHCLAM_DNS_FIX AUTOIT_EA06 BZIP2 LIBXML2 PCRE ICONV JSON JIT

Database information
--------------------
Database directory: /var/lib/clamav
daily.cvd: version 25901, sigs: 3835550, built on Thu Aug 13 09:01:24 2020
bytecode.cvd: version 331, sigs: 94, built on Thu Sep 19 18:12:33 2019
main.cvd: version 59, sigs: 4564902, built on Mon Nov 25 14:56:15 2019
Total number of signatures: 8400546

Platform information
--------------------
uname: Linux 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1+deb9u1 (2020-06-07) x86_64
OS: linux-gnu, ARCH: x86_64, CPU: x86_64
Full OS version: Debian GNU/Linux 9.12 (stretch)
zlib version: 1.2.8 (1.2.8), compile flags: a9
Triple: x86_64-pc-linux-gnu
CPU: nocona, Little-endian
platform id: 0x0a2172720806030001060300

Build information
-----------------
GNU C: 6.3.0 20170516 (6.3.0)
GNU C++: 6.3.0 20170516 (6.3.0)
CPPFLAGS: -Wdate-time -D_FORTIFY_SOURCE=2
CFLAGS: -g -O2 -fdebug-prefix-map=/build/clamav-Jw2Blr/clamav-0.102.3+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64 -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
CXXFLAGS: -g -O2 -fdebug-prefix-map=/build/clamav-Jw2Blr/clamav-0.102.3+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64
LDFLAGS: -Wl,-z,relro -Wl,-z,now -Wl,--as-needed
Configure: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-silent-rules' '--libdir=/usr/lib/x86_64-linux-gnu' '--libexecdir=/usr/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' 'CFLAGS=-g -O2 -fdebug-prefix-map=/build/clamav-Jw2Blr/clamav-0.102.3+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fdebug-prefix-map=/build/clamav-Jw2Blr/clamav-0.102.3+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -D_FILE_OFFSET_BITS=64' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now -Wl,--as-needed' '--with-dbdir=/var/lib/clamav' '--sysconfdir=/etc/clamav' '--disable-clamav' '--disable-unrar' '--enable-milter' '--enable-dns-fix' '--with-libjson' '--with-system-libmspack' '--with-libcurl=/usr' '--with-gnu-ld' '-with-system-llvm=/usr/bin/llvm-config' '--with-llvm-linking=dynamic' '--with-systemdsystemunitdir=/lib/systemd/system' 'build_alias=x86_64-linux-gnu' 'OBJCFLAGS=-g -O2 -fdebug-prefix-map=/build/clamav-Jw2Blr/clamav-0.102.3+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security'
sizeof(void*) = 8
Engine flevel: 114, dconf: 114

--- data dir ---
total 213272
-rw-r--r-- 1 clamav clamav 296388 May 26 09:56 bytecode.cvd
-rw-r--r-- 1 clamav clamav 100220586 Aug 13 15:01 daily.cvd
-rw-r--r-- 1 clamav clamav 117859675 May 26 09:53 main.cvd

-- System Information:
Debian Release: 9.12
APT prefers oldstable-updates
APT policy: (500, 'oldstable-updates'), (500, 'oldstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-12-amd64 (SMP w/1 CPU core)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968), LANGUAGE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages clamav-freshclam depends on:
ii clamav-base 0.102.3+dfsg-0~deb9u1
ii debconf [debconf-2.0] 1.5.61
ii dpkg 1.18.25
ii init-system-helpers 1.48
ii libc6 2.24-11+deb9u4
ii libclamav9 0.102.3+dfsg-0~deb9u1
ii logrotate 3.11.0-0.1
ii lsb-base 9.20161125
ii procps 2:3.3.12-3+deb9u1
ii ucf 3.0036

Versions of packages clamav-freshclam recommends:
ii ca-certificates 20200601~deb9u1

Versions of packages clamav-freshclam suggests:
pn apparmor <none>
pn clamav-docs <none>

-- debconf information:
clamav-freshclam/PrivateMirror:
clamav-freshclam/NotifyClamd: true
clamav-freshclam/update_interval: 24
* clamav-freshclam/autoupdate_freshclam: daemon
clamav-freshclam/LogRotate: True
clamav-freshclam/proxy_user:
clamav-freshclam/Bytecode: yes
clamav-freshclam/internet_interface:
clamav-freshclam/http_proxy:
clamav-freshclam/SafeBrowsing: false
clamav-freshclam/local_mirror: update.dfn-cert.de

--
Stephan Jänecke (PKI-Team + IT-Services)
Mail: jaen...@dfn-cert.de Phone: +49 40 808077-709

DFN-CERT Services GmbH, https://www.dfn-cert.de/, Phone +49 40 808077-555
Sitz / Register: Hamburg, AG Hamburg, HRB 88805, Ust-IdNr.: DE 232129737
Nagelsweg 41, 20097 Hamburg, Germany. CEO: Dr. Klaus-Peter Kossakowski

[Sebastian Andrzej Siewior]

On 2020-08-13 17:21:04 [+0200], Stephan Jänecke wrote:
> starting with version 0.102.3 freshclam ignores DatebaseCustomURL
> options.

Are you sure about the versions? I've been looking at the changes
0.102.2..0.102.3 and there is nothing the freshclam area. That option
appears to be there. It might have happen earlier when they switched to
libcurl instead of doing their own http…

> As this option is used to specify custom paths to update databases on
> our local mirror, updates fail after upgrading from version 0.101.4.

okay. Then it is a 0.101 -> 0.102 kind of thing.

> Let me give you an example. Instead of downloading the database from
> `http://update.dfn-cert.de/av-sigs/clamav/db/main.cvd` freshclam will
> try `https://update.dfn-cert.de/main.cvd` and fail.
…
> Looking at the code I figured out that freshclam can be motivated to honour
> the values in DatabaseCustomURL options. Once I specified the executable
> parameter `--update-db=custom` freshclam happily updated the databases
> from the custom paths.
…
> Now I'm wondering: is my site incorrectly specifying custom database
> paths using `DatabaseCustomURL` and the breakage on update has been
> intentionally introduced by upstream? If so, what would be the correct
> way to introduce custom paths?

It sounds like you want to use PrivateMirror. But then you don't have
same path so it probably won't work. I don't know.

You have
DatabaseMirror = "update.dfn-cert.de"

set which means it will look for

update.dfn-cert.de/main.cvd
update.dfn-cert.de/daily.cvd

so it works as expected so far. You have also set DatabaseCustomURL so
it should look additionally for

update.dfn-cert.de/av-sigs/clamav/db/main.cvd

Now the fact that it does not might have something to do with the part
that it fails while looking for update.dfn-cert.de/main.cvd. But from
looking at the code (in this heat) it should do so.

By using "--update-db=custom" then you limit the download to the custom
URL only which is what you specify with DatabaseCustomURL. This skips
the download of main/daily/bytecode.

> Or did I indeed find a bug?

Maybe something that was not planned. It might be possible that your
custom URL contained the 'main.cvd' file which then overwrote the
'main.cvd' from the official mirror. Now it does it no longer and would
in your case download the main.cvd twice.
If you could verify the part with PrivateMirror then I/you could open a
bug with upstream asking what the recommended way is to use a private
mirror with a different hierarchy. This is what you intend to do I
guess.

> Best regards,
>
> Stephan Jänecke

Sebastian

[Stephan Jänecke]

Hi Sebastian,

Sebastian wrote:
> It sounds like you want to use PrivateMirror. But then you don't have
> same path so it probably won't work. I don't know.
>
> You have
> DatabaseMirror = "update.dfn-cert.de"

so I replaced DatabaseMirror by PrivateMirror which results in:

```
-> ^remote_cvdhead: file not found: http://update.dfn-cert.de/daily.cvd
```

When specifiying the update-db parameter the paths specified in
DatabaseCustomURL are used.

> If you could verify the part with PrivateMirror then I/you could open a
> bug with upstream asking what the recommended way is to use a private
> mirror with a different hierarchy.

In case there is nothing further to add, I would like to file an
upstream bug report. I'll document relevant changes here.

> This is what you intend to do I guess.

Indeed, I would like to place the databases somewhere other than
document root.

Cheers,

Stephan

[Sebastian Andrzej Siewior]

On 2020-08-20 17:25:01 [+0200], Stephan Jänecke wrote:
> Hi Sebastian,
Hi Stephan,

> In case there is nothing further to add, I would like to file an
> upstream bug report. I'll document relevant changes here.

Okay. It is https://bugzilla.clamav.net/.

Sebastian

[Sebastian Andrzej Siewior]

On 2020-08-20 20:16:50 [+0200], To Stephan Jänecke wrote:
Hi Stephan,
>
> > In case there is nothing further to add, I would like to file an
> > upstream bug report. I'll document relevant changes here.
>
> Okay. It is https://bugzilla.clamav.net/.

Where do we stand here?

Sebastian

[Stephan Jänecke]

Good question. I filed a report two months ago[1], which hasn't been
touched to this date. Its visibility is currently limited to the
security group, so you might have issues viewing the report.

Prodding from your side might help. In the meantime I'll ask around on
clamav-users.

Stephan

[1] https://bugzilla.clamav.net/show_bug.cgi?id=12607

[Sebastian Andrzej Siewior]

forwarded -1 https://bugzilla.clamav.net/show_bug.cgi?id=12607

On 2020-11-02 11:38:10 [+0100], Stephan Jänecke wrote:
> Good question. I filed a report two months ago[1], which hasn't been
> touched to this date. Its visibility is currently limited to the
> security group, so you might have issues viewing the report.
>
> Prodding from your side might help. In the meantime I'll ask around on
> clamav-users.

Did something pop up on clamav-users in the meantime?

> Stephan

Sebastian