Bug#743305: clamav-freshclam: Stops downloading updates (daily.cvd) from local mirror

[Simon Hobson]

Package: clamav-freshclam
Version: 0.98.1+dfsg-1+deb7u3
Severity: normal

Dear Maintainer,


I have a private mirror set up so that only one machine needs to download updates. These are then shared via Apache and other machines are configured to use this main server as their sole mirror.
This was working fine with the previous version (0.97), but stopped working after the recent updates to 0.98.1

Sometimes the mirror is not up to date when the slave checks for updates, the mirror is then marked as "bad" in some way, and then the slave will not try and download from it again. From the freshclam log :

> Received signal: wake up
> ClamAV update process started at Tue Apr 1 13:42:23 2014
> main.cvd is up to date (version: 55, sigs: 2424225, f-level: 60, builder: neo)
> Downloading daily.cvd [100%]
> WARNING: Mirror 172.nn.nn.nn is not synchronized.
> Trying again in 5 secs...
> ClamAV update process started at Tue Apr 1 13:42:31 2014
> main.cvd is up to date (version: 55, sigs: 2424225, f-level: 60, builder: neo)
> WARNING: Can't download daily.cvd from virusdb.<redacted>
> Trying again in 5 secs...


A workaround is to remove /var/lib/clamav/mirrirs.dat at which point the slave can then use the mirror again. However, this is only a crude workaround.



-- Package-specific info:
--- configuration ---
Checking configuration files in /etc/clamav

Config file: clamd.conf
-----------------------
LogFile = "/var/log/clamav/clamav.log"
LogFileUnlock disabled
LogFileMaxSize = "4294967295"
LogTime = "yes"
LogClean disabled
LogSyslog disabled
LogFacility = "LOG_LOCAL6"
LogVerbose disabled
LogRotate = "yes"
ExtendedDetectionInfo = "yes"
PidFile = "/var/run/clamav/clamd.pid"
TemporaryDirectory disabled
DatabaseDirectory = "/var/lib/clamav"
OfficialDatabaseOnly disabled
LocalSocket = "/var/run/clamav/clamd.ctl"
LocalSocketGroup = "clamav"
LocalSocketMode = "666"
FixStaleSocket = "yes"
TCPSocket disabled
TCPAddr disabled
MaxConnectionQueueLength = "15"
StreamMaxLength = "10485760"
StreamMinPort = "1024"
StreamMaxPort = "2048"
MaxThreads = "12"
ReadTimeout = "180"
CommandReadTimeout = "5"
SendBufTimeout = "200"
MaxQueue = "100"
IdleTimeout = "30"
ExcludePath disabled
MaxDirectoryRecursion = "15"
FollowDirectorySymlinks disabled
FollowFileSymlinks disabled
CrossFilesystems = "yes"
SelfCheck = "3600"
VirusEvent disabled
ExitOnOOM disabled
AllowAllMatchScan = "yes"
Foreground disabled
Debug disabled
LeaveTemporaryFiles disabled
User = "clamav"
AllowSupplementaryGroups = "yes"
Bytecode = "yes"
BytecodeSecurity = "TrustSigned"
BytecodeTimeout = "60000"
BytecodeUnsigned disabled
BytecodeMode = "Auto"
DetectPUA disabled
ExcludePUA disabled
IncludePUA disabled
AlgorithmicDetection = "yes"
ScanPE = "yes"
ScanELF = "yes"
DetectBrokenExecutables disabled
ScanMail = "yes"
ScanPartialMessages disabled
PhishingSignatures = "yes"
PhishingScanURLs = "yes"
PhishingAlwaysBlockCloak disabled
PhishingAlwaysBlockSSLMismatch disabled
HeuristicScanPrecedence disabled
StructuredDataDetection disabled
StructuredMinCreditCardCount = "3"
StructuredMinSSNCount = "3"
StructuredSSNFormatNormal = "yes"
StructuredSSNFormatStripped disabled
ScanHTML = "yes"
ScanOLE2 = "yes"
OLE2BlockMacros disabled
ScanPDF = "yes"
ScanSWF = "yes"
ScanArchive = "yes"
ArchiveBlockEncrypted disabled
ForceToDisk disabled
MaxScanSize = "104857600"
MaxFileSize = "26214400"
MaxRecursion = "16"
MaxFiles = "10000"
MaxEmbeddedPE = "10485760"
MaxHTMLNormalize = "10485760"
MaxHTMLNoTags = "2097152"
MaxScriptNormalize = "5242880"
MaxZipTypeRcg = "1048576"
ScanOnAccess disabled
OnAccessIncludePath disabled
OnAccessExcludePath disabled
OnAccessExcludeUID disabled
OnAccessMaxFileSize = "5242880"
DevACOnly disabled
DevACDepth disabled
DevPerformance disabled
DevLiblog disabled
DisableCertCheck disabled

Config file: freshclam.conf
---------------------------
LogFileMaxSize = "4294967295"
LogTime disabled
LogSyslog disabled
LogFacility = "LOG_LOCAL6"
LogVerbose disabled
LogRotate disabled
PidFile = "/var/run/clamav/freshclam.pid"
DatabaseDirectory = "/var/lib/clamav/"
Foreground disabled
Debug disabled
AllowSupplementaryGroups disabled
UpdateLogFile = "/var/log/clamav/freshclam.log"
DatabaseOwner = "clamav"
Checks = "48"
DNSDatabaseInfo = "current.cvd.clamav.net"
DatabaseMirror = "virusdb.<redacted>"
PrivateMirror disabled
MaxAttempts = "5"
ScriptedUpdates disabled
TestDatabases = "yes"
CompressLocalDatabase disabled
ExtraDatabase disabled
DatabaseCustomURL disabled
HTTPProxyServer disabled
HTTPProxyPort disabled
HTTPProxyUsername disabled
HTTPProxyPassword disabled
HTTPUserAgent disabled
NotifyClamd = "/etc/clamav/clamd.conf"
OnUpdateExecute disabled
OnErrorExecute disabled
OnOutdatedExecute disabled
LocalIPAddress disabled
ConnectTimeout = "30"
ReceiveTimeout = "30"
SubmitDetectionStats disabled
DetectionStatsCountry disabled
DetectionStatsHostID disabled
SafeBrowsing disabled
Bytecode = "yes"

clamav-milter.conf not found

Software settings
-----------------
Version: 0.98.1
Optional features supported: MEMPOOL IPv6 FRESHCLAM_DNS_FIX AUTOIT_EA06 BZIP2 JIT

Database information
--------------------
Database directory: /var/lib/clamav/
WARNING: freshclam.conf and clamd.conf point to different database directories
daily.cvd: version 18725, sigs: 863475, built on Tue Apr 1 06:22:57 2014
bytecode.cvd: version 236, sigs: 43, built on Wed Feb 5 17:36:14 2014
main.cvd: version 55, sigs: 2424225, built on Tue Sep 17 15:57:28 2013
Total number of signatures: 3287743

Platform information
--------------------
uname: Linux 2.6.32-5-xen-686 #1 SMP Sun Sep 23 13:33:12 UTC 2012 i686
OS: linux-gnu, ARCH: i386, CPU: i486
Full OS version: Debian GNU/Linux 7.4 (wheezy)
zlib version: 1.2.7 (1.2.7), compile flags: 55
Triple: i386-pc-linux-gnu
CPU: amdfam10, Little-endian
platform id: 0x0a114c4c0404070201040702

Build information
-----------------
GNU C: 4.7.2 (4.7.2)
GNU C++: 4.7.2 (4.7.2)
CPPFLAGS: -D_FORTIFY_SOURCE=2
CFLAGS: -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall
CXXFLAGS: -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall
LDFLAGS: -Wl,-z,relro
Configure: 'CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-z,relro' '--build=i486-linux-gnu' '--prefix=/usr' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--disable-clamav' '--with-dbdir=/var/lib/clamav/' '--sysconfdir=/etc/clamav' '--enable-milter' '--disable-clamuko' '--with-gnu-ld' '--enable-dns-fix' '--disable-unrar' '--libdir=/usr/lib' '--with-system-tommath' '--without-included-ltdl' 'build_alias=i486-linux-gnu'
sizeof(void*) = 4
Engine flevel: 76, dconf: 76

--- data dir ---
total 84696
-rw-r--r-- 1 clamav clamav 67484 Feb 5 18:29 bytecode.cvd
drwxr-xr-x 2 clamav clamav 4096 Jul 28 2013 clamav-74bc56fa4f979749a824966e631825bf
-rw-r--r-- 1 clamav clamav 21830089 Apr 1 10:11 daily.cvd
-rw-r--r-- 1 clamav clamav 64720632 Sep 18 2013 main.cvd

-- System Information:
Debian Release: 7.4
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 2.6.32-5-xen-686 (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages clamav-freshclam depends on:
ii clamav-base 0.98.1+dfsg-1+deb7u3
ii debconf [debconf-2.0] 1.5.49
ii libc6 2.17-97
ii libclamav6 0.98.1+dfsg-1+deb7u3
ii logrotate 3.8.1-4
ii lsb-base 4.1+Debian8+deb7u1
ii ucf 3.0025+nmu3
ii zlib1g 1:1.2.7.dfsg-13

clamav-freshclam recommends no packages.

Versions of packages clamav-freshclam suggests:
pn clamav-docs <none>

-- Configuration Files:
/etc/logrotate.d/clamav-freshclam changed [not included]

-- debconf information:
clamav-freshclam/autoupdate_freshclam: daemon
clamav-freshclam/proxy_user:
clamav-freshclam/NotifyClamd: false
clamav-freshclam/local_mirror: db.local.clamav.net
clamav-freshclam/http_proxy:
clamav-freshclam/update_interval: 48
clamav-freshclam/PrivateMirror:
clamav-freshclam/internet_interface:


--
To UNSUBSCRIBE, email to debian-bugs-...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

[Andreas Cadhalpun]

Hi Simon,

thanks for this bug report.

On 01.04.2014 17:45, Simon Hobson wrote:
> I have a private mirror set up so that only one machine needs to download updates. These are then shared via Apache and other machines are configured to use this main server as their sole mirror.
> This was working fine with the previous version (0.97), but stopped working after the recent updates to 0.98.1

In 0.98 a configuration option for a private mirror was added.
Your freshclam.conf has 'PrivateMirror disabled', so I'm a little bit
surprised that it works at all. How did you point freshclam to your
private mirror?

Please try this again with 'PrivateMirror <IP-adress>' and let us know,
if it works.

> Config file: clamd.conf
> -----------------------
[...]
> DatabaseDirectory = "/var/lib/clamav"
[...]
>
> Config file: freshclam.conf
> ---------------------------
[...]
> DatabaseDirectory = "/var/lib/clamav/"
[...]
> PrivateMirror disabled
[...]

>
> Database information
> --------------------
> Database directory: /var/lib/clamav/
> WARNING: freshclam.conf and clamd.conf point to different database directories

If you want to get rid of this warning, just remove the trailing slash
from the DatabaseDirectory in freshclam.conf.

Best regards,
Andreas

[Simon Hobson]

I'll need to leave it for a little while to see that it's all fine, but adding PrivateMirror seems to fix it. It wasn't in the config file - so the disabled value has to be the default that's being reported.
Had a look in the changelog - found it at line 962 so it's no wonder I'd not noticed it's silent arrival.

As to how I made it work, I just shared /var/lib/clamav on my 'master' server with an Apache vhost (I already have Apache running for other stuff), and set "DatabaseMirror virusdb.<mydomain>" in freshclam.conf on the other servers. I don't know where it came from, but I recall (some time ago now) finding something about how to set it up this way. It may not have been this site I was referring to back then, but it's certainly Option 2 I did :
https://github.com/vrtadmin/clamav-faq/blob/master/mirrors/CvdPrivateMirror.md

The same test is referenced here :
http://osdir.com/ml/clamav-users/2012-07/msg00030.html

> This solution is really simple to implement but it's only effective
> if your clients are all on the same local network and bandwidth is
> not an issue for you.

> Configure a local webserver on one of your machines (say machine1.mylan) and
> let freshclam download the *.cvd files fromhttp://database.clamav.net to
> the webserver’s DocumentRoot.

> Add the following line to freshclam.conf on machine1.mylan.

> ScriptedUpdates no
> First the database will be downloaded to the local webserver and then
> the other clients on the network will update their copy of the database
> from it. For this to work you have to change freshclam.conf on each client
> so that it reads.

> DatabaseMirror machine1.mylan
> ScriptedUpdates no



Over on the ClamAV-users mailing list, Shawn Webb suggested that the issue is to do with some new code intended to make downloads more robust :
http://lurker.clamav.net/message/20140401.183417.1930bc2b.en.html
I'm not really in a position to test the patch.

[Andreas Cadhalpun]

Hi Simon,

let's hope using PrivateMirror fixes the problem.
If not, I could send you a deb compiled with the patch Shawn Webb suggested.

Best regards,
Andreas

[Simon Hobson]

Overnight I left Freshclam stopped on the master, so by this morning it was 3 revisions behind.

Using PrivateMirror, one slave correctly recovered once the master was updated.

However, there is a logging/feedback issue with PrivateMirror. The slave was sat there reporting that :
> daily.cvd is up to date (version: 18732 ...
This is correct in as much as the PrivateMirror had version 18732, but incorrect in that the current version at the time was 18735. Nagios correctly picked up the fact that it was 3 revisions behind.

Also, the logging is a bit verbose with warnings (as reported with Logwatch) :
> Last Status:
> Reading CVD header (main.cld): WARNING: main.cld not found on remote server
> Reading CVD header (main.cvd): OK

> main.cvd is up to date (version: 55, sigs: 2424225, f-level: 60, builder: neo)

> Reading CVD header (daily.cld): WARNING: daily.cld not found on remote server
> Reading CVD header (daily.cvd): OK
> daily.cvd is up to date (version: 18732, sigs: 865515, f-level: 63, builder: neo)
> Reading CVD header (bytecode.cld): WARNING: bytecode.cld not found on remote server
> Reading CVD header (bytecode.cvd): OK
> bytecode.cvd is up to date (version: 236, sigs: 43, f-level: 63, builder: dgoddard)
While before it just logged :
> Last Status:

> main.cvd is up to date (version: 55, sigs: 2424225, f-level: 60, builder: neo)

> daily.cvd is up to date (version: 18716, sigs: 860736, f-level: 63, builder: neo)
> bytecode.cvd is up to date (version: 236, sigs: 43, f-level: 63, builder: dgoddard)


Thanks for the offer of a build with the patch. I'm happy to test it if it helps.

[Andreas Cadhalpun]

Hi,

On 03.04.2014 11:38, Simon Hobson wrote:
> Overnight I left Freshclam stopped on the master, so by this morning it was 3 revisions behind.
>
> Using PrivateMirror, one slave correctly recovered once the master was updated.

So it apparently works correct now. :)

> However, there is a logging/feedback issue with PrivateMirror. The slave was sat there reporting that :
>> daily.cvd is up to date (version: 18732 ...
> This is correct in as much as the PrivateMirror had version 18732, but incorrect in that the current version at the time was 18735. Nagios correctly picked up the fact that it was 3 revisions behind.

I think this is a feature not a bug. If you use a PrivateMirror, you can
use completely different versions.

> Also, the logging is a bit verbose with warnings (as reported with Logwatch) :
>> Last Status:
>> Reading CVD header (main.cld): WARNING: main.cld not found on remote server
>> Reading CVD header (main.cvd): OK
>> main.cvd is up to date (version: 55, sigs: 2424225, f-level: 60, builder: neo)
>> Reading CVD header (daily.cld): WARNING: daily.cld not found on remote server
>> Reading CVD header (daily.cvd): OK
>> daily.cvd is up to date (version: 18732, sigs: 865515, f-level: 63, builder: neo)
>> Reading CVD header (bytecode.cld): WARNING: bytecode.cld not found on remote server
>> Reading CVD header (bytecode.cvd): OK
>> bytecode.cvd is up to date (version: 236, sigs: 43, f-level: 63, builder: dgoddard)
> While before it just logged :
>> Last Status:
>> main.cvd is up to date (version: 55, sigs: 2424225, f-level: 60, builder: neo)
>> daily.cvd is up to date (version: 18716, sigs: 860736, f-level: 63, builder: neo)
>> bytecode.cvd is up to date (version: 236, sigs: 43, f-level: 63, builder: dgoddard)

You followed my advice and used 'PrivateMirror <ip-address>'?
Try with 'PrivateMirror <DNS-name>'. From a quick look at the code, this
might remove the additional messages.

> Thanks for the offer of a build with the patch. I'm happy to test it if it helps.

As it works now, I don't think this will be necessary.

Best regards,
Andreas

[Simon Hobson]

>> Also, the logging is a bit verbose with warnings (as reported with Logwatch) :
>>> Last Status:
>>> Reading CVD header (main.cld): WARNING: main.cld not found on remote server

>>> ...

>
> You followed my advice and used 'PrivateMirror <ip-address>'?
> Try with 'PrivateMirror <DNS-name>'. From a quick look at the code, this might remove the additional messages.

That was with it set as a DNS name.
From the Apache logs, it clearly tries to read the header from *.cld first, and only when those aren't found does it try *.cvd.
The older action (just setting "DatabaseMirror myserver.domain") has it just fetch the *.cvd files.