Timeout While Copying File

[Shane McEwan]

G'day!

Since upgrading to CFEngine 3.5.2 I occasionally get an error from
various hosts about a timeout while copying a file:

2014-01-16T23:01:36+0000 error: Timeout - remote end did not respond
with the expected amount of data (received=67, expecting=2048). (recv:
Resource temporarily unavailable)
2014-01-16T23:01:36+0000 error: Error in client-server stream (has
10.20.30.40:/var/cfengine/masterfiles/services/dev/zabbix-server/sbin/zabbix_server
shrunk?)
2014-01-16T23:01:36+0000 error:
/cfe_internal_update/files/'$(master_location)': Was not able to copy
'/var/cfengine/masterfiles/services/dev/zabbix-server/sbin/zabbix_server' to
'/var/cfengine/masterfiles/services/dev/zabbix-server/sbin/zabbix_server'

It happens on several different files and hosts once or twice per day,
the only common denominator is that the files are all greater than 1MB.
It doesn't happen all the time and I've never been able to make it
happen while running cf-agent manually. It's also not related to
legitimate transfers of files when I make changes to masterfiles . . .
these errors can happen in the middle of the night when nothing is changing.

Also, these files that are mentioned in the errors have not been changed
for months so why is CFEngine even trying to transfer them? I suspect
the error messages are spurious and being triggered by some other
problem because the destination files are never overwritten, all that
happens is I end up with an empty .cfnew file in the destination directory.

Ignoring the fact that I should be installing some of these files as
software packages rather than distributing them via CFEngine, does
anyone have any ideas what could be going on here?

Thanks!

Shane.

[Tony Lill]

There's another thread about if_repaired being triggered when it
shouldn't be, this is also on a file that is being copied when it
shouldn't. You're possibly seeing the same bug, at least with regards to
why files are being copied.

I see the network error occasionally. I've also had cf-agent actually
replace good files with zero-length copies and other nasty things, but
that was only when I was having real network issues and getting a lot of
these errors. Try checking if there is unusual network traffic or errors
at the time, or unusual load on the policy server.

[Shane McEwan]

Thanks, Tony

On 17/01/14 23:05, Tony Lill wrote:
> There's another thread about if_repaired being triggered when it
> shouldn't be, this is also on a file that is being copied when it
> shouldn't. You're possibly seeing the same bug, at least with regards to
> why files are being copied.

I don't have an if_repaired on these files so it's possible a repair is
happening and my problem is related. I'll try putting an if_repaired on
it and see what happens.

> I see the network error occasionally. I've also had cf-agent actually
> replace good files with zero-length copies and other nasty things, but
> that was only when I was having real network issues and getting a lot of
> these errors. Try checking if there is unusual network traffic or errors
> at the time, or unusual load on the policy server.

Thankfully I haven't had the empty files replace the real files. That
would be annoying! I'm not aware of any network problems however the
policy server's disks are quite slow and overloaded so that could be
compounding the problem.

Thanks again!

Shane.

[Shane McEwan]

On 17/01/14 10:48, Shane McEwan wrote:
> G'day!
>
> Since upgrading to CFEngine 3.5.2 I occasionally get an error from
> various hosts about a timeout while copying a file

I've done a bit more digging and I think I'm encountering bug #3467
(https://cfengine.com/dev/issues/3467) where the cf_promises_validated
on the policy hub is updated every time cf-agent runs even if the policy
files haven't changed. This is triggering all the clients to copy the
masterfiles every run.

It doesn't explain why I'm getting timeouts, nor why it's trying to copy
a file that hasn't changed even though I'm doing a digest copy but at
least I know why the copy is being initiated.

Time to upgrade to 3.5.3 and see if that fixes the problem.

Thanks!

Shane.

[Neil Watson]

On Tue, Jan 21, 2014 at 04:57:11PM +0000, Shane McEwan wrote:
>a file that hasn't changed even though I'm doing a digest copy but at
>least I know why the copy is being initiated.
>
>Time to upgrade to 3.5.3 and see if that fixes the problem.

Be very careful with 3.5.3 is has a serious bug that breaks copies that
involve trailing '/' like recursive copies. Access promises with
trailing '/' also seem to break. After testing 3.5.3 I returned to
3.5.2.

--
Neil H Watson
http://evolvethinking.com/evolve-thinkings-free-cfengine-library/
Hardening with Cfengine http://evolvethinking.com/products/
VIM and Cfengine https://github.com/neilhwatson/vim_cf3

[Shane McEwan]

On 21/01/14 17:14, Neil Watson wrote:
> On Tue, Jan 21, 2014 at 04:57:11PM +0000, Shane McEwan wrote:
>> a file that hasn't changed even though I'm doing a digest copy but at
>> least I know why the copy is being initiated.
>>
>> Time to upgrade to 3.5.3 and see if that fixes the problem.
>
> Be very careful with 3.5.3 is has a serious bug that breaks copies that
> involve trailing '/' like recursive copies. Access promises with
> trailing '/' also seem to break. After testing 3.5.3 I returned to
> 3.5.2.

I've been skimming that thread so thanks for the reminder because I
didn't make the connection!

I think all of my directory promises are free of a trailing slash or
have '/.' if I want to affect the directory directly but I'll double check.

Shane.