CF engine is getting aborted with core dump

[Ashish Singh]

Hellow everyone,

Cf engine server stopped working and getting crashed when agent is bootstraping it ....earlier it was working frok more than 1 year suddenly it stops working and getting aborted with core dump .

CFengine server version : CFEngine Core 3.6.5

CF agent version : CFEngine Core 3.6.6

Please help me as unable to diagnose the issue if you have seen this issue 

when i run thecf engine in versbose mode to  diagnose  

command  cf-serverd- v 

below  are the last few lines when i run tehc command

 

2022-11-11T13:20:42+0000   notice: Server is starting...
2022-11-11T13:21:20+0000  verbose: Obtained IP address of '10.157.18.5' on socket 7 from accept
2022-11-11T13:21:20+0000  verbose: New connection (from 10.157.18.5, sd 7), spawning new thread...
2022-11-11T13:21:20+0000     info: 10.157.18.5> Accepting connection
2022-11-11T13:21:20+0000  verbose: 10.157.18.5> Setting socket timeout to 600 seconds.
2022-11-11T13:21:20+0000  verbose: 10.157.18.5> Peeked CAUTH in TCP stream, considering the protocol as Classic
2022-11-11T13:21:21+0000  verbose: 10.157.18.5> Peer's identity is: MD5=62811fa38d1928e96c245a075d75d514
cf-serverd: mdb.c:1886: mdb_page_touch: Assertion `mp->mp_p.p_pgno != pgno' failed.
Aborted (core dumped)

[Nick Anderson]

Ashish Singh <frozen...@gmail.com> writes:

Hellow everyone,

Cf engine server stopped working and getting crashed when agent is bootstraping it ….earlier it was working frok more than 1 year suddenly it stops working and getting aborted with core dump .

Hi Ashish,

First, are you aware that the version of CFEngine you are running is quite old? The 3.6 series went out of support back on July 17th 2016. The current latest supported version is 3.18.2, released June 29th 2022. 3.15.7 (the last in the 3.15 series), 3.18.3 and 3.21.0 (the first in the next LTS series) should be released before the end of the year. What are the impediments that hold back your upgrades?

To me it looks like you have some corruption in one of the embedded databases, probably cf_lastseen.lmdb since it crashes after some peer related logging. Newer versions of cfengine have a utility cf-check which can help look for an remediate lmdb database issues.Try stopping the CFEngine services and moving cf_lastseen.lmdb out of the way and then starting the services back up again.

service stop cfengine3
for each in $(ls /var/cfengine/state/cf_lastseen.lmdb*); do mv $each{,$(date +-%F-%T.bak)}; done
service start cfengine3

– Nick Anderson | Doer of Things | (+1) 785-550-1767 | https://northern.tech

[Ashish Singh]

Hi Nick ,

Thanks for your response ! 

Yes we are aware of it as its a legacy setup where we are using CFengine engine and it is very old setup which was migrated to our cloud infrastructure and version were kept the same to make the application running as we faced issue with bootstrap and it was working fine with this version but we will definitely explore that option again but right now we need to fix this setup to make our agent functional as it is not working .

Regarding your recommendation :

service stop cfengine3
for each in $(ls /var/cfengine/state/cf_lastseen.lmdb*); do mv $each{,$(date +-%F-%T.bak)}; done
service start cfengine3

i don't see any file with the name cf_lasteen.lmdb . Could you please look at the screenshot once and help us troubleshoot the issue  

[Nick Anderson]

Ashish Singh <frozen...@gmail.com> writes:

Hi Nick ,

Thanks for your response !

Yes we are aware of it as its a legacy setup where we are using CFengine engine and it is very old setup which was migrated to our cloud infrastructure and version were kept the same

Yeah, it happens. But, there is a lot of great stuff in CFEngine in the last 8 years (Augments and Custom Promise Types to name a couple) and you should definitely upgrade.

Regarding your recommendation : i don't see any file with the name cf_lasteen.lmdb . Could you please look at the screenshot once and help us troubleshoot the issue

Oh, I guess cf_lastseen.lmdb isn't in state. It's in WORKDIR, at least in 3.6.x.

[root@hub cfengine]#  find /var/cfengine/ -name "cf_lastseen*"
/var/cfengine/cf_lastseen.lmdb-lock
/var/cfengine/cf_lastseen.lmdb.lock
/var/cfengine/cf_lastseen.lmdb

So, perhaps …

service stop cfengine3
for each in $(ls /var/cfengine/cf_lastseen.lmdb*); do mv $each{,$(date +-%F-%T.bak)}; done
service start cfengine3