ManifoldCF 2.6 files connector memory leak

[wilhel...@gmail.com]

Hi,

since the upgrade to datafari 3.2.1 I notice that the manifoldcf process constantly adds up memory usage during rescans of my files (not during the scan of websites, though!). Even though I have quite an amount of files (~500.000), I never ran into this issue with datafari 3.1 aka ManifoldCF 2.5. With 2.6 I see the process eating up to 90% of the memory (24GB) plus using swap up to some point (also 24GB). It then might drop out with an exception and die.

I never experienced this with datafari 3.1 and I did manual scans every night per cronjob. Does anybody have the same problem?

Googling does not really help. I have reduced the amount of workers in manifoldcf and progresql to half of the usual size and that helps a bit, but not really. I found this by google:

https://serverfault.com/questions/707027/manifoldcf-keeps-running-out-of-memory

Anyway, any help is appreciated. If there is no solution, I might wait for datafari 4.0 and manifoldcf 2.8.1

Thanks,

Wilhelm

[Julien Massiera]

Hi Wilhelm

your problem seems not related to Datafari itself but ManifoldCF. So you can either wait for Datafari 4.0.0 or contact the Apache ManifoldCF user mailing list to explain your issue.

Regards,
Julien

--
You received this message because you are subscribed to the Google Groups "Datafari" group.
To unsubscribe from this group and stop receiving emails from it, send an email to datafari+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Julien MASSIERA
Expert en technologies de recherche
France Labs – Les experts du Search
Vainqueur du challenge Internal Search de EY à Viva Technologies 2016
www.francelabs.com

[wilhel...@gmail.com]

Hi Julien,

I just wanted to let you know that the problem somehow went away. As I am using arch, which is a rolling release, it appears that some underlying library might have caused the problem. Unfortunately, I cannot tell you which package update resolved the case. Anyway, MCF is not leaking memory anymore.

As I am using system PostgreSQL and arch updated already to 10.1, I would like to tell you that datafari 3.2.1 also runs happily with PostgreSQL 10.1 after updating the database.

Regards,

Wilhelm

[Julien]

Hi Wilhelm,

 

Glad to see that the problem was solved with an update, we will keep your feedback in mind.
Thank you also for the validation of PostgreSQL 10.1 on Datafari 3.2.1, it will help us to shorten the delay to implement it in Datafari !

 

Regards,
Julien

 

De : wilhel...@gmail.com
Envoyé le :mercredi 15 novembre 2017 09:19
À : Datafari
Objet :Re: ManifoldCF 2.6 files connector memory leak

Garanti sans virus. www.avast.com