Re: Gitit RAM memory usage with plugins

[fiddlosopher]

When you compile with plugins support, the 'ghc' library is linked in.  This library includes all that is needed to interpret and compile Haskell programs -- so you're basically linking in a full compiler. So a significant increase in memory usage is expected.

By comparison, here is what I see on my system (second instance uses plugins, first instance does not; both use the same binary compiled with plugins support).

website  13629  0.0  3.9  85452 30480 pts/3    Sl+  14:12   0:00 /home/jgm/.cabal/bin/gitit -f my.conf

website  13749  4.1 20.4 203672 157040 pts/2   Sl+  14:19   0:03 /home/jgm/.cabal/bin/gitit -f my.conf

(And, by the way, this is the result soon after starting both servers.) The difference is similar to what you are seeing on your system.

John

On Monday, October 22, 2012 9:16:22 AM UTC-7, Ondřej Grover wrote:

Hello,

First of all, let me congratulate everyone involved, you've made something that has completely changed our documentation capabilities.

We were considering writing our own LaTeX <-> HTML converter tool as we have (and need to have) lost of LaTeX and HTML documentation, but we had no transparent way of integrating it.

So Gitit saved us.

Nearly a month later, we have added some custom plugins. but then the RAM usage of the gitit instance spiked up quite high, although these are extremely simple plugins, they just search for a string and execute a system command.

So, is this normal: ?
buon plugins # ps aux | grep [g]itit
svoboda  12336  0.2  3.7 423564 230652 ?       Ssl  17:42   0:04 gitit -f gitit.conf
svoboda  17456  1.0  0.5 300816 35728 ?        Ssl  Oct21  15:54 gitit -f gitit.conf

The second instance does not use plugins.

I've read in the docs that the Plugins framework increases the memory footprint, but isn't this too much? I mean, the compiled plugins have only a few kB.
Could I make the footprint smaller by compiling them and installing them myself under Network.Gitit ?

Thank you for your time and effort.

Kind regards,

Ondrej Grover
GOLEM tokamak server admin,
FNSPE CTU Prague

[Ondřej Grover]

Thank you for confirming that it's not abnormal.

Still, I'd like to investigate what could I do to get around it. I see these alternatives:

1) Directly patch the gitit source and integrate my plugins
2) Patch only Plugins.hs and remove "import ghc" and in gitit.conf and put smth like Network.Gitit.MyPlugin which would be a compiled module which would have linked only necessary libs.

Could approach no. 2 work?

If something like that could work, I propose an enhancement (perhaps I could even write a patch for it) that there would be two compile options : --compiled-plugins and  --interpreted plugins, the former would not link full ghc and import only compiled modules, the latter would link ghc and interpret them

Kind regards,

Ondrej Grover

[Ondřej Grover]

Or, one more approach, could Plugins.hs be compiled with --shared?

On Monday, October 22, 2012 8:28:53 PM UTC+2, fiddlosopher wrote:

[John MacFarlane]

+++ Ondřej Grover [Oct 22 12 23:38 ]:

> Thank you for confirming that it's not abnormal.
> Still, I'd like to investigate what could I do to get around it. I see
> these alternatives:
> 1) Directly patch the gitit source and integrate my plugins

This would certainly work, but it makes it harder to keep up with
updates.

> 2) Patch only Plugins.hs and remove "import ghc" and in gitit.conf and
> put smth like Network.Gitit.MyPlugin which would be a compiled module
> which would have linked only necessary libs.
> Could approach no. 2 work?
> If something like that could work, I propose an enhancement (perhaps I
> could even write a patch for it) that there would be two compile
> options : --compiled-plugins and --interpreted plugins, the former
> would not link full ghc and import only compiled modules, the latter
> would link ghc and interpret them

I believe the ghc module is needed even to load compiled modules.

A third option would be to change the plugin system so that plugins
are executable programs (filters for json-serialized pandoc documents),
or, better, add a parallel "filters" system.

John

[John MacFarlane]

+++ Ondřej Grover [Oct 22 12 23:46 ]:

> Or, one more approach, could Plugins.hs be compiled with --shared?

That may also be an option.