cruiser-config.xml exceeds size limit

[Hugo Ferreira]

Hi all,

Our GoCD stopped working with this error:

org.eclipse.jgit.errors.LargeObjectException: 2506f3dcaf16646bf525968cd53bd023872d72c0 exceeds size limit

We are running around 1500 pipelines and cruiser-config.xml has around 50mb.

Any way we can bump this limit or workaround this issue?

Cheers,

[Chad Wilson]

Hi Hugo

Can you please report the specific GoCD version you are using and the full stack trace of the error from the server logs?

It's probably necessary to know the specific operation that failed and the version you are using to know what limits might be applying.

It may also help to know what git object type that is with git cat-file -t 2506f3dcaf16646bf525968cd53bd023872d72c0 (will have to run inside the config folder on your GoCD server).

I believe 50MB might some kind of default limit where Git and JGit start to treat some files differently, but not sure about which limit you are hitting, nor the consequence of changing them (some may need to be adjusted together, and will have other impacts on memory usage): https://gerrit.googlesource.com/jgit/+/refs/heads/master/Documentation/config-options.md

There are a few other similar issues reported against tools that use JGit, but I didn't come to any specific conclusions about what caused them, so probably need more specific details here.

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/2857c3a4-ead0-40f2-b16c-d792fd57cf88n%40googlegroups.com.

[Hugo Ferreira]

Hi Chad,

We were running the 19.10 version of GoCD.

I checked your link and setting  bigFileThreshold = 100 MiB on .git/config core section seems to have fixed it.

In the meantime, we took this opportunity and split this Go Server to 2 new instances on the 21.2.0 version and haven't had any major issues since then.

Cheers,

[Chad Wilson]

OK, good to hear you have a way forward.

I am curious as to whether the problem would have persisted with that large a config file on one of the more recent versions, as I believe there have been a number of JGit upgrades since then. Maybe we'll never know? :-)

I also wonder a bit how your config got to 50MB and what is in it. With 1500 pipelines, that seems to imply ~34kB per pipeline which seems a lot. Perhaps you might want to look into the way pipelines are being modelled and what the XML is composed of?

Sometimes that can imply that perhaps folks are modelling pipelines using many small stages or a large number of GoCD-modelled tasks for every step in a job; rather than perhaps source controlling the build steps alongside the actual source it's building using simple scripts or using build automation tooling to assist - which can other advantages besides reducing config overhead for GoCD.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/0b413d9d-e78b-4d32-a0f3-8f851b9658f0n%40googlegroups.com.

[Sriram Narayanan]

There could also be other opportunities for improvement such as:

- moving from individual pipelines to templates

- organising tasks ( what is a job in Jenkins) into jobs that can run in parallel in a stage, and then sequencing stages that must run together into a pipeline.

And of course, Chad has reminded below that instead of writing the commands as part of the pipeline, it is better to move those commands into a version controlled scripts that is checked out as a material for the pipeline with each script being appropriately referenced as a task in a job.

— Sriram

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH-%3D%3D-b%2Ba%3DvhMEQhOTLuSq%2B2mojLv0Qf-TgJrdct6QG0OQ%40mail.gmail.com.