Pipeline Memory Issues

[ddesc...@fitbit.com]

We have a Jenkins server that runs a pipeline, loads a GIT repo, and then loads other custom groovy scripts that contain more pipeline commands from that repo. After a day or so we end up with 100s of thousands of loaded classes on the server. This amount then stays static for several days or a week. For that week, GC seems to chug away, then there is a three-hour window where memory creeps up, and GC starts spinning really hard. The server never OOMs, just becomes unresponsive.

- I've seen advice on turning this on, but we actually upgraded to groovy 2.4.8, so I think it's moot: groovy.use.classvalue

- We've experimented with G1GC, and CMS

Java -> 1.8

Groovy -> 2.4.8

Jenkins -> 2.7.1

Workflow plugins:

name / version /  enabled / pinned

workflow-aggregator	2.2	true	false
workflow-api	2.1	true	false
workflow-basic-steps	2.0	true	false
workflow-cps	2.9	true	false
workflow-cps-global-lib	2.1	true	false
workflow-durable-task-step	2.3	true	false
workflow-job	2.3	true	false
workflow-multibranch	2.8	true	false
workflow-remote-loader	1.2	true	false
workflow-scm-step	2.2	true	false
workflow-step-api	2.2	true	false
workflow-support	2.2	true	false