Can we reduce fetch material time?

[Komgrit Aneksri]

Hello Team,

I have >1000 material and They are using "Regularly fetch updates to this repository" option

I have issue about fetch materials so they will take time 5-10 mins after commit and push code to git for run our pipeline.

Can we reduce fetch material time? or any your suggest solutions.

Best Regards,

Komgrit

[Chad Wilson]

Hiya Komgrit 👋

It depends where the time is being spent, but 5-10 mins is bad and definitely not good enough!

Usually such a long wait implies a lot of errors and throttling or some other slowness in GoCD.

1. Which git repository manager are you using?
• Gitlab if I recall correctly? Is it the Cloud/SaaS version or self managed? Or Dedicated?
2. Are errors only seen on the server logs or also within agents and job console logs when trying to run/execute jobs? Or neither?
3. Are you mainly using SSH or HTTPS clones?
   
4. Did you ever complete your migration from H2 to Postgres, or you are still using H2?
5. Which GoCD version are you on now?

I cannot remember details for your setup, but if polling is the problem, possible techniques to directly reduce polling include

• try and change everything to disable polling/fetches/autoUpdates and use notification webhooks from the repo manager instead so GoCD only checks for changes when a change is made on the repo: https://api.gocd.org/current/#webhook This even works with most custom material plugins (if you use the Git path material plugin, for example) but can require a lot of configuration depending on your Gitlab setup.
• if still getting throttled just due to sheer numbers of parallel clones inside agents/jobs - set up a HTTPS proxy cache for you git repo manager and send notifications here; only `git fetch` when notified, and point GoCD at this proxy/cache instead.
  

If the errors are not from throttling on Gitlab, it's possible they are just due to speed of processing the # of materials; and you should dig into what exactly is slow. In the logs, this is usually seen in the MaterialUpdateListener thread logging or [Material Update] - things like Skipping update of material with an error message. You can turn on additional debug logging for logging category com.thoughtworks.go.server.perf in logback-include.xml to get a better idea of what might be happening - but only worth doing if there are no other obvious errors/exceptions/logs that explain why it's so slow. Even without this special configuration, in such cases you will usually see more obvious logs for problems.

If the issues are not to do with the upstream repository manager (you don't see errors in UI or logs from Git that imply getting throttled) I'd be interested to help you see what is actually slow here. If you can share a redacted snapshot from /go/api/support that might help. 
 
-Chad

--
You received this message because you are subscribed to the Google Groups "GoCD Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/go-cd/4b883fc0-80db-4afc-b22f-fd1c2f848f0cn%40googlegroups.com.

[Komgrit Aneksri]

Hello Chad,

Thank you again for your supporting.

These below are quick answering.

1. Which git repository manager are you using?

1. • Gitlab if I recall correctly? Is it the Cloud/SaaS version or self managed? Or Dedicated?
     

1. • - We are using gitlab self managed.

1. Are errors only seen on the server logs or also within agents and job console logs when trying to run/execute jobs? Or neither?
   

1. - I will find the error in logs and ping you later.

1. Are you mainly using SSH or HTTPS clones?
   

1. - HTTPS clone

1. Did you ever complete your migration from H2 to Postgres, or you are still using H2?
   

1. - We migrated to Postgres already.

1. Which GoCD version are you on now?
   

1. - v23.1.0
   
   

As for another your suggestion to checking. i will try to look at them and response you back.

Best Regards,

Komgrit