Programmatically searching the artifacts from multiple jobs

[Tony C]

Hi,

I've been using Jenkins at work, for just over a year. I"m not the main architect of our groovy infrastructure. I'm just a user

who has several jobs which run each night.

Each of those jobs saves a handful of artifacts (log files) from that job.

I usually go into each job and look at a specific file (common to each job), for specific text that would indicate the job had a problem.

Since there are so many jobs being run, this takes quite a bit of time. Every day.

I would like to be able to search all of the jobs which ran on a certain date and scan the artifacts for specific text.

I would probably do this outside of Jenkins, by writing a script to scan the artifact files.

However, I don't know how to gain access to those jobs programmatically, let alone scanning the artifacts of each of those jobs.

I've used the Python-Jenkins library 

https://python-jenkins.readthedocs.io/en/latest/

but it doesn't have the ability to do what I want- scanning the artifacts of several builds (or a specific build).

Thanks

[niristotle okram]

Short answer, you need something like splunk or elastic search to parse the logs and check for specific strings. 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/97a539f0-1e12-4796-9692-23c662cbb340%40googlegroups.com.

--

Sent from mobile device, excuse typos if any.

[Björn Pedersen]

Hi,

I think the best solution would be to use warnings-ng  and set up  suitable custom parser (I assume that none of the many available parsers fit your use-case). Possibly in combination with the Build Failure Analyzer plugin your jobs will show the correct status and where the errors happed.

Björn

[Tony C]

Thanks

[Tony C]

Thanks

On Sunday, August 11, 2019 at 8:47:27 PM UTC-7, ok999 wrote:

Short answer, you need something like splunk or elastic search to parse the logs and check for specific strings. 

On Sun, Aug 11, 2019 at 5:55 PM Tony C <capp...@gmail.com> wrote:

Hi,

I've been using Jenkins at work, for just over a year. I"m not the main architect of our groovy infrastructure. I'm just a user

who has several jobs which run each night.

Each of those jobs saves a handful of artifacts (log files) from that job.

I usually go into each job and look at a specific file (common to each job), for specific text that would indicate the job had a problem.

Since there are so many jobs being run, this takes quite a bit of time. Every day.

I would like to be able to search all of the jobs which ran on a certain date and scan the artifacts for specific text.

I would probably do this outside of Jenkins, by writing a script to scan the artifact files.

However, I don't know how to gain access to those jobs programmatically, let alone scanning the artifacts of each of those jobs.

I've used the Python-Jenkins library 

https://python-jenkins.readthedocs.io/en/latest/

but it doesn't have the ability to do what I want- scanning the artifacts of several builds (or a specific build).

Thanks

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/97a539f0-1e12-4796-9692-23c662cbb340%40googlegroups.com.

[Ullrich Hafner]

Or you can use the https://wiki.jenkins.io/display/JENKINS/Text-finder+Plugin 

It provides already a way to grep files for a pattern and set the build status accordingly.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ccdab323-3894-4635-a06f-5f7a74c3b3ec%40googlegroups.com.

[Tony Cappellini]

Thanks, I may have to resort to that.

I was hoping to find a way to get access to the artifact files,
outside of Jenkins though.
Then I could write my own script to run independently of Jenkins

> You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nNwRWXHOs5E/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3F83F7A5-5C3D-4786-8770-83B26B85DEC8%40gmail.com.

[Matthew Perrett]

If you had access to the Jenkins server you could copy the files locally via scp or similar and run your script. You could also configure your job to save them somewhere in the file system with a date stamp in the name to allow you to search historically results.

You could also set up a downstream job that gets triggered on completion and checks the logs for errors, which could be configured to notify you via slack/email etc if there were any issues.

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/97a539f0-1e12-4796-9692-23c662cbb340%40googlegroups.com.

[Tony C]

>>If you had access to the Jenkins server you could copy t

I do have access to the server

I'd write a script to go parse the artifact logs, if I only knew where they were.

I don't know if that requires root access or not, I dont have root access.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

[Tony C]

If I can get the admin to install this, I'll give it a try

thanks

On Monday, August 12, 2019 at 2:30:32 PM UTC-7, Ullrich Hafner wrote:

Or you can use the https://wiki.jenkins.io/display/JENKINS/Text-finder+Plugin 

It provides already a way to grep files for a pattern and set the build status accordingly.

Am 12.08.2019 um 06:49 schrieb 'Björn Pedersen' via Jenkins Users <jenkins...@googlegroups.com>:

Hi,

I think the best solution would be to use warnings-ng  and set up  suitable custom parser (I assume that none of the many available parsers fit your use-case). Possibly in combination with the Build Failure Analyzer plugin your jobs will show the correct status and where the errors happed.

Björn

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

[Martin d'Anjou]

Could the run selector plugin help?
https://github.com/jenkinsci/run-selector-plugin

Martin

[Tony Cappellini]

Perhaps. I'll have to give it a try to see what its limitations are.

When you have 5-10 artifacts per job, it takes a lot of time saving
them, so they can be attached to a Jira ticket.
Jenkins doesn't provide a way to save those files. You need to view
each one, copy it's contents into an editor, then save those files,
manually.

thanks

> --
> You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nNwRWXHOs5E/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/211067ec-a99a-4251-990f-a4e66e8b9d58%40googlegroups.com.

[Martin d'Anjou]

You can get the artifacts from Jenkins using curl:

curl -X GET "http://localhost:8080/job/p1/2/artifact/t2.txt" -o t2.txt

You can also get all the artifacts:

curl -X GET "http://localhost:8080/job/p1/2/artifact/*zip*/archive.zip" -o a.zip

For more see:

https://stackoverflow.com/questions/35920756/is-there-any-jenkins-api-to-get-artifacts-name-and-download-it

Martin

On Sunday, August 18, 2019 at 2:40:37 PM UTC-4, Tony Cappellini wrote:

Perhaps. I'll have to give it a try to see what its limitations are.

When you have 5-10 artifacts per job, it takes a lot of time saving
them, so they can be attached to a Jira ticket.
Jenkins doesn't provide a way to save those files. You need to view
each one, copy it's contents into an editor, then save those files,
manually.

thanks

On Sun, Aug 18, 2019 at 10:49 AM Martin d'Anjou
<martin....@gmail.com> wrote:
>
> Could the run selector plugin help?
> https://github.com/jenkinsci/run-selector-plugin
>
> Martin
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/nNwRWXHOs5E/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to jenkins...@googlegroups.com.

[Tony Cappellini]

Thanks Martin. That looks like the most programmatic solution of all.

To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/030c4641-e194-4562-aae0-ba7b2c6bc430%40googlegroups.com.