Grouping jobs by project, status, or attributes other than folder location

44 views
Skip to first unread message

Gus Reiber

unread,
Mar 30, 2015, 12:29:54 PM3/30/15
to jenkin...@googlegroups.com
Hey all,
   I currently using the folder plugin to group my projects together, but am finding that it is difficult to get cross-cutting job status/stage/attribute views of the jobs contained within those folders. Here are a set of examples of the sorts of queries I would like to be able to generate views based upon:
  • Show me all the jobs related to a particular project (for me, this is typically the folder)
  • Show me all the jobs that deploy to "production" (many folders might have a job that does this)
  • Show me all the jobs that use this set of slaves
  • Show me all the jobs triggered by a submission from 'John Dowe' (I can never trust that J.D., he is all over the place)
  • Show me all the jobs tied to this repository  
The metadata plugin seems like it is geared towards this issue:

....so I guess I have two questions:
-- Does anyone else attempt to make queries against Jenkins of this sort?
-- Has anyone used the Metadata plugin, or some other plugin or home-grown solution to solve this issue?

Bruno P. Kinoshita

unread,
Mar 30, 2015, 3:51:47 PM3/30/15
to jenkin...@googlegroups.com
Hey Gus

> -- Does anyone else attempt to make queries against Jenkins of this sort?

For scientists and researchers it's extremely important to be able to store, retrieve and search metadata. We have an idea of how to extend the metadata plug-in [1] (probably via several Pull Requests), so that it provides the following features:
  • Automatically collect metadata of Jenkins items (jobs, builds, artifacts, etc)
  • Let users collect extra metadata with builders/publishers/etc. For instance, Cell Profiler [2] has a metadata module that can collect metadata from file names. So a file named TRIAGE-CP001-20151120035003.txt could generate metadata such as {code: "CP001", date:"2015-11-20 03:50:03UTC"} if the user selected a File Name Metadata Collector
  • Store the metadata in an embedded triple store (Jena) using an ontology created for Jenkins, which will import other ontologies such as DC, W3C Prov, etc.
  • A search module and interface, that supports both Lucene (via jena-text module) and SPARQL.
The idea behind it is to have a pre defined ontology, which will ease learning about the data model used for the metadata, and transform Jenkins into a SPARQL endpoint. That way you'll be able to even query remote Jenkins for metadata too.

We have had some discussions about it already, but I will try to have a working version in the following months. As an example, this question "Show me all the jobs that use this set of slaves" could be answered by a SPARQL query similar to this one (syntax not tested, just a quick example):

PREFIX jenkins: http://
SELECT ?jobs
WHERE {
?jobs a jenkins:Job .
?jobs jenkins:hasSlave ?slave .
?slave jenkins:slaveName ?slaveName .
?slaveName IN ('...')
}

> -- Has anyone used the Metadata plugin, or some other plugin or home-grown solution to solve this issue?

I used it and had a look at the code, and I feel like I can extend it to support the ontology and SPARQL. @Ioannis used it too, maybe he can give his feedback on it too.

This is something that is necessary for experiments and scientific pipelines. It would be interesting if we could in some way join forces to work on it :-)

Cheers


From: Gus Reiber <gre...@cloudbees.com>
To: jenkin...@googlegroups.com
Sent: Monday, March 30, 2015 1:29 PM
Subject: Grouping jobs by project, status, or attributes other than folder location

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/3f9c0f9e-abb0-4092-804e-489f505be2ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Gus Reiber

unread,
Mar 31, 2015, 1:41:20 PM3/31/15
to jenkin...@googlegroups.com
Hey Bruno,
   Yeah, the SQL style syntax is exactly the sort of thing I am wishing I could apply to Job properties and other entities within Jenkins as well.
Since we need Folders for RBAC anyway, Folders end up being our single and dominant form of metadata and we use them to define product based job groups. That works okay, but in particular, its a poor mechanism for fetching the list of all products that went to production this past Thursday, which on the surface of it, would be a perfectly normal question to ask a build manager.

As I see it, there are three types of settings/metadata I would want to track....
  • Config settings established prior to a job run: what repositories, what security context, what Docker image, which type of build-step constructors are being used, what slave pool should be used, what test environments should be used, who gets notified, etc....
  • Settings that are purely metadata: what project name do I want to apply to these groups of Jenkins items, what are the sorts of stage/environment names I want to use across projects to track comparative progress between projects...
  • Results determined by the job execution: What artifacts were made, what compute units actually did the build, how long did it take, did it succeed, where did the artifacts go...
For the Config settings, it seems helpful to me that these setting be robustly accessible prior to the job running, as one of the key things one might want to examine is how the planned process ends up comparing to the factual results. These being computers, of course they will do what you tell them, but you might have open pieces in your job-chain that someone else can edit, or your rules may be so abstract and complicated comparing the expected to the actual is often a very useful visualization.

For the purely metadata, I would hope that they would be minimal, as it is often difficult to know in advance the ways you will want to sort and group your information later.

And for the results set, it really feels to me that the Jenkins Job centered approach might not be right here. It might be better to look first at the artifacts, which are obviously related to the job, but are the product of it, and thus slightly different.

Have you used our new CloudBees Analytics plugins at all?
They are meant to do some of this results and post-process settings sorting.

Thanks Bruno,
Always a pleasure.

Gus 



--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/3ndjfF1otus/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/2095356288.690480.1427744927153.JavaMail.yahoo%40mail.yahoo.com.

Kanstantsin Shautsou

unread,
Mar 31, 2015, 4:57:38 PM3/31/15
to jenkin...@googlegroups.com
Is it open-sourced? Could you provide link?

Gus Reiber

unread,
Mar 31, 2015, 5:06:46 PM3/31/15
to jenkin...@googlegroups.com
I am sorry, it is not open sourced.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/3ndjfF1otus/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/f9b33044-ce2b-44b2-a4d1-f04587386f26%40googlegroups.com.

Bruno P. Kinoshita

unread,
Mar 31, 2015, 7:29:51 PM3/31/15
to jenkin...@googlegroups.com
> As I see it, there are three types of settings/metadata I would want to track....

Agreed on the three of them. I didn't have a clear picture of this three types, but I will try to shamelessly use what you suggested to group metadata collectors :)

The Taverna Workflow [1] (which code was recently donated to ASF for the incubation), does a great job supporting metadata. It has a plug-in for data provenance [2], which allows researchers to track workflows and artifacts being generated by these workflows. The plug-in utilizes the PROV ontology [3]. I think it covers some of the types of settings/metadata that you mentioned.

By extending the Metadata Plug-in to use the same ontology and support metadata, a Jenkins workflow (maybe created with the new Workflow Plug-in) could be uploaded to the myexperiment [4] too :-)

This way we would be able to share workflows, re-run existing workflows, and (hopefully) make it easier to reproduce workflows. At least that's what I'm planning to investigate once I've relocated. Or **maybe**, we could even re-run a simple workflow created in Jenkins in Taverna, as long as there was some kind of compatibility between plug-ins.

> Have you used our new CloudBees Analytics plugins at all?

Wasn't aware of it. Could you send a link to some docs or presentation about it, please? Tried searching for it but I couldn't find a link.

Cheers
Bruno

Sent: Tuesday, March 31, 2015 2:41 PM
Subject: Re: Grouping jobs by project, status, or attributes other than folder location

Reply all
Reply to author
Forward
0 new messages