commit access

[David van Laatum]

I created https://github.com/jenkinsci/database-plugin/pull/1 last year and would now like to make some other changes.

Can I have commit access to the database-plugin and the mysql and postgresql drivers?

https://github.com/jenkinsci/database-postgresql-plugin

https://github.com/jenkinsci/database-mysql-plugin

[Ullrich Hafner]

Are you using these plug-ins in a new plug-in? It would be interesting to hear about your experience…

The plug-in has no active maintainer anymore, interested in taking over?

Ulli

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/01a1f6a9-d0c0-4146-8f42-f5c5197bca62%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David van Laatum]

Sure. I am using it to store info about our nearly 20,000 unit tests (per branch) in a db across the distributed cluster that runs them. Has worked mostly well for ages but recently started failing on hibernate trying to invoke the validation stuff and failing because it didn't have everything it needed

[Kanstantsin Shautsou]

Hi, could you describe detailed how do you use it/architecture?

[Daniel Beck]

Please tell us your jenkins-ci.org user name so we can make you default assignee for these plugins in Jira.

[David van Laatum]

davidvanlaatum

[David van Laatum]

https://github.com/davidvanlaatum/UnitTestDB is my code, I use it along with another of my plugins https://github.com/davidvanlaatum/DistributedTests. We use them together to run the nearly 20,000 phpunit tests across a cluster of servers to take the total runtime down from several hours to under an hour. 

The problem we had with the existing jenkins stuff was that it was hard to split up the unit tests using the matrix plugin into equal runtime groups and if one unit test fatal errored a bunch of unit tests just disappeared from the list and sometimes no one noticed for quite some time. 

This lead me to write these two plugins. The distributed one runs a master job that sets up a copy of the code does any db updates via liquibase etc then runs a php script to generate a json file containing a list of tests, sets up a apache virtual host for the unit tests to talk to, then starts a configurable number of slave jobs that run the checkout/setup and start grabbing the next group of tests off the stack and runs them. If there is history around runtime of a group it sorts them such that groups that take the longest run first. When each slave finishes it copies the junit.xml files back to the master jobs workspace and runs the normal junit plugins to add the results to jenkins then the UnitTestDB is called and adds info to the db from the junit plugins data. 

The db tracks how long unit tests have been failing and who possibly caused the failure with the option for people to claim or say it wasn't them. One key difference is that if it doesn't see a unit test in a particular run it doesn't remove the failure. This fixes the problem of a unit test starts failing then another change introduces a fatal error that causes the unit test not to run, the fatal error gets fixed and the original failure now shows up as started happening in a new build. On the UnitTestDB page where people claim etc there is also an option to say a particular unit test no longer exists. 

I also have another jenkins job that runs several times a day to email people that are currently blamed for a failure along with any unclaimed failures.

I have been considering bypassing the normal jenkins junit plugins and send directly to the db as storing 20,000 unit tests per build in jenkins makes it use a lot of memory and very sluggish.

[David van Laatum]

Should also add I do have plans to release both plugins just haven't done the final touches to make them useable by anyone else yet

[Devika Pradhan]

Devikalnt

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/65CF4433-9E1C-4F07-95DC-BAFDB373EB6B%40beckweb.net.

For more options, visit https://groups.google.com/d/optout.

The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"

[Daniel Beck]

Done! I made you committer and default assignee of all three plugins.

> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/316ca4b9-6b4d-4308-baa9-e8b9a67fdabe%40googlegroups.com.

[Jesse Glick]

On Thu, Jul 23, 2015 at 8:18 PM, David van Laatum <da...@vanlaatum.id.au> wrote:
> The problem we had with the existing jenkins stuff was that it was hard to
> split up the unit tests using the matrix plugin into equal runtime groups

You can use the Parallel Test Executor plugin for this, with a
freestyle project or Workflow, e.g.

https://github.com/jenkinsci/parallel-test-executor-plugin/blob/7bdb270ffe02838a36f56d4de16ebe61cd3a9556/demo/JENKINS_HOME/jobs/flow/config.xml#L12

[David van Laatum]

Had a look at it. I think I did look at it at the time but didn't work for us then. Not sure if there is more info in the wiki as its down. From my brief play I like some bits of it but others not so much. 

• I kinda like the separate sub job thing but could be confusing to people
  
• No easy way to tell where a particular test ran, ie which server, so if it breaks you don't know where to go looking for extra info or which build log to look at for a php fatal error backtrace
  
• The way it excludes tests is both a positive and a negative to me would prefer the parent discovered all tests and included in the sub, but I know that's not always an option, so maybe an option to do either. Can see this being a problem for php projects as unit tests could fail from the same test running against the same DB at the same time if it didn't see a particular test in the last run (which happens to us on a regular basis when someone introduces a fatal error)
• It splits up the tasks at the start vs handing them out as the last one completes so if one test takes a lot longer than normal it holds up the others

Is there a nice way to pass repo vers etc down now? I spent ages looking at the code and the way I ended up doing it always felt a bit hacky

[Jesse Glick]

On Sun, Aug 2, 2015 at 12:20 AM, David van Laatum <da...@vanlaatum.id.au> wrote:
> I kinda like the separate sub job thing

Note that this is just a workaround for limitations of freestyle
projects. The Workflow step keeps everything in one job.

> The way it excludes tests is both a positive and a negative to me would
> prefer the parent discovered all tests and included in the sub, but I know
> that's not always an option

I think there is just no general and reliable way to enumerate all
tests in a project using an unknown framework/language.

> It splits up the tasks at the start vs handing them out as the last one
> completes so if one test takes a lot longer than normal it holds up the
> others

Yes, this is a fundamental limitation of the design—it expects that
execution times of individual test cases do not vary widely from their
averages.