Problem with Clearcase plugin running many builds in parallel

[Sascha Herrmann]

Running:

Jenkins ver. 1.532.3

Jenkins ClearCase Plug-in ver. 1.5 using snapshot views

Clearcase 7.1.2.12

Hi,

we're currently setting up a new CI machine. The machine is quite powerful and we're trying to run as many build jobs in parallel as we can.

No we run into two issues related to Clearcase:

1. It seems that after the catcs and before the setcs, an "endview" is issued. We run into the problem that the setcs then fails (with an immediate timeout):

07:12:55 [Misc_HEAD_Printing] $ "C:\Program Files\IBM\RationalSDLC\ClearCase\bin\cleartool" endview -server nightly_Misc_HEAD_Printing_view

...

07:12:55 Running cleartool setcs, this operation may take a while
07:12:55 [ccview] $ "C:\Program Files\IBM\RationalSDLC\ClearCase\bin\cleartool" setcs -current -overwrite
07:12:55 cleartool: Error: Timed out trying to start view_server for view <host name>:E:\oracle\clearcase\nightly_Misc_HEAD_Printing_view.vws

Our IT is currently puzzled why this happens. They're asking why the plugin issues the "endview" command. So I am wondering if someone else has encountered that problem and knows how to circumvent it. I assume that it is related to the amount of parallel build jobs, as this problem does not show up on our old CI server (which kind of runs build solely in sequence).

2. As I said, we're running as many builds as possible. But there seems to be a (hardcoded?) limit of how many view_server processes the albd can spawn. And we're running into that limit, resulting in errors. I found no place where I can raise that limit in CC. So for every CC build job, we're adding a batch build step, issuing an endview command after the build, stopping the view_server process. Which is a fine workaround. But it would be great if the plugin would feature an option to end the view after the checkout. Not sure if that makes sense in general. It would for us. That's more like a feature request :o)

Regards,

Sascha

[Jes Struck]

Hey sascha,

with out vein an expert in this specific plugin, could you maybe tell if you by, parallel builds mean that you have a lot of different jobs build at the same time or you have a lot of concurrent executions of the same job? If the later then look for view name, they might use the same name for all the executions.

I have never heard about this limit though i have heard that windows has a limit on how many process it can run at a time, are you sure it's not that limit you are running into. try running test limit -p for getting the number of process. and test limit -t for number of threads.

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jes Struck,  Consultant j...@praqma.net +45 31 42 09 00 Skype:jes.struck

Praqma A/S www.praqma.net Allerød Stationsvej 4 DK-3450 Allerød +45 36 77 27 62

[Sascha Herrmann]

Hi Jes!

Me again :o)

Totally forgot. You being an expert in the plugin, what is the reason for the "endview" before the "setcs"? Our IT pinpoints this as the problem, although I don't see why there should be an immediate timeout in the first place. The commands issued earlier by the plugin start the view_server just fine. It seems to me more like a CC issue. The plugin is just calling cleartool after all. Maybe it's a timing issue, the view_server process for the view still in the process of going down, and at that very time the attempt to start it again?! I have no other explanation for an immediate timeout. But that's just guesswork of course.

Thanks for your reply.

Sascha

[Dirk Heinrichs]

Am 23.06.2014 10:15, schrieb Sascha Herrmann:

But there seems to be a (hardcoded?) limit of how many view_server processes the albd can spawn.

Not that I'm aware of.

BTW: Why snapshot views? If you don't use dynamic views, CC isn't any better than Subversion or Git.

Bye...

    Dirk

--

Dirk Heinrichs, Senior Systems Engineer, Engineering Solutions
Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach
Tel: +49 2226 1596666 (Ansage) 1149
Email: d...@recommind.com
Skype: dirk.heinrichs.recommind
www.recommind.com

[Vincent Latombe]

Hi,

the reason the endview is issued is to end any ongoing update process on that view (that may have been interrupted in a previous run, leaving clearcase in an unclean state).

Maybe the reason setcs fails for you is because endview is still running behind and takes too much time? I'm surprised to see this, I run a pretty big clearcase infrastructure myself and I never experienced this issue.

About calling endview after the checkout, it is definitely doable and I think it is a good idea.

Vincent

2014-06-24 8:33 GMT+02:00 Sascha Herrmann <goo...@maledictis.de>:

Many different jobs at the same time. The limit seems to exist. There are not that many processes running on that machine since we solely use it for Jenkins. Concurrent Perforce builds we can have as many as we want, no problem there. But with Clearcase we noticed that we get errors from cleartool. Only way to stop these was to end the view_server process after a build (these processes tend to linger, one per view, if we don't).

No such thing as a limit command on Windows.

Sascha

Am Montag, 23. Juni 2014 10:39:22 UTC+2 schrieb Jes Struck:

[Sascha Herrmann]

Hmmm .... Google Groups ... couldn't edit the posting ... tried to delete .... error on delete .... wrote a new reply .... now original reply gone after all.

We're on Windows, so there is no limit command. But we can run as many Perforce builds concurrently as we want. I doubt it's an OS limit.

Sascha

[Sascha Herrmann]

Hi!

Thanks for your reply. IMHO CC isn't better than Subversion or Git.

We're using snapshot views because we were running on problems with dynamic views with our particular CC version on our particular (old) Windows Server version we're running.

So we had to convert all dynamic views (also the views we were using for Crucible) to snapshot views.

Sascha

[Sascha Herrmann]

Hi!

I have quickly added a "Thread.sleep( 5000 )" into BaseSnapshotCheckoutAction#checkout() between endview and setcs on our build machine and will monitor if that improves things.

Maybe ClearTool.endViewServer() would be the better place for that delay. Not sure.

5000 millis is probably too much, yes. Just wanted to be on the safe side for the test.

First build flows ran successfully. Seems to confirm that there is a timing problem in our environment.

Will update the thread with the results in a couple of days when I know for sure.

Sascha

[Sascha Herrmann]

Builds run fine now. Seems that endview takes some time to finish before the next command can be called.

Would it be possible to get a delay like that into the official plugin? Maybe configurable with a system property?

Sascha

[Vincent Latombe]

Hi Sascha,

could you try the given build [1] and see whether you still encounter the issue? I have added a call to endview after the checkout. I believe it may also help with the delay you are observing when called initially (because it may be dependant on the number of active views on the host).

If that do not work, I am open to add a configurable delay, you could add it to the endview command and open a pull request against the project.

Cheers,

Vincent

[1] https://jenkins.ci.cloudbees.com/job/plugins/job/clearcase-plugin/70/org.jenkins-ci.plugins$clearcase/artifact/org.jenkins-ci.plugins/clearcase/1.6-SNAPSHOT/clearcase-1.6-SNAPSHOT.hpi

Vincent

[Sascha Herrmann]

Hi Vincent,

thanks. I will try. I doubt that it will bring an improvement, because we're already ending the view_server processes after the build using a batch script as I mentioned previously.

I will update the post with my findings.

Thanks again.

Sascha

[Sascha Herrmann]

That was quick. It doesn't work. Immediate timeouts after endview again.

Sascha

[Vincent Latombe]

Hi,

I have implemented your suggestion. I have added a configurable delay after any endview call (for n. It is set to 0 by default and you can configure it through the global plugin settings.

Corresponding build is
https://jenkins.ci.cloudbees.com/job/plugins/job/clearcase-plugin/71/org.jenkins-ci.plugins$clearcase/artifact/org.jenkins-ci.plugins/clearcase/1.6-SNAPSHOT/clearcase-1.6-SNAPSHOT.hpi

Let me know if it is okay for you, then I'll issue a 1.5.1 with the fix.

Cheers,

Vincent

[Sascha Herrmann]

Hi Vincent,

sure that's okay. I was going for a simple system property since the error (from your reaction) seemed a bit exotic and has only been reported by me so far.

But configuration in the plugin settings is great, thanks. So I will not create that pull request,

Sascha

[Sascha Herrmann]

Hi!

Version 1.5.1 doesn't work or I am doing it wrong.

I set the delay to "2" in the settings, but when I come back to the settings after saving, it's "0" again, and there is no delay applied in endview according to the logs.

Sascha

[Vincent Latombe]

Errrrh, if it comes back to 0, there must be an issue, I will check that back.

Vincent

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

[Sascha Herrmann]

Hi!

Thanks. It seems to be just a thing of saving the setting. When I put the "2" directly into the endViewDelay tag in the XML and then restart Jenkins, then the delay is applied fine and I see the "2" in the settings.

Sascha

[Vincent Latombe]

I just released 1.5.2 with the form fix.

Vincent

--

[Sascha Herrmann]

Hi Vincent,

1.5.2 works fine. Thanks for the assistance and the fix.

Sascha

[Suganya Ravikumar]

We are using ClearCase 9.0.1.4 and version 1.6.3 of the plugin and run into this issue. How do I get the endViewDelay fix?

14:47:45 [Label Job] $ /usr/atria//bin/cleartool endview -server Jenkins_view

 14:47:45

14:47:45 [Label Job] $ /usr/atria//bin/cleartool catcs -tag Jenkins_view

14:47:45 cleartool: Error: Timed out trying to start view_server for

Thanks

Suganya