SCM ClearCase 1.0: Decorator fails if a file is checked-out locally (un-reserved, un-mastered)

[ber...@hotmail.com]

Hi,

thanks for the ClearCase support with SQ 5.x.

I run locale analysis with activated SCM ClearCase plug-in and some of my targets could not be stored in SQ DB. I noticed that a checked-out file will create an exception and I have to disable SCM support for local analysis.

I am not sure whether https://jira.sonarsource.com/browse/SONARSCCLC-2 will solve this issue but I guess the option '-nco' could be used to handle this situation.

http://www.paperlined.org/dev/reference/clearcase_v4.0/annotate.html

HANDLING OF CHECKED-OUT VERSIONS.  Default: An error occurs if you specify a checked-out version. (The type manager can annotate checked-in versions only.)

-nco

If you specify a checked-out version, annotate uses the version from which it was checked out.

Best regards

Bert

ERROR: Error during Sonar runner execution

org.sonar.runner.impl.RunnerException: Unable to execute Sonar

        at org.sonar.runner.impl.BatchLauncher$1.delegateExecution(BatchLauncher.java:91)

        at org.sonar.runner.impl.BatchLauncher$1.run(BatchLauncher.java:75)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.sonar.runner.impl.BatchLauncher.doExecute(BatchLauncher.java:69)

        at org.sonar.runner.impl.BatchLauncher.execute(BatchLauncher.java:50)

        at org.sonar.runner.api.EmbeddedRunner.doExecute(EmbeddedRunner.java:102)

        at org.sonar.runner.api.Runner.execute(Runner.java:100)

        at org.sonar.runner.Main.executeTask(Main.java:70)

        at org.sonar.runner.Main.execute(Main.java:59)

        at org.sonar.runner.Main.main(Main.java:53)

Caused by: java.lang.IllegalStateException: The ClearCase annotate command [cleartool annotate -out - -fmt VERSION:%Ln@@@USER:%u@@@DATE:%Nd@@@ -nheader -f src/resource.h] failed: cleartool: Error: Operation "annotate" unavailable for manager "_ms_word"

        (Operation pathname was: "C:\IBM\RationalSDLC\ClearCase\lib\mgrs\_ms_word\annotate")

[Julien HENRY]

Hi Bert,

Thanks for your feedback on the ClearCase plugin. As a foreword I want to mention I have zero knowledge of ClearCase "world". The plugin was implemented by copying behavior that was implemented in Maven SCM ClearCase provider. All that to say that we are eagerly waiting for feedback/contribution from ClearCase users to make this plugin of good quality.

First question: what do you mean by "local analysis"? Do you mean a SonarQube preview analysis? Or is it a concept related to ClearCase?

Concerning checked-out version, if my understanding is correct, this is a file where you may have local modification compared to what was "pushed" on the server. If I'm correct, in this case we prefer to skip those files with a warning (this is the purpose of https://jira.sonarsource.com/browse/SONARSCCLC-2). The -nco option is not a good idea since checked-out file content may differ from original version and so result of annotate could be totally wrong.

Second question: do you know if cleartool returns specific exit code in this case? For example in the JazzRTC plugin we are using exit code to distinguish failures that are due to file not being checked-in (new file or file with local modification).

++

Julien

[ber...@hotmail.com]

Hi Julien,

Thanks for your fast response. We use ClearCase now since a decade but I am not the most experianced contact.

Sorry for the wording but "local analysis" means I run it on my desktop system and not on a build system and you are right using the option '-nco' will use annotate information of the checked-in source file and not the local file but you will get some analysis results. At the moment the exception will stop the analysis.

I checked the error code and it looks like this:

V:\XXX\sources\XXX>Test.bat
 
V:\XXX\sources\XXX>cleartool annotate \swapp\sources\XXX\XXXDisplayManagement\src\XXXHelper.h
cleartool: Error: You may not annotate a checked-out version (use no checkouts option): "\swapp\sources\XXX\XXXDisplayManagement\src\XXXHelper.h".
 
V:\XXX\sources\XXX>echo 1
1
 
V:\XXX\sources\XXX>cleartool annotate \swapp\sources\XXX\XXXDisplayManagement\src\XXXDisplayExceptions.cpp
Annotated result written to "\swapp\sources\XXX\XXXDisplayManagement\src\XXXDisplayExceptions.cpp.ann".
 
V:\XXX\sources\XXX>echo 0
0

I think you can use the same approach like you did with JazzRTC and check the error code <> 0.

Please inform me when you have a solution available and I will run some tests.

Best regards

Bert

[Julien HENRY]

Hi Bert,

I would like to avoid to blindly ignore all errors when exit code is != 0 because it would hide real configuration issues (like wrong credentials).

Do you know if there is a (fast) command we can run to know the status of a file (new / checked out) before we try to run the annotate command? Something similar to svn status.

++

Julien

[stuart...@gmail.com]

Plug in looks great and very useful - like Bert, we are seeing an issue (although slightly idfferent), when using the annotate command. Our build process generates some view private code files that the plug in attempts to annotate, and fails with:

10:24:33  ERROR: Caused by: The ClearCase annotate command [cleartool annotate -out - -fmt VERSION:%Ln@@@USER:%u@@@DATE:%Nd@@@ -nheader -f local/version.cs] failed: cleartool: Error: Not a vob object: "local/version.cs".

It would be beneficial if the plug in did not attempt to annotate files that were not a clearcase element - this could perhaps be done with a simple check at the start of the blame method (I don't have a java development environment otherwise I would just write the code and submit a pull request!)

private void blame(FileSystem fs, InputFile inputFile, BlameOutput output) {

    String filename = inputFile.relativePath();
       // new method to check if the file is a VOB object
       if (!IsVOBObject(filename) return;

IsVOBObject can do something like a "cleartool ls -s " + filename, and read the output

If it is not a VOB object, there will be no versioning info attached

What do you think? Maybe there is an alternative better way though?

Sorry for putting code in a post, as above, I don't have a java dev environment set up to build this,

thanks, Stu

[ber...@hotmail.com]

Hi Julien,

I could not find any documentation on cleartool error codes but I found a short description of the CAL (ClearCase access library) which is not promising at all (see

http://www-01.ibm.com/support/docview.wss?uid=swg21149702). An additional command will increase the duration for the analysis and parsing stderr for "You may not annotate a checked-out version" might also be helpful.

Best regards

Bert

[Julien HENRY]

Hi Bert and Stuart,

I see the following options:

1) Run an additional cleartool command (ls -s ?) prior to run annotate that would tell if the file is a vob and if it is checked-out. The problems here are a) find the right command + output parsing b) extended duration of the process

2) Only run cleartool annotate and rely on stderr parsing to distinguish "accepted" errors from real problems (bad credentials, server down, ...). The problem here is could we rely on stability of error messages depending on user locale and ClearCase version?

3) Simply ignore all errors. The problem is that a bad configuration (like wrong credential) or server down will not stop the analysis and SonarQube will be fed with empty SCM data. It means features like issue auto-assign will be broken (new issues will not be assigned).

WDYT?

[Stuart Smith]

Option 2 for me. Would not want to introduce performance issues or any other potential for errors. I think the likelihood of the error messages changing between versions of clearcase is reasonably low, and I don't believe the application is localised. We could always store the acceptable error messages in an external file that can be easily updated (or localised) if required.

[ber...@hotmail.com]

Option 2 is the best choice. The performance impact is minimal.

[Julien HENRY]

OK guys, let's give a try to this option.

Would you mind testing this SNAPSHOT version of the plugin?

https://github.com/SonarCommunity/sonar-scm-clearcase/releases/download/1.1-rc1/sonar-scm-clearcase-plugin-1.1-SNAPSHOT.jar

Thanks

Julien

[stuart...@gmail.com]

Great stuff - this all works for me now and fixes my issue! For some reason all of our annotations come out as the guy who did the merge, but I think this may be a problem our end with our clearcase config, so I'll go and investigate further

thanks, Stu

[Julien HENRY]

Great stuff - this all works for me now and fixes my issue!

Cool!

 

For some reason all of our annotations come out as the guy who did the merge, but I think this may be a problem our end with our clearcase config, so I'll go and investigate further

For many other SCM (Git, SVN,  Perforce) this is a way to tune the "annotate" command to decide if computation of annotation should stop on merge (reporting user who did the merge as last committer) or follow branch history (more expensive but allow to report what we expect to be reported as line author in SonarQube). Maybe there is a similar option in ClearCase, but I have not seen it in documentation.

[stuart...@gmail.com]

Yes, this is what I expected Clearcase to do as well, but it seems it is not possible. After looking through the docs and some experimentation, I cannot get the annotate command to tell me who was really the author of a line, once the file has been merged back onto our MAIN branch. It will always only report who changed (merged) the file last :/

[ber...@hotmail.com]

Hi Julien,

there is still the exception whenever a file is checked-out.

Best regards

Bert

23:49:32.518 DEBUG - Release semaphore on project : org.sonar.api.resources.Project@6d1c20f[id=2576,key=XXXX:XXXX:Dev,qualifier=TRK], with key batch-XXXX:XXXX:Dev
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
Total time: 7:10.987s
Final Memory: 68M/3524M
INFO: ------------------------------------------------------------------------

ERROR: Error during Sonar runner execution
org.sonar.runner.impl.RunnerException: Unable to execute Sonar
        at org.sonar.runner.impl.BatchLauncher$1.delegateExecution(BatchLauncher.java:91)
        at org.sonar.runner.impl.BatchLauncher$1.run(BatchLauncher.java:75)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.sonar.runner.impl.BatchLauncher.doExecute(BatchLauncher.java:69)
        at org.sonar.runner.impl.BatchLauncher.execute(BatchLauncher.java:50)
        at org.sonar.runner.api.EmbeddedRunner.doExecute(EmbeddedRunner.java:102)
        at org.sonar.runner.api.Runner.execute(Runner.java:100)
        at org.sonar.runner.Main.executeTask(Main.java:70)
        at org.sonar.runner.Main.execute(Main.java:59)
        at org.sonar.runner.Main.main(Main.java:53)

Caused by: java.lang.IllegalStateException: The ClearCase annotate command [cleartool annotate -out - -fmt VERSION:%Ln@@@USER:%u@@@DATE:%Nd@@@ -nheader -f src/XXXX.h] failed: cleartool: Error: You may not annotate a checked-out version (use no checkouts option): "src/XXXX.h".
 
        at org.sonar.plugins.scm.clearcase.ClearCaseBlameCommand.blame(ClearCaseBlameCommand.java:70)
        at org.sonar.plugins.scm.clearcase.ClearCaseBlameCommand.blame(ClearCaseBlameCommand.java:53)
        at org.sonar.batch.scm.ScmSensor.execute(ScmSensor.java:84)
        at org.sonar.batch.scan.SensorWrapper.analyse(SensorWrapper.java:59)
        at org.sonar.batch.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:59)
        at org.sonar.batch.phases.SensorsExecutor.execute(SensorsExecutor.java:51)
        at org.sonar.batch.phases.DatabaseModePhaseExecutor.execute(DatabaseModePhaseExecutor.java:120)
        at org.sonar.batch.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:264)
        at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:92)
        at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:77)
        at org.sonar.batch.scan.ProjectScanContainer.scan(ProjectScanContainer.java:235)
        at org.sonar.batch.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:230)
        at org.sonar.batch.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:220)
        at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:92)
        at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:77)
        at org.sonar.batch.scan.ScanTask.scan(ScanTask.java:57)
        at org.sonar.batch.scan.ScanTask.execute(ScanTask.java:45)
        at org.sonar.batch.bootstrap.TaskContainer.doAfterStart(TaskContainer.java:135)
        at org.sonar.api.platform.ComponentContainer.startComponents(ComponentContainer.java:92)
        at org.sonar.api.platform.ComponentContainer.execute(ComponentContainer.java:77)
        at org.sonar.batch.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:158)
        at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:95)
        at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:67)
        at org.sonar.runner.batch.IsolatedLauncher.execute(IsolatedLauncher.java:48)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.sonar.runner.impl.BatchLauncher$1.delegateExecution(BatchLauncher.java:87)
        ... 9 more

[Julien HENRY]

Hi Bert,

I'll update the code. Can you just tell me in which situation you get the error:

Operation "annotate" unavailable for manager "_ms_word"

Thanks

Julien

...

[ber...@hotmail.com]

Hi Julien,

this error occurs with ClearCase V8.0.1.8 and a checked-out file (unreserved, unmastered). I don't know why the "_ms_word" information is included in this message.

Best regards

Bert

[Julien HENRY]

Hi Bert,

Thanks for the precisions. I have updated the plugin:

https://github.com/SonarCommunity/sonar-scm-clearcase/releases/download/1.1-rc2/sonar-scm-clearcase-plugin-1.1-SNAPSHOT.jar

Tell me if it works for you.

++

Julien

[ber...@hotmail.com]

Hi Julien,

thanks for the update. This work perfect and I looking forward to use it in production.

Best regards

Bert

[stuart...@gmail.com]

Hi,

I have another precision that needs adding:

ERROR: Caused by: The ClearCase annotate command [cleartool annotate -out - -fmt VERSION:%Ln@@@USER:%u@@@DATE:%Nd@@@ -nheader -f Debug/version.h] failed: cleartool: Error: Cannot perform operation for derived object: "Debug/version.h".

is it ok to add this one too? happy to test when done,

thanks, Stu

On Sunday, 6 September 2015 10:36:50 UTC+1, ber...@hotmail.com wrote:

[Julien HENRY]

Hi Stuart,

Ticket created:

https://jira.sonarsource.com/browse/SONARSCCLC-3

++

Julien

[Julien HENRY]

Please test https://github.com/SonarCommunity/sonar-scm-clearcase/releases/download/1.2-rc1/sonar-scm-clearcase-plugin-1.2-SNAPSHOT.jar

As a side note we usually consider that derived files should not be included in a SQ analysis (no point to check code quality on generated code).

++

[stuart...@gmail.com]

Perfect, works a dream, thanks again!

[tge...@gomidjets.com]

If you need a GUI layer for ClearCase annotate, check this:

http://almtoolbox.com/annotate

It can also integrated with Visual Studio and Eclipse. An API is also available.

[Julien HENRY]

Hi,

We would be more interested by a cross platform headless Java library instead of relying on command line parsing.

Anyway thanks for the pointer.

Julien