SonarQube 5.2 analysis errors

[Jorge Costa]

Hi,

Is it so that we can now get analysis errors in server altought the analysis in client has passed? 

i get this:

08:50:51.129 INFO  - Analysis reports generated in 186ms, dir size=55 KB

08:50:51.172 INFO  - Analysis reports compressed in 42ms, zip size=34 KB

08:50:51.272 INFO  - Analysis reports sent to server in 100ms

08:50:51.273 INFO  - ANALYSIS SUCCESSFUL, you can browse http://localhost:9000/dashboard/index/com.fmrco.ist.fot:tws

08:50:51.273 INFO  - Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report.

08:50:51.274 INFO  - More about the report processing at http://localhost:9000/api/ce/task?id=AVB5gB9wh-rUOjTl4j2E

INFO: ------------------------------------------------------------------------

INFO: EXECUTION SUCCESS

INFO: ------------------------------------------------------------------------

Total time: 6.730s

Final Memory: 9M/423M

INFO: ------------------------------------------------------------------------

But then i go to sonar and and dont see the project. and at then i get this:

java.lang.IllegalStateException: Cannot persist sources of com.fmrco.ist.fot:tws:CppProjectMultiModule/CppProjectMultiModule.cpp
	at org.sonar.server.computation.step.PersistFileSourcesStep$FileSourceVisitor.visitFile(PersistFileSourcesStep.java:131) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visitNode(DepthTraversalTypeAwareCrawler.java:72) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:44) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:91) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:47) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:91) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:47) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.step.PersistFileSourcesStep.execute(PersistFileSourcesStep.java:88) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:39) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.taskprocessor.report.ReportTaskProcessor.process(ReportTaskProcessor.java:53) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.taskprocessor.CeWorkerRunnableImpl.executeTask(CeWorkerRunnableImpl.java:78) [sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.taskprocessor.CeWorkerRunnableImpl.run(CeWorkerRunnableImpl.java:55) [sonar-server-5.2-RC2.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_60]
	at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [na:1.8.0_60]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) [na:1.8.0_60]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_60]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_60]
	at java.lang.Thread.run(Unknown Source) [na:1.8.0_60]
Caused by: java.lang.IllegalArgumentException: End offset 87 is defined outside the length (86) of the line 1
	at org.sonar.server.computation.source.RangeOffsetHelper.validateEndOffsetNotGreaterThanLineLength(RangeOffsetHelper.java:66) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.source.RangeOffsetHelper.offsetToString(RangeOffsetHelper.java:39) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.source.HighlightingLineReader.read(HighlightingLineReader.java:68) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.source.ComputeFileSourceData.read(ComputeFileSourceData.java:75) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.source.ComputeFileSourceData.compute(ComputeFileSourceData.java:52) ~[sonar-server-5.2-RC2.jar:na]
	at org.sonar.server.computation.step.PersistFileSourcesStep$FileSourceVisitor.visitFile(PersistFileSourcesStep.java:128) ~[sonar-server-5.2-RC2.jar:na]
	... 18 common frames omitted
2015.10.17 15:32:59 ERROR [o.s.s.c.t.CeWorkerRunnableImpl] Executed task | project=com.fmrco.ist.fot:tws | id=AVB1ycPykLNmnhSxThXJ | time=1423ms

This is a big issue for us, because we can go for days until we figure out that the analysis is actually broken. Is there some leverage for this, at least email sent to admins and project administrators that the analysis failed?

br,

jc

[wirth.guent...@gmail.com]

Natural solution would be to have an option in sonar runner to wait for the result and provide it on client side. To go one step further this should be the default behavior to be backward compatible (turning wait of an additional option). See no other possibility to integrate SQ in build automation otherwise?

@SonarQube: do you see a possibility to add such a feature to sonar runner 2.5?

[Simon Brandhof]

Hi Jorge, 

Can you check the encoding of CppProjectMultiModule.cpp ? It seems that it's different than the encoding configured on the project.

In any case this limitation relates to https://jira.sonarsource.com/browse/SONAR-6100.

Regards

Simon BRANDHOF | SonarSource

Tech Lead & Co-Founder
http://twitter.com/SimonBrandhof

On 18 October 2015 at 15:00, <wirth.guent...@gmail.com> wrote:

Natural solution would be to have an option in sonar runner to wait for the result and provide it on client side. To go one step further this should be the default behavior to be backward compatible (turning wait of an additional option). See no other possibility to integrate SQ in build automation otherwise?

@SonarQube: do you see a possibility to add such a feature to sonar runner 2.5?

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/3cc2bb74-e41d-4d9d-bb06-a7f0450f92c6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jorge Costa]

Could be more clear the error message, but anyway I got it working. However the question was more about the fact that we no longer can use this in a continuous integration system, any thoughts about that

Br

Jc

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/CA%2BZZQ37i_duUMLpu3nP1ARKSJXQK1n2CT2y-aY9Vca_%3DCwPfaA%40mail.gmail.com.

[Fabrice Bellingard]

On Mon, Oct 19, 2015 at 9:07 AM, Jorge Costa <jmec...@gmail.com> wrote:

Could be more clear the error message, but anyway I got it working. However the question was more about the fact that we no longer can use this in a continuous integration system, any thoughts about that

Why couldn't you use it in a CI system any longer? We've always been using SQ at SonarSource, and this is still the case, SQ 5.2 hasn't changed anything. Processing of analysis reports on server-side should very rarely fail, this is why we haven't implemented a notification feature for the moment. Obviously, this might come later on if we feel that it's really required. But as I said, we've been living with SQ 5.2 for now almost 3 months and we did not feel this need for the moment.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/CACOtS-pCs-TSnWDZSfwwiQ_cznL6uWP67xhMZq_1pKhBaEu7FA%40mail.gmail.com.

[Jorge Costa]

I suppose you don't cover every user case. It was funny that the first time I run the analysis I got straight away the failure. Perhaps just unlucky.

To be clear obviously we can use it in ci, but it can cause issues for sure and when they happen people start doubting our ci system. This we want to avoid, because we use sonar branches and this step is fundamental for or code review police.

And imo its just weird that a analyses passes but doesn't. 

[Fabrice Bellingard]

On Tue, Oct 20, 2015 at 4:21 PM, Jorge Costa <jmec...@gmail.com> wrote:

I suppose you don't cover every user case. It was funny that the first time I run the analysis I got straight away the failure. Perhaps just unlucky.

That's the purpose of the RC phase ;-) And we're grateful to you that you give us this feedback so that we can get the chance to fix this before GA!

 

To be clear obviously we can use it in ci, but it can cause issues for sure and when they happen people start doubting our ci system. This we want to avoid, because we use sonar branches and this step is fundamental for or code review police.

I cannot refer to the documentation as it's not published yet (but almost ready!), but just to make things clear: when a problem happens on the processing of the report, users will clearly see this on the project dashboard:

 

And imo its just weird that a analyses passes but doesn't. 

The paradigm has changed, so this is not surprising. If you are doing continuous inspection (like what we advocate), then you have the notifications available in SonarQube to replace everything that you are doing on CI side. And as I said, we might add a notification about failing report processing - but only if we feel that it's really required.

[Jorge Costa]

On Tue, 20 Oct 2015 at 18:15 Fabrice Bellingard <fabrice.b...@sonarsource.com> wrote:

On Tue, Oct 20, 2015 at 4:21 PM, Jorge Costa <jmec...@gmail.com> wrote:

I suppose you don't cover every user case. It was funny that the first time I run the analysis I got straight away the failure. Perhaps just unlucky.

That's the purpose of the RC phase ;-) And we're grateful to you that you give us this feedback so that we can get the chance to fix this before GA!

 

To be clear obviously we can use it in ci, but it can cause issues for sure and when they happen people start doubting our ci system. This we want to avoid, because we use sonar branches and this step is fundamental for or code review police.

I cannot refer to the documentation as it's not published yet (but almost ready!), but just to make things clear: when a problem happens on the processing of the report, users will clearly see this on the project dashboard:

 

not in our case, sonar data is taken from branch and displaying along with source code review (and if it takes more than 3 clicks to get to that information usually people just dont do it). 

And imo its just weird that a analyses passes but doesn't. 

The paradigm has changed, so this is not surprising. If you are doing continuous inspection (like what we advocate), then you have the notifications available in SonarQube to replace everything that you are doing on CI side. And as I said, we might add a notification about failing report processing - but only if we feel that it's really required.

Continuous Integration paradigm hasnt changed, and imo Continuous Inspection is a element of continuous integration so i dont see any reason why a element of a continuous integration would not follow the same rules that apply to rest (you still use Fail Fast approach dont you, at least during analysis). 

I am sure it should not be a big issue to just have the sonar runner poll server for analysis finished, and fails if it does not success. i bet that there are even fancier ways of doing it.  

hope i was able to show my point.

anyway, so far new features look promising.

[Fabrice Bellingard]

On Tue, Oct 20, 2015 at 5:33 PM, Jorge Costa <jmec...@gmail.com> wrote:

 

not in our case, sonar data is taken from branch and displaying along with source code review (and if it takes more than 3 clicks to get to that information usually people just dont do it). 

I'm sorry Jorge, I don't get what you're saying, you're not describing your process enough so I don't understand your use case. I suspect that you have put a specific process in place that is "over-engineered" and that therefore does not fit correctly with the approach we are pushing.

And imo its just weird that a analyses passes but doesn't. 

The paradigm has changed, so this is not surprising. If you are doing continuous inspection (like what we advocate), then you have the notifications available in SonarQube to replace everything that you are doing on CI side. And as I said, we might add a notification about failing report processing - but only if we feel that it's really required.

Continuous Integration paradigm hasnt changed, and imo Continuous Inspection is a element of continuous integration so i dont see any reason why a element of a continuous integration would not follow the same rules that apply to rest (you still use Fail Fast approach dont you, at least during analysis). 

I am sure it should not be a big issue to just have the sonar runner poll server for analysis finished, and fails if it does not success.

But what's the purpose in the end? Why would the CI tool have to wait for the SQ report to be processed? (this is useless, this would just prevent an executor to run other jobs meanwhile...) Why would the CI job have to be red when an issue occurs *inside* SonarQube? 

[Jorge Costa]

On Wed, 21 Oct 2015 at 11:22 Fabrice Bellingard <fabrice.b...@sonarsource.com> wrote:

On Tue, Oct 20, 2015 at 5:33 PM, Jorge Costa <jmec...@gmail.com> wrote:

 

not in our case, sonar data is taken from branch and displaying along with source code review (and if it takes more than 3 clicks to get to that information usually people just dont do it). 

I'm sorry Jorge, I don't get what you're saying, you're not describing your process enough so I don't understand your use case. I suspect that you have put a specific process in place that is "over-engineered" and that therefore does not fit correctly with the approach we are pushing.

standard github flow, feature branches for every issue. We use stash that provides code review and "sonar for stash plugin (https://marketplace.atlassian.com/plugins/ch.mibex.stash.sonar4stash)" that provides sonar analysis directly on the pull request. nothing really "over-engineered" here 

  

And imo its just weird that a analyses passes but doesn't. 

The paradigm has changed, so this is not surprising. If you are doing continuous inspection (like what we advocate), then you have the notifications available in SonarQube to replace everything that you are doing on CI side. And as I said, we might add a notification about failing report processing - but only if we feel that it's really required.

Continuous Integration paradigm hasnt changed, and imo Continuous Inspection is a element of continuous integration so i dont see any reason why a element of a continuous integration would not follow the same rules that apply to rest (you still use Fail Fast approach dont you, at least during analysis). 

I am sure it should not be a big issue to just have the sonar runner poll server for analysis finished, and fails if it does not success.

But what's the purpose in the end? Why would the CI tool have to wait for the SQ report to be processed? (this is useless, this would just prevent an executor to run other jobs meanwhile...)

thats one reason, not the most important thought, the thing we are concern is the lack of visibility of failed builds (analysis builds) during our ci stage that in turn will cause sonar issues to slip into master.

 

Why would the CI job have to be red when an issue occurs *inside* SonarQube? 

if you call the ci job, run analysis on project x, it will be weird that its green but its not working, dont you agree? for me a failure in server or locally that belongs to a certain analysis job should be considered as failure always not only when locally it fails

[Fabrice Bellingard]

On Wed, Oct 21, 2015 at 10:44 AM, Jorge Costa <jmec...@gmail.com> wrote:

I'm sorry Jorge, I don't get what you're saying, you're not describing your process enough so I don't understand your use case. I suspect that you have put a specific process in place that is "over-engineered" and that therefore does not fit correctly with the approach we are pushing.

standard github flow, feature branches for every issue. We use stash that provides code review and "sonar for stash plugin (https://marketplace.atlassian.com/plugins/ch.mibex.stash.sonar4stash)" that provides sonar analysis directly on the pull request. nothing really "over-engineered" here 

So this is the same way that we're doing pull request analyses here at SonarSource with our GitHub plugin. In such case, you're supposed to do "preview" analyses, and so to not push data the server. So your initial concern is not valid because no processing is supposed to be done on server-side (so no potential error you could miss).

What I understand is that you want to prevent any code to be pushed into the main development branch if it might break your requirements (i.e. quality gate in SonarQube). This case will be covered by the "what if" feature that is not available yet => https://jira.sonarsource.com/browse/SONAR-6763

[Jorge Costa]

On Wed, 21 Oct 2015 at 12:02 Fabrice Bellingard <fabrice.b...@sonarsource.com> wrote:

On Wed, Oct 21, 2015 at 10:44 AM, Jorge Costa <jmec...@gmail.com> wrote:

I'm sorry Jorge, I don't get what you're saying, you're not describing your process enough so I don't understand your use case. I suspect that you have put a specific process in place that is "over-engineered" and that therefore does not fit correctly with the approach we are pushing.

standard github flow, feature branches for every issue. We use stash that provides code review and "sonar for stash plugin (https://marketplace.atlassian.com/plugins/ch.mibex.stash.sonar4stash)" that provides sonar analysis directly on the pull request. nothing really "over-engineered" here 

So this is the same way that we're doing pull request analyses here at SonarSource with our GitHub plugin. In such case, you're supposed to do "preview" analyses, and so to not push data the server. So your initial concern is not valid because no processing is supposed to be done on server-side (so no potential error you could miss).

if preview mode would create coverage and duplication data we would use it (and we use c++, c#, f#, JavaScript, etc). coverage is very important when applying "hold your ground strategy in sonar" so we need it.

 

What I understand is that you want to prevent any code to be pushed into the main development branch if it might break your requirements (i.e. quality gate in SonarQube). This case will be covered by the "what if" feature that is not available yet => https://jira.sonarsource.com/browse/SONAR-6763

we dont use sonar quality gates, we just try to make sure the current pr does not worsens our technical debt -> our quality gate is in the pull request. so any of the following should be fixed:

SONAR-6763 does not cover the fact that the main analysis might be not working although everything is green in ci (even considering the risk of this happening is small)