After upgrade to 5.2 Background Tasks fail with java.lang.IllegalArgumentException: Multiple entries with same key: MeasureKey{metricKey='lines'...

[David]

The following error occurs on most of our .NET projects since upgrading to 5.2.  Analysis succeeds with the same Quality Profiles for a smaller project but on larger projects it fails. The same projects have been analysed since 4.2 with much the same Quality Profiles.

2016.01.05 06:53:51 INFO  [o.s.s.c.s.ComputationStepExecutor] Execute component visitors | time=90421ms
2016.01.05 06:54:29 ERROR [o.s.s.c.t.CeWorkerRunnableImpl] Failed to execute task AVIQjQLLDCtH7wO-WxeD
java.lang.IllegalArgumentException: Multiple entries with same key: MeasureKey{metricKey='lines', ruleId=-6253, characteristicId=-6253}=org.sonar.db.measure.PastMeasureDto@5435ade6 and MeasureKey{metricKey='lines', ruleId=-6253, characteristicId=-6253}=org.sonar.db.measure.PastMeasureDto@102c023b
	at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150) ~[guava-17.0.jar:na]
	at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104) ~[guava-17.0.jar:na]
	at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70) ~[guava-17.0.jar:na]
	at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:254) ~[guava-17.0.jar:na]
	at com.google.common.collect.Maps.uniqueIndex(Maps.java:1166) ~[guava-17.0.jar:na]
	at com.google.common.collect.Maps.uniqueIndex(Maps.java:1140) ~[guava-17.0.jar:na]
	at com.google.common.collect.FluentIterable.uniqueIndex(FluentIterable.java:424) ~[guava-17.0.jar:na]
	at org.sonar.server.computation.step.ComputeMeasureVariationsStep$VariationMeasuresVisitor.setVariationMeasures(ComputeMeasureVariationsStep.java:145) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.step.ComputeMeasureVariationsStep$VariationMeasuresVisitor.computeMeasuresWithVariations(ComputeMeasureVariationsStep.java:124) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.step.ComputeMeasureVariationsStep$VariationMeasuresVisitor.visitAny(ComputeMeasureVariationsStep.java:115) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visitNode(DepthTraversalTypeAwareCrawler.java:60) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:44) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:91) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:47) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:91) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:47) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.step.ComputeMeasureVariationsStep.execute(ComputeMeasureVariationsStep.java:92) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:39) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.taskprocessor.report.ReportTaskProcessor.process(ReportTaskProcessor.java:53) ~[sonar-server-5.2.jar:na]
	at org.sonar.server.computation.taskprocessor.CeWorkerRunnableImpl.executeTask(CeWorkerRunnableImpl.java:78) [sonar-server-5.2.jar:na]
	at org.sonar.server.computation.taskprocessor.CeWorkerRunnableImpl.run(CeWorkerRunnableImpl.java:55) [sonar-server-5.2.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_75]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_75]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_75]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
2016.01.05 06:54:29 ERROR [o.s.s.c.t.CeWorkerRunnableImpl] Executed task | project=Vitality:a4b0c5d3:Dev | id=AVIQjQLLDCtH7wO-WxeD | time=246364ms

Our plugins are:

2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep] SonarQube plugins:
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - StyleCop 1.1 (stylecop)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Motion Chart 1.7 (motionchart)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - JavaScript 2.9 (javascript)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - ReSharper 2.0 (resharper)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Findbugs 3.3 (findbugs)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Android 1.1 (android)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Analysis Bootstrapper for Visual Studio Projects 1.2 (visualstudio)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Timeline 1.5 (timeline)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Web 2.4 (web)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - XML 1.4 (xml)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Widget Lab 1.8.1 (widgetlab)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Git 1.1 (scmgit)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - SVN 1.2 (scmsvn)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - LDAP 1.5.1 (ldap)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Tab Metrics 1.4.1 (tabmetrics)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Checkstyle 2.4 (checkstyle)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - NDepend 1.0-SNAPSHOT (ndepend)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - PMD 2.5 (pmd)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Java 3.8 (java)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - C# 4.3 (csharp)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Cobertura 1.6.3 (cobertura)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - Useless Code Tracker 1.0 (uselesscodetracker)
2016.01.05 06:51:00 INFO  [o.s.s.c.s.LogScannerContextStep]   - CSS 1.6-RC1 (css)


Also:
Upgrading plugin to CSharp 4.4 has made no difference. ie. the deployment is using all the latest plugins.
Analysis of a large Java project did not produce this problem.

Note that we use the sonar-runner, not the MSBuild runner because the latter does not work with all our projects (e.g. very large projects or where solution files are not in the root path, multiple solution files exist or various other combinations of configurations trip it up). 

We would greatly appreciate any suggested workarounds. 

[Julien Lancelot]

Hi David,

This issue is strange. Can you try to use SonarQube 5.3-RC2 ? 

I don't think it will solve this, but at least you'll have on which file this error is thrown.

Regards,

Julien LANCELOT | SonarSource

http://sonarsource.com

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/5013d6f0-116a-478a-b44c-27d554c23152%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[b...@hotmail.com]

It looks like from the log all the solution components have been loaded and it is starting on metrics but failing immediately. Could it be loading the 'lines' metric multiple times? I notice that it is the first one in the metrics table.  

I see another user with this issue:

https://stackoverflow.com/questions/34413589/sonarqube-background-tasks-failing-with-illegalargumentexception-for-java

We will test with 5.3-RC2 now.

[David]

Under the same network architecture fresh installs of both 5.3-RC2 (reporting server 1) and 5.2 (reporting server 2) using new embedded databases process the project correctly.

The correct final output should have been as below indicating the fault was with the Compute measure variations step.

2016.01.06 20:23:16 INFO  [o.s.s.c.s.ComputationStepExecutor] Compute measure variations | time=6897ms

2016.01.06 20:23:16 INFO  [o.s.s.c.s.ComputationStepExecutor] Computes Quality Gate measures | time=9ms

2016.01.06 20:23:16 INFO  [o.s.s.c.s.ComputationStepExecutor] Compute Quality profile measures | time=11ms

2016.01.06 20:23:16 INFO  [o.s.s.c.s.ComputationStepExecutor] Generate Quality profile events | time=15ms

2016.01.06 20:23:16 INFO  [o.s.s.c.s.ComputationStepExecutor] Generate Quality gate events | time=16ms

2016.01.06 20:23:17 INFO  [o.s.s.c.s.ComputationStepExecutor] Persist components | time=965ms

2016.01.06 20:23:20 INFO  [o.s.s.c.s.ComputationStepExecutor] Persist snapshots | time=3417ms

2016.01.06 20:24:05 INFO  [o.s.s.c.s.ComputationStepExecutor] Persist measures | time=44137ms

2016.01.06 20:30:17 INFO  [o.s.s.c.s.ComputationStepExecutor] Persist issues | time=372335ms

2016.01.06 20:30:17 INFO  [o.s.s.c.s.ComputationStepExecutor] Persist project links | time=24ms

2016.01.06 20:30:17 INFO  [o.s.s.c.s.ComputationStepExecutor] Persist events | time=92ms

2016.01.06 20:30:28 INFO  [o.s.s.c.s.ComputationStepExecutor] Persist sources | time=11491ms

2016.01.06 20:30:29 INFO  [o.s.s.c.s.ComputationStepExecutor] Persist tests | time=73ms

2016.01.06 20:30:29 INFO  [o.s.s.c.s.ComputationStepExecutor] Persist cross project duplications index | time=0ms

2016.01.06 20:30:30 INFO  [o.s.s.c.s.ComputationStepExecutor] Enable snapshot | time=1032ms

2016.01.06 20:30:30 INFO  [o.s.s.c.s.ComputationStepExecutor] Index components | time=793ms

2016.01.06 20:30:41 INFO  [o.s.s.c.s.ComputationStepExecutor] Purge db | time=10357ms

2016.01.06 20:30:41 INFO  [o.s.s.c.s.ComputationStepExecutor] Apply permissions | time=4ms

2016.01.06 20:31:14 INFO  [o.s.s.c.s.ComputationStepExecutor] Index issues | time=33364ms

2016.01.06 20:31:14 INFO  [o.s.s.c.s.ComputationStepExecutor] Index tests | time=228ms

2016.01.06 20:31:14 INFO  [o.s.s.c.s.ComputationStepExecutor] Send issue notifications | time=5ms

2016.01.06 20:31:14 INFO  [o.s.s.c.s.ComputationStepExecutor] Publish task results | time=0ms

2016.01.06 20:31:33 INFO  [o.s.s.c.t.CeWorkerCallableImpl] Executed task | project=28f189d7 | id=AVIYmIp7WErocsgvSCuZ | time=697286ms

Therefore the data migration process or the database state appear the most likely causes. 

* The migration process performed from 5.1 to 5.2 is one potential cause. (sonar log of the upgrade shows no errors)

* The second potential cause is that the database is SQL Server 2012 and the collation is Latin1_General_CI_AS and not the required CS AS.

Fixing collation differences in SQL Server databases can't simply be done by changing collation at the database level, existing column collation must be changed which is laborious.  The usual shortcut way requires creating a new database with correct collation and importing all the data into that but the constraints won't migrate. This was attempted on a test database but on start sonarqube reports an exception which seems to indicate it doesn't like the new database:

Caused by: java.sql.BatchUpdateException: Cannot insert the value NULL into column 'id', table 'SAL-DB-REPORT-SITS-3.dbo.properties'; column does not allow nulls. INSERT fails.

at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:1178) ~[sqljdbc41.jar:na]

at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297) ~[commons-dbcp-1.4.jar:1.4]

at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297) ~[commons-dbcp-1.4.jar:1.4]

at org.apache.ibatis.executor.BatchExecutor.doFlushStatements(BatchExecutor.java:103) ~[mybatis-3.2.7.jar:3.2.7]

From this point to change collation (which may not be the cause of the IllegalArgumentException) will likely require a complicated in place change which we are loath to do, but we cannot lose the history.

Any suggestions appreciated.

David

[Julien Lancelot]

Hi David,

If the collation of your database is not case sensitive, then it could explain the "Multiple entries with same key" error.

You'll then have to create a new database with the correct collation and migrate the data.

If you need help for that, you can contact our commercial support that will be able to help you.

Regards,

Julien LANCELOT | SonarSource

http://sonarsource.com

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/0f8e8dae-8a44-47d9-905d-c721cdadc8f7%40googlegroups.com.

[b...@hotmail.com]

Thanks Julien.

We performed the following:

1. Scripted out all database objects with Tasks -> Generate Scripts -> Script entire Database and all Database objects

2. Altered the script with the following 2 changes and ran it to create a new database:

      Change DB name to MY-NEW-DB

      Under USE MASTER added: ALTER DATABASE [MY-NEW-DB] collate Latin1_General_CS_AS

3. Used SQL Compare to compare the old and new database to verify all indexes, constraints, types etc were the same but collation on relevant columns only was changed.

4. Ran Tasks->Import Data ensuring 'Enable Identity Insert' checked. All data transferred to the new case sensitive database correctly.

5. Started SonarQube with new database and verified all ok then re-ran the project analysis.

The problem unfortunately remains with the exception thrown by either the call to the 'lines' metric or the 'accessor's metric depending on the project. The suspicion is that this is therefore not related to case sensitivity otherwise the import should have shown failed records.

We are now looking at the data within the project_measures table as a possible cause.

Do you have any thoughts on this?

With thanks,

David

[David]

Hi,

We reverted the database to a pre 5.2 version (5.1), performed the steps mentioned previously to alter the database collation correctly, imported the data and verified that it ran correctly with 5.1.  All constraints, indexes etc in place during import and DBCC CHECKDB showed all correct. No issues.

We then installed 5.3 and allowed it to upgrade the database and then ran our projects. No errors or warnings during the migration.

We then ran the same code analysis but unfortunately the problem remains.

In 5.3 the exception now has more information as follows:

2016.01.10 13:57:51 ERROR [o.s.s.c.t.CeWorkerCallableImpl] Failed to execute task AVIr0TT71v-aaxAc90Hv
org.sonar.server.computation.component.VisitException: Visit of Component {key=YYY:0127c372:XXX.Application.Service:XXX-BRANCH:Implementations/YYY/ZZZ/CCC,type=DIRECTORY} failed
	at org.sonar.server.computation.component.VisitException.rethrowOrWrap(VisitException.java:44) ~[sonar-server-5.3.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:42) ~[sonar-server-5.3.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:99) ~[sonar-server-5.3.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:55) ~[sonar-server-5.3.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:40) ~[sonar-server-5.3.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:99) ~[sonar-server-5.3.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:55) ~[sonar-server-5.3.jar:na]
	at org.sonar.server.computation.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:40) ~[sonar-server-5.3.jar:na]
	at org.sonar.server.computation.step.ComputeMeasureVariationsStep.execute(ComputeMeasureVariationsStep.java:93) ~[sonar-server-5.3.jar:na]
	at org.sonar.server.computation.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:39) ~[sonar-server-5.3.jar:na]
	at org.sonar.server.computation.taskprocessor.report.ReportTaskProcessor.process(ReportTaskProcessor.java:72) ~[sonar-server-5.3.jar:na]
	at org.sonar.server.computation.taskprocessor.CeWorkerCallableImpl.executeTask(CeWorkerCallableImpl.java:81) [sonar-server-5.3.jar:na]
	at org.sonar.server.computation.taskprocessor.CeWorkerCallableImpl.call(CeWorkerCallableImpl.java:56) [sonar-server-5.3.jar:na]
	at org.sonar.server.computation.taskprocessor.CeWorkerCallableImpl.call(CeWorkerCallableImpl.java:35) [sonar-server-5.3.jar:na]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_66]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_66]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_66]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_66]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_66]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
Caused by: java.lang.IllegalArgumentException: Multiple entries with same key: MeasureKey{metricKey='lines', ruleId=-6253, characteristicId=-6253, developer=null}=org.sonar.db.measure.PastMeasureDto@71c6f620 and MeasureKey{metricKey='lines', ruleId=-6253, characteristicId=-6253, developer=null}=org.sonar.db.measure.PastMeasureDto@804bff2

	at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150) ~[guava-17.0.jar:na]

etc as before.

On other branches it fails on different components (always of type DIRECTORY).

The database contains data for 26 projects with nearly 1 million issues and has been regularly upgraded back since we started with sonar 2 years ago. It has passed through version 4.2->4.3->4.4->4.5->5.0->5.1 successfully and the history of improvements is important to us.

It seems that this exception is not related to the collation of the database but something in the data is causing the problem.  

Suggestions very welcome.

With thanks,

David

On Thursday, January 7, 2016 at 10:19:30 AM UTC, Julien Lancelot wrote:

[Julien Lancelot]

Hi David, 

It seems indeed that the collation issue was not the cause of the duplicated measures (but it's a very good thing to have now a case sensitive database !).

I've added an answer to the Stackoverflow thread in order to help the cleaning of the database : https://stackoverflow.com/questions/34413589/sonarqube-background-tasks-failing-with-illegalargumentexception-for-java/34718883#34718883

Please add some comments to this thread if needed.

Thanks.

Regards,

Julien LANCELOT | SonarSource

http://sonarsource.com

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/ba1d2e4a-56fd-48ab-99e2-f22eefcb1453%40googlegroups.com.

[David]

Thank-you Julien.

We have run the 1st SQL statement in your stack overflow answer for each of the example failures we have but we find no duplicates.  I am guess that this is the underlying statement called in the DepthTraversalTypeAwareCrawler.

2 records for each metric name are found if the HAVING count(*) > 1 is commented out.

I am getting the feeling increasingly that this is looking like a coding error resulting from the disconnect between database and the runner with previous versions of the database.

Note that I don't have the stack overflow credits to comment on there. 

David

[Julien Lancelot]

Hi David,

You should NOT remove the HAVING count(*) > 1, otherwise you'll remove data that are not duplicated.

If I understand correctly, nothing is returned when the HAVING count(*) > 1 is present ? 

Regards,

Julien LANCELOT | SonarSource

http://sonarsource.com

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/a23cebae-4e84-4bc0-a65a-144956dca1ee%40googlegroups.com.

[David]

Hi Julien,

We are working with a copy of the data and the clause was commented simply to verify it. We never make changes to databases until we have a repeatable process properly rehearsed against copies. Fortunately we have never had a need to alter anything within the SonarQube database given how fantastic a job you guys do.

Yes, that is correct, nothing is returned when that clause is removed. We tried 3 database copies 1) 5.1 operating normally (CI), 2) after upgrade to 5.2 CS (after the analysis failed) and then 3) a 5.1 CI upgraded to 5.3 CS - again after an analysis had failed. Different projects had failed on different files and different metrics e.g. 'lines' and 'accessors' so these components were tested against the metrics noted in the logs.  They always turn up no records.  Further we have two separate environments operating identically and they both produce the same errors making us pretty sure the cause wasn't environmental e.g. during a prior upgrade or a prior analysis.

Our hunch is that the data is not duplicated and that there is a bug in the code somewhere that is not taking into account legacy data (e.g. multiple nulls in previously unused fields).  Will extra tracing, or a build with more logging around the nested code where that exception is thrown help?  We can alter and build from source if necessary.

With thanks,

David

[b...@hotmail.com]

We still have not found a solution to this.  One observation is that the driver is the MS JDBC driver and not the JTDS driver.  When altering the MS driver to use selectMethod=cursor it still fails but on a different metric 'accessor' instead of 'lines' whereas without that change in every test it failed on 'lines'.  Is there a way to revert to the JTDS driver to confirm?

On Tuesday, January 5, 2016 at 11:09:47 PM UTC, David wrote:

[David]

We have resolved this by repeatedly excluding directories using sonar.exclusions until scanning completed successfully.  After that removing the excluded directories from the sonar.exclusions list and running again was successful. 

Analysis takes between 2 and 6 hours on 54 separate projects so it took time, but we got there in the end.

Thanks again for your help. 

David

On Tuesday, January 5, 2016 at 11:09:47 PM UTC, David wrote:

[Julien Lancelot]

Hi David,

I'm happy to know that you've solved your issue.

For your information, but excluding some directories you've purged the history measures of these folders, that's why your issue has disappeared.

So you've solved your issue, and that's a good news, but we have no idea how you get into this situation.

Regards,

Julien LANCELOT | SonarSource

http://sonarsource.com

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/893420fd-fe51-4dcb-984b-1bd198fd237e%40googlegroups.com.

[David]

Thanks Julien,

We haven't altered the database so it could only be via the upgrade script or normal operation. We will keep an eye on it and see if it crops up again.

Perhaps there is more information you might output when type=DIRECTORY.

A similar duplication fault has appeared when attempting to change quality profile on an (upgraded) project.  Changing profiles on a new project works ok, but the upgraded ones do not. 

Message reads:

com.microsoft.sqlserver.jdbc.SQLServerException: Cannot insert duplicate key row in object 'db_owner.project_qprofiles' with unique index 'uniq_project_qprofiles'. The duplicate key value is (fd3a4483-2b4a-4eae-a41f-e4772fc44906, js-XXX-01568)

I will make a new posting if I can't find a previous posting on this. Fortunately this time we can see the entry.

Thanks again.

[Julien Lancelot]

Hi David,

This issue looks to be a collation issue.

Are you still working on a case sensitive database ?

Regards

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/5ae1ef5c-14e9-4a20-8d6d-e4bd0aab1917%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

[David]

Hi Julien,

It seems that it is a collation issue of legacy data.  This is a 5.3 database that is now case sensitive (db and column level). A copy of the 5.1 database was created from script with the case explicitly set to CS_AS at the start of the script, then the data imported and then run through the 5.3 upgrade process, so we are certain that it is now completely case sensitive. Analysis has been run on all 50+ projects for the last week repeatedly.  However it appears it is still tripped by legacy data in some tables (that may have been inserted when it wasn't case sensitive). As mentioned changing profile on any new projects works ok.

The message is:

### Error updating database.  Cause: com.microsoft.sqlserver.jdbc.SQLServerException: Cannot insert duplicate key row in object 'db_owner.project_qprofiles' with unique index 'uniq_project_qprofiles'. The duplicate key value is (fd3a4483-2b4a-4eae-a41f-e4772fc44906, css-heal-86335).

### The error may involve defaultParameterMap

### The error occurred while setting parameters

### SQL: UPDATE project_qprofiles SET profile_key=? WHERE project_uuid=?

### Cause: com.microsoft.sqlserver.jdbc.SQLServerException: Cannot insert duplicate key row in object 'db_owner.project_qprofiles' with unique index 'uniq_project_qprofiles'. The duplicate key value is (fd3a4483-2b4a-4eae-a41f-e4772fc44906, css-heal-86335).

The table contains the following for that project_uuid:

71 fd3a4483-2b4a-4eae-a41f-e4772fc44906 css-company-base-enterprise-quality-profile-33548

72 fd3a4483-2b4a-4eae-a41f-e4772fc44906 xml-company-base-enterprise-quality-profile-87055

73 fd3a4483-2b4a-4eae-a41f-e4772fc44906 java-company-base-strategic-without-pmd-checkstyle-28133

74 fd3a4483-2b4a-4eae-a41f-e4772fc44906 js-company-base-enterprise-quality-profile-16612

75 fd3a4483-2b4a-4eae-a41f-e4772fc44906 web-company-base-enterprise-quality-profile-27119

I suppose the best way to clean this up (using the application itself) would be to export all profiles, reimport them and reassign them to the various projects? We have a few test instances we can experiment on to test that theory...

David

[David]

Correction. It does happen with new (since upgrade) projects assigned old (before upgrade) quality profiles and vice versa.

On a test instance...

Exporting the profiles from one language and restoring them didn't work.

Exporting all profiles, shutting down, deleting /temp and /data and starting again and importing all profiles then reassigning to projects did work.  Note that deleting /temp and /data may (or may not) be needed.

Thanks,

David

[Julien Lancelot]

Hi David,

I'm happy to see that your issue is fixed. This collation issue is really a hard one !

Do you remind what was the issue when exporting and restoring profile from one language ?

Regards,

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/2e780606-dda6-471f-909c-d065a1333975%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[David]

By 'didn't work' I meant, that it didn't solve the problem.  Export and import operated as expected.

We did find another method that worked though:

Changing the profile through the project administration page (droplist + update button) was the method by which it threw the exception. Note that it failed when attempting to change the project away from the Default profile. But, by using the new QP administration pages, changing the default profile to another profile then deselecting the errant projects from that original default profile appeared to work; but only after a long wait.  The length of the wait gave us concern that it had failed and swallowed an exception, however it did seem to work as it was then possible to allocate profiles at will.

With thanks

David

[ahlo...@gmail.com]

Hi David, Julien,

I recently upgraded to 5.3 and ran into the original reported problem as well. I tried the queries in this StackOverflow post but it didn't return any records. I also tried switching quality profiles and that did not solve it for me as well. In the end, I had to exclude the directory reported from analysis. That did the trick. The analysis can be completed successfully. However, upon drilling into the components of the SonarQube project, there are now 2 directories with same name. These 2 directories have the same name as the one reported. In each folder there are different files but all files are supposed to be in the same physical folder.

Is there away for me to fix this problem so that all files in the same folder are shown in together?

Regards,

Kah Loon

[David]

Hello Kah Loon,

We have not seen this particular behaviour. if you exclude the path and it's parent (if the loss of history is not too costly to you) does it fix the trouble.

We excluded, ran once, then included again and the problem went away (but so did the history of the excluded items.

Also, was (or is) your database case insensitive?  

I am really not convinced that the root cause of thsi problem has anything to do with the case sensitivity of the database.

Best regards,

David

[ahlo...@gmail.com]

Hi David,

My database is still case insensitive.

I excluded the entire path by adding this to my exclusion list: - **/SubFolderName/Subfoldername/**.

I will try to exclude the parent as well. It will be a shame to lose history but if it means a cleaner and more accurate representation of the actual file/folder structure, then i guess it has to be done.

Thank you.

Regards,

Kah Loon

[David]

You should make a copy of the database and test if the suggestion works, it is only a theory and may not solve the problem at all.

Making a copy of the database and converting the copy to be case sensitive and testing the impact would seem to be a good idea.

We did that as suggested by Julien at SonarSource and have not had further trouble of this sort since.

David

[ahlo...@gmail.com]

hi David,

I will try that as well. Thank you.

Regards,

Kah Loon