SonarQube MSBuild Scanner on macOS with mono and MSBuild 15

[benjami...@gmail.com]

Hi all,

this is my first time posting to this group so .. "thanks for having me" :-)

I'm working as tech lead in a mobile agency and would love to implement SonarQube in our CI process to get more insights about the iOS and Android apps that we produce. Besides creating the occasional native app based on ObjC/Swift or Java, we heavily work using Xamarin, a technology which allows you to write apps for iOS and Android using C# and mono. As we are developing plenty of iOS apps, our CI infrastructure is based entirely on Mac Minis with macOS on which I try to get the SonarQube MSBuild Scanner to work.

Following the steps described in the MSBuild Scanner's documentation (https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner+for+MSBuild) and adjusting them so that mono is executing the scanner's executable, I noticed the following:

1. Running the begin task of the Scanner using mono does create the ".sonarqube" directory within the project directory. It does not seem to modify the .csproj files referenced in the solution to add the Scanner's build targets. I've added these manually by inserting 

<Import Project="<some-folder>/sonar-scanner-msbuild-2/Targets/SonarQube.Integration.ImportBefore.targets" />

2. Once the targets have been included, I can run MSBuild 15 (which is now shipped with mono on macOS) using the following command and can successfully build the solution

msbuild /t:Rebuild /p:CscToolExe=csc.exe /p:CscToolPath=/Library/Frameworks/Mono.framework/Versions/Current/lib/mono/msbuild/15.0/bin/Roslyn/

Currently, it seems to be necessary to explicitly change the underlying compiler using MSBuild command line flags which has been suggested in a response by someone from the MSBuild team on Github (https://github.com/Microsoft/msbuild/issues/1956).

3. The end task of the Scanner seems to understand that MSBuild has created the expected data. Before finding out about the CscToolExe and CscToolPath flags, any invocation of MSBuild and subsequent invocation of the end task led to the end task complaining that no build had taken place. When using the CscToolExe and CscToolPath flags, however, the end task starts and fails with a different result

BB-MBP:SomeProject bb$ mono ~/Downloads/sonar-scanner-msbuild-2/SonarQube.Scanner.MSBuild.exe end

SonarQube Scanner for MSBuild 2.2

Default properties file was found at /Users/bb/Downloads/sonar-scanner-msbuild-2/SonarQube.Analysis.xml

Loading analysis properties from /Users/bb/Downloads/sonar-scanner-msbuild-2/SonarQube.Analysis.xml

Post-processing started.

SonarQube Scanner for MSBuild 2.2

WARNING: File is not under the project directory and cannot currently be analysed by SonarQube. File: /var/folders/pb/rf1yd2qs1cd2srkcrrffhmr0000101/T/.NETPortable,Version=v4.5,Profile=Profile111.AssemblyAttributes.cs, project: /Users/bb/Desktop/SomeProject/SomeProject/SomeProject.csproj

SONAR_SCANNER_OPTS is not configured. Setting it to the default value of -Xmx1024m

Calling the SonarQube Scanner...

Execution failed. The specified executable does not exist: /Users/bb/Downloads/sonar-scanner-msbuild-2/sonar-scanner-2.8\bin\sonar-scanner.bat

The SonarQube Scanner did not complete successfully

05:55:53.303  Creating a summary markdown file...

05:55:53.306  Post-processing failed. Exit code: 1

The file it complains about (sonar-scanner.bat) does, of course, exist but is not executable on macOS.

If I understand this correctly, MSBuild Scanner is very close to working on macOS using mono and MSBuild 15. It seems as if the only necessary change would be to invoke the "regular" scanner's binary in a platform-specific way, using "sonar-scanner-msbuild-2/sonar-scanner-2.8/bin/sonar-scanner" on macOS and "sonar-scanner-msbuild-2/sonar-scanner-2.8/bin/sonar-scanner.bat" on Windows.

Do you agree with my findings and what would you recommend for moving ahead on this? Is this something someone in the group can help with? I was also thinking about investigating the Scanner's source code and trying to apply this fix. Since I'm not very knowledgable about its source or the way you integrate with MSBuild, I wouldn't know where to start, though. Furthermore, I will have to see whether I can setup a system that would be able to build the Scanner from source.

I'm very much looking forward to your feedback on this!

Cheers, Ben

[yvesde...@gmail.com]

I am also very interested in this. I'm integrating SonarQube in our continuous integration process for Xamarin, and we are also working with Mac Minis as build servers. I got it working on my local Windows PC, but ran into the same issue as Ben on MacOS..

Any help with this would be greatly appreciated!

Kind regards

Yves

Op donderdag 13 april 2017 06:20:33 UTC+2 schreef benjami...@gmail.com:

[Amaury Leve]

Hi guys,

We have no plan to work on this feature for the short-term but I still have created a ticket about this. Feel free to up-vote it as we are always paying attention to our community.

Cheers,

DotNet Team

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/2eb7451f-473e-4636-a65d-65158240b30b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Amaury LEVÉ | SonarSource

Software Developer - .Net Team

http://sonarsource.com

Are you using SonarLint in your IDE? 

[vu...@affinipay.com]

That is disappointing. I've been evaluating SonarQube for our organization but it looks like we'll have to choose another solution as .NET project support is critical but we have a mono build pipeline.

Looking forward to future support!

[yvesde...@gmail.com]

That's very unfortunate, it would have made the integration of SonarQube a lot easier. We are already using SonarQube for our other projects (native Android, iOS, web), so I would have liked to align this. 

Op donderdag 20 april 2017 14:34:45 UTC+2 schreef Amaury Leve:

[joaoa...@gmail.com]

I was just trying exactly the same on my CI server until I got the sonar-scanner.bat issue... :(

Any updates on this?

The Xamarin community is massive now and most of us along with the company web technologies, Sonarqube in this case. Please add support as it doesn't seem to be really a massive change, just porting the bat file to run on a mac.

Thanks.

[Julien HENRY]

Hi guys,

One way to help this topic moving forward is to contribute a pull request.

If the only issue is when invoking the underlying Java scanner, the fix should "just" be to call the .sh script instead of the .bat when the current platform is not Windows:

https://github.com/SonarSource/sonar-scanner-msbuild/blob/master/SonarScanner.Shim/SonarScanner.Wrapper.cs#L120

The fix should be pretty easy, but we miss time to setup a test environment.

++

Julien

[Benjamin Broll]

Dear Julien,

I would be willing to give this a try and I also have a system which I can use to test whether that modification works. However, I do not have the ability to build the scanner from source as I don't have all the stated requirements available to me. Building on VS4Mac fails (which was to be expected, I guess).

If you or anybody else could provide me with a build which uses the ".sh" instead of the ".bat" for invoking the Java scanner, I'd be more than happy to take it for a test run.

Cheers, Ben

[Amaury Leve]

Hi Benjamin,

I have created a beta version of the scanner for msbuild calling the .sh script when not on a windows machine. Could you test it out and make us some feedback?

https://github.com/SonarSource/sonar-scanner-msbuild/releases/tag/3.1-beta.1

Cheers,

Amaury

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/CAEZqhZpbFpjcR0qscOf5t6g9xtwY4AUkqOzbdDbWrW82Dv7d_A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Amaury Levé | SonarSource

Software Developer - .Net Team

http://sonarsource.com

[Benjamin Broll]

Hi Amaury,

many thanks, I'll give it a try right away!

Cheers, Ben

[Benjamin Broll]

Hi Amaury,

thanks again for the beta build!

I've just tested with a newly created project. There are two things which I notice:

1. When invoking "SonarQube.Scanner.MSBuild.exe end", the scanner complains that it cannot find "sonar-scanner.sh". Indeed, the beta version does not contain "sonar-scanner.sh" but a file called "sonar-scanner". If I rename that file to "sonar-scanner.sh", the step proceeds.

2. Once the first item was fixed, the following output was written to terminal:

SonarQube Scanner for MSBuild 3.1

Default properties file was found at /Users/bb/Downloads/sonar-scanner-msbuild-3/SonarQube.Analysis.xml

Loading analysis properties from /Users/bb/Downloads/sonar-scanner-msbuild-3/SonarQube.Analysis.xml

Post-processing started.

SONAR_SCANNER_OPTS is not configured. Setting it to the default value of -Xmx1024m

Calling the SonarQube Scanner...

INFO: 

INFO: usage: sonar-scanner [options]

INFO: 

INFO: Options:

INFO:  -D,--define <arg>     Define property

INFO:  -h,--help             Display help information

INFO:  -v,--version          Display version information

INFO:  -X,--debug            Produce execution debug output

ERROR: Unrecognized option: ^^^^^^-^^^D^^^p^^^r^^^o^^^j^^^e^^^c^^^t^^^.^^^s^^^e^^^t^^^t^^^i^^^n^^^g^^^s^^^=^^^/^^^U^^^s^^^e^^^r^^^s^^^/^^^b^^^b^^^/^^^D^^^e^^^s^^^k^^^t^^^o^^^p^^^/^^^S^^^o^^^n^^^a^^^r^^^Q^^^u^^^b^^^e^^^T^^^e^^^s^^^t^^^/^^^.^^^s^^^o^^^n^^^a^^^r^^^q^^^u^^^b^^^e^^^/^^^o^^^u^^^t^^^/^^^s^^^o^^^n^^^a^^^r^^^-^^^p^^^r^^^o^^^j^^^e^^^c^^^t^^^.^^^p^^^r^^^o^^^p^^^e^^^r^^^t^^^i^^^e^^^s^^^

The SonarQube Scanner did not complete successfully

14:47:39.241  Creating a summary markdown file...

14:47:39.247  Post-processing failed. Exit code: 1

It looks like the Java scanner was properly invoked (the help text is displayed) but that there are some encoding issues (I suppose) when passing the command line arguments to the process.

We don't seem to be far from getting it to work .. do you have any idea what we could do regarding the encoding problem?

I'm very much looking forward to your feedback! If there is anything else that I can do / test, please let me know.

Cheers, Ben

[Amaury Leve]

Hi Ben,

Thanks for the quick update! I won't be able to provide you with a new scanner to test today as I have a couple of other tasks to finish but I will try to do it early next week.

Cheers,

Amaury

[Benjamin Broll]

Hi Amaury,

no problem, I'm glad that you are able to support this at all :-)

As soon as you send over a new build I can give it another try. If you need some more infos, just let me know!

Cheers and enjoy the weekend,

Ben

[Benjamin Broll]

Hi Amaury,

have you, by any chance, been able to look into this again? If I can provide any additional information that would help, do let me know!

Cheers, Ben

[fuse...@gmail.com]

Hello guys!

I'm also working with OSX & SonarQube.

The thing is the programs always thinks it has been run via batch script and adds escape symbols to the project.settings path.

I've changed SonarScannerWrapper.cs:149 line to "ProcessRunnerArguments scannerArgs = new ProcessRunnerArguments(exeFileName, false, logger)", and it starts successfully.

But I faced to another problem. It can't start SonarLint.Runner.exe.

This is the log:

ERROR: Error during SonarQube Scanner execution

org.sonar.api.utils.command.CommandException: java.io.IOException: Cannot run program "/Users/obivan/Development/Source/Dev/ConsoleProject/ConsoleProject/.sonarqube/out/.sonar/SonarLint.Runner/SonarLint.Runner.exe": error=13, Permission denied

at org.sonar.api.utils.command.CommandExecutor.execute(CommandExecutor.java:108)

at org.sonar.plugins.csharp.CSharpSensor.analyze(CSharpSensor.java:147)

at org.sonar.plugins.csharp.CSharpSensor.analyse(CSharpSensor.java:108)

at org.sonar.batch.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:58)

at org.sonar.batch.phases.SensorsExecutor.execute(SensorsExecutor.java:50)

at org.sonar.batch.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:83)

at org.sonar.batch.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:192)

at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)

at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)

at org.sonar.batch.scan.ProjectScanContainer.scan(ProjectScanContainer.java:241)

at org.sonar.batch.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:236)

at org.sonar.batch.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:234)

at org.sonar.batch.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:226)

at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)

at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)

at org.sonar.batch.task.ScanTask.execute(ScanTask.java:47)

at org.sonar.batch.task.TaskContainer.doAfterStart(TaskContainer.java:86)

at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:142)

at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:127)

at org.sonar.batch.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:106)

at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:119)

at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:63)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)

at com.sun.proxy.$Proxy0.execute(Unknown Source)

at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:233)

at org.sonarsource.scanner.api.EmbeddedScanner.runAnalysis(EmbeddedScanner.java:151)

at org.sonarsource.scanner.cli.Main.runAnalysis(Main.java:123)

at org.sonarsource.scanner.cli.Main.execute(Main.java:77)

at org.sonarsource.scanner.cli.Main.main(Main.java:61)

Caused by: java.io.IOException: Cannot run program "/Users/obivan/Development/Source/Dev/ConsoleProject/ConsoleProject/.sonarqube/out/.sonar/SonarLint.Runner/SonarLint.Runner.exe": error=13, Permission denied

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)

at org.sonar.api.utils.command.CommandExecutor.execute(CommandExecutor.java:74)

... 32 more

Caused by: java.io.IOException: error=13, Permission denied

at java.lang.UNIXProcess.forkAndExec(Native Method)

at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)

at java.lang.ProcessImpl.start(ProcessImpl.java:134)

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)

... 33 more

ERROR: 

ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging.

The SonarQube Scanner did not complete successfully

10:52:44.451  Creating a summary markdown file...

10:52:44.457  Post-processing failed. Exit code: 1

[Amaury Leve]

Hi guys,

I am really sorry but I have had no time to make more progress on this topic and I don't think I will have some on the upcoming week.

Just to let you know that .Net Core support is really high on our top priorities and we will try to have some sprint on it soon.

Cheers,

Amaury

--

You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/529bdbc9-c72a-4ee5-b893-2a1304e1d179%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[wil...@livelikevr.com]

Nice to hear you will be working on it in a future sprint Amaury.

We are looking forward to it, this will simplify the static analysis of our Unity project.

[Lior Tal]

joining in since i'm also interested in support for running the MSBuild scanner on mac OS.

what's the current status? are all the different components open sourced, so that i can have a look to patch things up myself? if so, why do we need someone to build a scanner instead of building it ourselves for testing?

-Lior

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/5092b6a7-6cc4-4d9a-880b-d4eee93bef43%40googlegroups.com.

[Amaury Leve]

Hi Lior,

We have started to work on the .Net Core feature but we know this is going to take us quite some time (couple of sprints) before we get to a full support.

Yes the scanner for msbuild and the vsts task are open sourced and we are welcoming all PRs (and suggestions). Some people were asking for a binary to test so we provided one but you can freely rebuild and test on your machine (if anything is unclear please shout).

Cheers,

Amaury

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/5092b6a7-6cc4-4d9a-880b-d4eee93bef43%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/CA%2Bwcyj9u9pMXbxuXr2M%2Btk4MCr-Z9SoEQdkD-_%3D932ybtJPU9Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Lior Tal]

Thanks Amaury. Last question - if i want to build a Mac OS compatible version, is there a specific branch i should be using?

Also, i used the built "beta" scanner from this release: https://github.com/SonarSource/sonar-scanner-msbuild/releases/tag/3.1-beta.1

Running using this version fails with this error:

Execution failed. The specified executable does not exist: /Users/liortal/Library/sonar-scanner-msbuild-3.1.0.665/sonar-scanner-3.0.3.778/bin/sonar-scanner.sh

The reason is that while there's a .bat file (sonar-scanner.bat), there's a binary for mac (sonar-scanner) and there is no file named sonar-scanner.sh:

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/5092b6a7-6cc4-4d9a-880b-d4eee93bef43%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/CA%2Bwcyj9u9pMXbxuXr2M%2Btk4MCr-Z9SoEQdkD-_%3D932ybtJPU9Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[fuse...@gmail.com]

There is .sh file actually. Just add .sh extension to sonar-runner and sonar-scanner files.

[michal....@sonarsource.com]

Hi,

It took some time, but we have good news for you.

We're proud to say that the next version of Scanner will work on Linux and Mac out of the box (for both .NET Framework and .NET Core).

We are going to relase the RC it early next week. The RC period will be a bit longer than usual to listen to you feedback. Any help with testing will be appreciated!

Thanks,

[Benjamin Broll]

Hi Michal,

That’s perfect news, thank you so much. I will give it a spin as soon as it’s available and will surely post my findings!

Have nice weekend!

Cheers, Ben

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/wLaU3ijlqDk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/baa46456-eeaf-46ae-bc85-512bfe1a1620%40googlegroups.com.

[Benjamin Broll]

Hi Michal,

For looking into the new version, where do I start exactly? In the past, I've used the download option from the following page:

https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner+for+MSBuild

The page shows a last update on Nov 02 and the version that it references is SonarQube Scanner for MSBuild 3.0.2.656.

Is that the version which I should be testing or where would I be able to get hold of a suitable version?

Cheers, Ben

[Amaury Leve]

Hi guys,

We have hit a couple of issues while doing some more tests. We are fixing them right now and will keep you in touch with the download link of the RC.

Cheers,

Amaury

You received this message because you are subscribed to the Google Groups "SonarQube" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/CAEZqhZrtbQAR%3D0UoAmyTgUjkB9L-XfqTnmXvJcP2tAP1kxutQA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Benjamin Broll]

Got it, thanks for the update!

Cheers, Ben

[michal....@sonarsource.com]

Hi Ben, everyone,

It's not quite early this week, but the RFF is here! See the announcement emails, together with links:
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/sonarqube/Ojez_30yoE0
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/sonarqube/sf8ZO8Cm-kE
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/sonarqube/an9BUr74RMc

We'll update the guide shortly: https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner+for+MSBuild

Happy testing. All feedback is very welcomed!

Thanks,
Michal

[Benjamin Broll]

Hi Michal,

thanks so much for the info!

I can confirm that I have successfully analyzed a first project on macOS :-) Will play around some more, testing with different projects of different sizes. Is there anything in particular that you would like to receive feedback on?

Cheers, Ben

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/9a40739e-3af2-4636-96c9-e839255102bb%40googlegroups.com.

[Amaury Leve]

Hi Benjamin,

That's good news. We are not looking for anything in particular. We have tested quite a lot of cases but it is hard to think of all use cases people can came up with so we wanted to be sure we are covering most (all?) of them.

FYI if you are using the scanner with the RC of SonarC# you'll be able to see that we merge issues from the different targets (if you use them). Regarding the metrics, for now we simply take the first one we found (not ideal but this is a first step).

Cheers,
Amaury

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/CAEZqhZpyNOR6t-ZLn1qzwovW16072x8rRecKybXv9P8xEVZk7A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Benjamin Broll]

Hi Amaury,

ok, got it! I have now used the scanner with several different projects and it has been working just fine for now.

Since I'm now a new SonarQube user, I will have to familiarize myself a little bit with the output and with working in SonarQube in general. If I happen to encounter anything, I will definitely be in touch again.

Many thanks for looking into this :-)

Cheers, Ben