Python Scanning Hangs with Alpine Linux Docker Image

[eth...@gmail.com]

I installed 2 docker images:

1) sonarqube:latest (it installs version 7.1 on java:8-alpine)

2) java:8-alpine image with the 3.1.0.1141 version of the sonar-scanner-cli (I tried also with version 3.0.3.778 and even with ubuntu:latest and the linux version of the cli-scanner with the built-in jre)

When running the scan on a java file - it works great. When running on a python file it hangs at:

INFO: Quality profile for py: Sonar way

INFO: Sensor Python Squid Sensor [python]

Anyone else have this issue?

[Pierre-Yves Nicolas]

Hi,

That's weird.

Can you share the python file which you try to analyse?

Do you have the same problem with a very small and simple python file?

Pierre-Yves

[eth...@gmail.com]

Hi,

It even hangs with a single line:

print "Hello World"

Indeed weird.

Running with -X doesn't help much:

12:09:48.937 DEBUG: Sensors : Python Squid Sensor -> PythonXUnitSensor -> SonarJavaXmlFileSensor -> Zero Coverage Sensor -> CPD Block Indexer

12:09:48.938 INFO: Sensor Python Squid Sensor [python]

12:09:48.969 DEBUG: 'src/example.py' generated metadata  with charset 'UTF-8'

12:09:49.108 DEBUG: Not enough content in 'src/example.py' to have CPD blocks, it will not be part of the duplication detection

12:09:49.187 DEBUG: Using pattern 'coverage-reports/*coverage-*.xml' to find reports

Ethan

[Pierre-Yves Nicolas]

Does it hang after the last DEBUG line you sent?

Can you try setting "sonar.python.coverage.reportPath" to a simple non-existent path with no wildcard, e.g. "coverage.xml"?

[eth...@gmail.com]

Yes, it hangs after that DEBUG line (of course, only when using -X, otherwise it hangs after the last INFO: Sensor Python Squid...).

But, if I scan a file that isn't written correctly like the following - then it ERRORs on that and then hangs in the same place.

New example.py:

def foo():

print "Hello World"

End of debug (and where it hangs):

17:41:34.318 INFO: Sensor Python Squid Sensor [python]
17:41:34.353 DEBUG: 'src/example.py' generated metadata  with charset 'UTF-8'
17:41:34.373 ERROR: Unable to parse file: /src/example.py
17:41:34.373 ERROR: Parse error at line 2 column 0:


    1: def foo():
  -->  print "Hello World"
    3: EOF


17:41:34.495 DEBUG: Using pattern '/sdfsf/cov.xml' to find reports

So, it seems that it is parsing the .py file - but getting stuck after. This is the version I'm using (from the start of the output):

INFO: SonarQube Scanner 3.1.0.1141

INFO: Java 1.8.0_111-internal Oracle Corporation (64-bit)

INFO: Linux 4.9.87-linuxkit-aufs amd64

INFO: User cache: /root/.sonar/cache

INFO: SonarQube server 7.1.0

Ethan

[Pierre-Yves Nicolas]

Please try setting "sonar.python.coverage.reportPath" to a simple non-existent path with no wildcard, e.g. "coverage.xml".

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/8vJUpFga-vc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/3a48f1f1-e2a4-4c2a-989d-09003c356d9c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[eth...@gmail.com]

Hi Pierre,

Anything else you want me to try? Any debug to export?

Thanks,

Ethan

On Tuesday, May 15, 2018 at 9:07:42 AM UTC+3, Ethan Schorer wrote:

I did - and it failed as I wrote above.

On Tue, May 15, 2018 at 9:04 AM Pierre-Yves Nicolas  wrote:

Please try setting "sonar.python.coverage.reportPath" to a simple non-existent path with no wildcard, e.g. "coverage.xml".

[Pierre-Yves Nicolas]

Hi Ethan,

Can you please provide the full logs of the analysis?

Thanks,

Pierre-Yves

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/8vJUpFga-vc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/9bc05e5a-3479-4a91-a32f-122bf77d7ecf%40googlegroups.com.

[eth...@gmail.com]

Python file:

def foo():

print "Hello World"

Debug:

12:46:48.955 INFO: Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties

12:46:48.967 INFO: Project root configuration file: NONE

12:46:49.031 INFO: SonarQube Scanner 3.1.0.1141

12:46:49.031 INFO: Java 1.8.0_111-internal Oracle Corporation (64-bit)

12:46:49.032 INFO: Linux 4.9.87-linuxkit-aufs amd64

12:46:49.439 DEBUG: keyStore is : 

12:46:49.439 DEBUG: keyStore type is : jks

12:46:49.439 DEBUG: keyStore provider is : 

12:46:49.440 DEBUG: init keystore

12:46:49.441 DEBUG: init keymanager of type SunX509

12:46:49.549 DEBUG: Create: /root/.sonar/cache

12:46:49.552 INFO: User cache: /root/.sonar/cache

12:46:49.552 DEBUG: Create: /root/.sonar/cache/_tmp

12:46:49.556 DEBUG: Extract sonar-scanner-api-batch in temp...

12:46:49.574 DEBUG: Get bootstrap index...

12:46:49.575 DEBUG: Download: http://sonarqube:9000/batch/index

12:46:49.647 DEBUG: Get bootstrap completed

12:46:49.677 DEBUG: Download http://sonarqube:9000/batch/file?name=sonar-scanner-engine-shaded-7.1-all.jar to /root/.sonar/cache/_tmp/fileCache1892982735338489899.tmp

12:46:50.033 DEBUG: Create isolated classloader...

12:46:50.048 DEBUG: Start temp cleaning...

12:46:50.063 DEBUG: Temp cleaning done

12:46:50.064 DEBUG: Execution getVersion

12:46:50.067 INFO: SonarQube server 7.1.0

12:46:50.069 INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)

12:46:50.070 DEBUG: Work directory: /.scannerwork

12:46:50.072 DEBUG: Execution execute

12:46:50.430 INFO: Publish mode

12:46:50.625 INFO: Load global settings

12:46:50.686 DEBUG: GET 200 http://sonarqube:9000/api/settings/values.protobuf | time=49ms

12:46:50.722 INFO: Load global settings (done) | time=99ms

12:46:50.733 INFO: Server id: AWNZC8elhD-r9REyobfY

12:46:50.749 INFO: User cache: /root/.sonar/cache

12:46:51.016 INFO: Load plugins index

12:46:51.026 DEBUG: GET 200 http://sonarqube:9000/api/plugins/installed | time=9ms

12:46:51.092 INFO: Load plugins index (done) | time=77ms

12:46:51.092 INFO: Load/download plugins

12:46:51.093 DEBUG: Download plugin 'sonar-scm-git-plugin-1.4.0.1037.jar' to '/root/.sonar/cache/_tmp/fileCache7738206173165278656.tmp'

12:46:51.096 DEBUG: GET 200 http://sonarqube:9000/deploy/plugins/scmgit/sonar-scm-git-plugin-1.4.0.1037.jar | time=2ms

12:46:51.153 DEBUG: Download plugin 'sonar-csharp-plugin-7.0.1.4822.jar' to '/root/.sonar/cache/_tmp/fileCache55735514133784293.tmp'

12:46:51.156 DEBUG: GET 200 http://sonarqube:9000/deploy/plugins/csharp/sonar-csharp-plugin-7.0.1.4822.jar | time=3ms

12:46:51.238 DEBUG: Download plugin 'sonar-flex-plugin-2.4.0.1222.jar' to '/root/.sonar/cache/_tmp/fileCache3476961560261342545.tmp'

12:46:51.246 DEBUG: GET 200 http://sonarqube:9000/deploy/plugins/flex/sonar-flex-plugin-2.4.0.1222.jar | time=5ms

12:46:51.269 DEBUG: Download plugin 'sonar-javascript-plugin-4.1.0.6085.jar' to '/root/.sonar/cache/_tmp/fileCache1911277640186726598.tmp'

12:46:51.272 DEBUG: GET 200 http://sonarqube:9000/deploy/plugins/javascript/sonar-javascript-plugin-4.1.0.6085.jar | time=3ms

12:46:51.318 DEBUG: Download plugin 'sonar-java-plugin-5.2.0.13398.jar' to '/root/.sonar/cache/_tmp/fileCache6505288089832400723.tmp'

12:46:51.321 DEBUG: GET 200 http://sonarqube:9000/deploy/plugins/java/sonar-java-plugin-5.2.0.13398.jar | time=3ms

12:46:51.454 DEBUG: Download plugin 'sonar-php-plugin-2.13.0.3107.jar' to '/root/.sonar/cache/_tmp/fileCache2806424604719222805.tmp'

12:46:51.456 DEBUG: GET 200 http://sonarqube:9000/deploy/plugins/php/sonar-php-plugin-2.13.0.3107.jar | time=2ms

12:46:51.489 DEBUG: Download plugin 'sonar-python-plugin-1.9.1.2080.jar' to '/root/.sonar/cache/_tmp/fileCache6788458010303238630.tmp'

12:46:51.492 DEBUG: GET 200 http://sonarqube:9000/deploy/plugins/python/sonar-python-plugin-1.9.1.2080.jar | time=3ms

12:46:51.509 DEBUG: Download plugin 'sonar-typescript-plugin-1.6.0.2388.jar' to '/root/.sonar/cache/_tmp/fileCache6578743597123216325.tmp'

12:46:51.511 DEBUG: GET 200 http://sonarqube:9000/deploy/plugins/typescript/sonar-typescript-plugin-1.6.0.2388.jar | time=2ms

12:46:51.541 DEBUG: Download plugin 'sonar-xml-plugin-1.5.0.1373.jar' to '/root/.sonar/cache/_tmp/fileCache5494212404552390923.tmp'

12:46:51.543 DEBUG: GET 200 http://sonarqube:9000/deploy/plugins/xml/sonar-xml-plugin-1.5.0.1373.jar | time=2ms

12:46:51.571 DEBUG: Download plugin 'sonar-scm-svn-plugin-1.7.0.1017.jar' to '/root/.sonar/cache/_tmp/fileCache3274453344149248140.tmp'

12:46:51.573 DEBUG: GET 200 http://sonarqube:9000/deploy/plugins/scmsvn/sonar-scm-svn-plugin-1.7.0.1017.jar | time=2ms

12:46:51.655 INFO: Load/download plugins (done) | time=563ms

12:46:51.726 DEBUG: Plugins:

12:46:51.727 DEBUG:   * SonarC# 7.0.1.4822 (csharp)

12:46:51.727 DEBUG:   * SonarPython 1.9.1.2080 (python)

12:46:51.728 DEBUG:   * SonarJava 5.2.0.13398 (java)

12:46:51.728 DEBUG:   * Git 1.4.0.1037 (scmgit)

12:46:51.728 DEBUG:   * SonarFlex 2.4.0.1222 (flex)

12:46:51.729 DEBUG:   * SonarXML 1.5.0.1373 (xml)

12:46:51.731 DEBUG:   * SonarPHP 2.13.0.3107 (php)

12:46:51.731 DEBUG:   * SonarTS 1.6.0.2388 (typescript)

12:46:51.731 DEBUG:   * Svn 1.7.0.1017 (scmsvn)

12:46:51.732 DEBUG:   * SonarJS 4.1.0.6085 (javascript)

12:46:52.300 INFO: Process project properties

12:46:52.311 DEBUG: Process project properties (done) | time=11ms

12:46:52.337 INFO: Load project repositories

12:46:52.346 DEBUG: GET 404 http://sonarqube:9000/batch/project.protobuf?key=PyTest | time=7ms

12:46:52.347 DEBUG: Project repository not available - continuing without it

12:46:52.353 INFO: Load project repositories (done) | time=16ms

12:46:52.508 DEBUG: Available languages:

12:46:52.508 DEBUG:   * C# => "cs"

12:46:52.508 DEBUG:   * Python => "py"

12:46:52.508 DEBUG:   * Java => "java"

12:46:52.509 DEBUG:   * Flex => "flex"

12:46:52.509 DEBUG:   * XML => "xml"

12:46:52.509 DEBUG:   * PHP => "php"

12:46:52.510 DEBUG:   * TypeScript => "ts"

12:46:52.510 DEBUG:   * JavaScript => "js"

12:46:52.531 INFO: Load quality profiles

12:46:52.556 DEBUG: GET 200 http://sonarqube:9000/api/qualityprofiles/search.protobuf?defaults=true | time=24ms

12:46:52.570 INFO: Load quality profiles (done) | time=40ms

12:46:52.612 INFO: Load active rules

12:46:52.821 DEBUG: GET 200 http://sonarqube:9000/api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWNZDAg9hD-r9REyob7Z&p=1&ps=500 | time=208ms

12:46:53.434 DEBUG: GET 200 http://sonarqube:9000/api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWNZDAmahD-r9REyocDM&p=1&ps=500 | time=453ms

12:46:53.547 DEBUG: GET 200 http://sonarqube:9000/api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWNZDApphD-r9REyocIn&p=1&ps=500 | time=81ms

12:46:53.573 DEBUG: GET 200 http://sonarqube:9000/api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWNZDAqYhD-r9REyocJY&p=1&ps=500 | time=19ms

12:46:53.688 DEBUG: GET 200 http://sonarqube:9000/api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWNZDAtJhD-r9REyocOt&p=1&ps=500 | time=114ms

12:46:53.819 DEBUG: GET 200 http://sonarqube:9000/api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWNZDAvThD-r9REyocRP&p=1&ps=500 | time=123ms

12:46:53.910 DEBUG: GET 200 http://sonarqube:9000/api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWNZDA2MhD-r9REyocUP&p=1&ps=500 | time=85ms

12:46:53.933 DEBUG: GET 200 http://sonarqube:9000/api/rules/search.protobuf?f=repo,name,severity,lang,internalKey,templateKey,params,actives,createdAt&activation=true&qprofile=AWNZDA8OhD-r9REyocVg&p=1&ps=500 | time=21ms

12:46:53.963 INFO: Load active rules (done) | time=1351ms

12:46:53.967 INFO: Load metrics repository

12:46:54.031 DEBUG: GET 200 http://sonarqube:9000/api/metrics/search?f=name,description,direction,qualitative,custom&ps=500&p=1 | time=64ms

12:46:54.053 INFO: Load metrics repository (done) | time=86ms

12:46:54.084 WARN: SCM provider autodetection failed. No SCM provider claims to support this project. Please use sonar.scm.provider to define SCM of your project.

12:46:54.092 INFO: Project key: PyTest

12:46:54.092 INFO: Project base dir: /

12:46:54.093 DEBUG: Start recursive analysis of project modules

12:46:54.095 INFO: -------------  Scan PyTest

12:46:54.266 INFO: Load server rules

12:46:54.458 DEBUG: GET 200 http://sonarqube:9000/api/rules/list.protobuf | time=191ms

12:46:54.481 INFO: Load server rules (done) | time=215ms

12:46:54.526 INFO: Base dir: /

12:46:54.529 INFO: Working dir: /.scannerwork

12:46:54.537 INFO: Source paths: src

12:46:54.538 INFO: Source encoding: UTF-8, default locale: en_US

12:46:54.709 DEBUG: Declared extensions of language C# were converted to sonar.lang.patterns.cs : **/*.cs

12:46:54.709 DEBUG: Declared extensions of language Python were converted to sonar.lang.patterns.py : **/*.py

12:46:54.709 DEBUG: Declared extensions of language Java were converted to sonar.lang.patterns.java : **/*.java,**/*.jav

12:46:54.710 DEBUG: Declared extensions of language Flex were converted to sonar.lang.patterns.flex : **/*.as

12:46:54.710 DEBUG: Declared extensions of language XML were converted to sonar.lang.patterns.xml : **/*.xml,**/*.xsd,**/*.xsl

12:46:54.710 DEBUG: Declared extensions of language PHP were converted to sonar.lang.patterns.php : **/*.php,**/*.php3,**/*.php4,**/*.php5,**/*.phtml,**/*.inc

12:46:54.711 DEBUG: Declared extensions of language TypeScript were converted to sonar.lang.patterns.ts : **/*.ts,**/*.tsx

12:46:54.711 DEBUG: Declared extensions of language JavaScript were converted to sonar.lang.patterns.js : **/*.js,**/*.jsx,**/*.vue

12:46:54.720 DEBUG: Initializers : 

12:46:54.723 INFO: Index files

12:46:54.752 DEBUG: 'src/example.py' indexed with language 'py'

12:46:54.757 DEBUG: 'src/sonar_scan_log.txt' indexed with language 'null'

12:46:54.758 INFO: 2 files indexed

12:46:54.759 INFO: Quality profile for py: Sonar way

12:46:55.797 DEBUG: 'Generic Coverage Report' skipped because one of the required properties is missing

12:46:55.798 DEBUG: 'Generic Test Executions Report' skipped because one of the required properties is missing

12:46:55.808 DEBUG: 'C#' skipped because there is no related file in current project

12:46:55.808 DEBUG: 'C# Properties' skipped because there is no related file in current project

12:46:55.809 DEBUG: 'C# Tests Coverage Report Import' skipped because there is no related file in current project

12:46:55.809 DEBUG: '[Deprecated] C# Integration Tests Coverage Report Import' skipped because there is no related file in current project

12:46:55.811 DEBUG: 'C# Unit Test Results Import' skipped because there is no related file in current project

12:46:55.812 DEBUG: 'PylintSensor' skipped because there is no related rule activated in the quality profile

12:46:55.813 DEBUG: 'PylintImportSensor' skipped because there is no related rule activated in the quality profile

12:46:55.813 DEBUG: 'SurefireSensor' skipped because there is no related file in current project

12:46:55.813 DEBUG: 'JaCoCoSensor' skipped because there is no related file in current project

12:46:55.814 DEBUG: 'JavaSquidSensor' skipped because there is no related file in current project

12:46:55.814 DEBUG: 'Flex' skipped because there is no related file in current project

12:46:55.815 DEBUG: 'Flex Cobertura' skipped because there is no related file in current project

12:46:55.815 DEBUG: 'XML Sensor' skipped because there is no related file in current project

12:46:55.815 DEBUG: 'PHP sensor' skipped because there is no related file in current project

12:46:55.816 DEBUG: 'Analyzer for "php.ini" files' skipped because there is no related file in current project

12:46:55.816 DEBUG: 'SonarTS' skipped because there is no related file in current project

12:46:55.817 DEBUG: 'SonarTS Coverage' skipped because there is no related file in current project

12:46:55.817 DEBUG: 'SonarJS' skipped because there is no related file in current project

12:46:55.817 DEBUG: 'SonarJS Coverage' skipped because there is no related file in current project

12:46:55.819 DEBUG: 'Generic Coverage Report' skipped because one of the required properties is missing

12:46:55.819 DEBUG: 'Generic Test Executions Report' skipped because one of the required properties is missing

12:46:55.824 DEBUG: 'C#' skipped because there is no related file in current project

12:46:55.825 DEBUG: 'C# Properties' skipped because there is no related file in current project

12:46:55.825 DEBUG: 'C# Tests Coverage Report Import' skipped because there is no related file in current project

12:46:55.826 DEBUG: '[Deprecated] C# Integration Tests Coverage Report Import' skipped because there is no related file in current project

12:46:55.826 DEBUG: 'C# Unit Test Results Import' skipped because there is no related file in current project

12:46:55.827 DEBUG: 'PylintSensor' skipped because there is no related rule activated in the quality profile

12:46:55.827 DEBUG: 'PylintImportSensor' skipped because there is no related rule activated in the quality profile

12:46:55.827 DEBUG: 'SurefireSensor' skipped because there is no related file in current project

12:46:55.827 DEBUG: 'JaCoCoSensor' skipped because there is no related file in current project

12:46:55.828 DEBUG: 'JavaSquidSensor' skipped because there is no related file in current project

12:46:55.828 DEBUG: 'Flex' skipped because there is no related file in current project

12:46:55.828 DEBUG: 'Flex Cobertura' skipped because there is no related file in current project

12:46:55.829 DEBUG: 'XML Sensor' skipped because there is no related file in current project

12:46:55.829 DEBUG: 'PHP sensor' skipped because there is no related file in current project

12:46:55.829 DEBUG: 'Analyzer for "php.ini" files' skipped because there is no related file in current project

12:46:55.830 DEBUG: 'SonarTS' skipped because there is no related file in current project

12:46:55.830 DEBUG: 'SonarTS Coverage' skipped because there is no related file in current project

12:46:55.830 DEBUG: 'SonarJS' skipped because there is no related file in current project

12:46:55.831 DEBUG: 'SonarJS Coverage' skipped because there is no related file in current project

12:46:55.836 DEBUG: Sensors : Python Squid Sensor -> PythonXUnitSensor -> SonarJavaXmlFileSensor -> Zero Coverage Sensor -> CPD Block Indexer

12:46:55.837 INFO: Sensor Python Squid Sensor [python]

12:46:55.862 DEBUG: 'src/example.py' generated metadata  with charset 'UTF-8'

12:46:55.876 ERROR: Unable to parse file: /src/example.py

12:46:55.876 ERROR: Parse error at line 2 column 0:

    1: def foo():

  -->  print "Hello World"

    3: EOF

12:46:55.880 DEBUG: Using pattern 'coverage.xml' to find reports

Thank you,

Ethan

[Pierre-Yves Nicolas]

I think the problem is there:

Project base dir: /

That's a quite unusual configuration, and it makes the Python analyzer iterate over all files in the file system to check their names.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/044ca66b-05d8-4387-8119-49af0819767d%40googlegroups.com.

[eth...@gmail.com]

EUREKA!! I guess the running directory was /

I set it using the variable -Dsonar.projectBaseDir=/src and it works.

But, I'm surprised it didn't even catch a division by zero bug :-( for the code:

def foo(x):

    if x == 0:

        print("Hi")

        y = 3/x

        return y

foo(0)

Ethan

[Pierre-Yves Nicolas]

Unfortunately, SonarPython is not able to catch that kind of bug, and that will not change in the short term.

SonarJava is much more advanced.

Pierre-Yves

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/4a991ab8-7d39-4398-84ea-6fc974185309%40googlegroups.com.

[eth...@gmail.com]

Thank you Pierre. I appreciate all your help.

Best regards,

Ethan