Sonar for the Django rpoject

[Ivan Sevastoyanov]

Hi guys,

I am new to Django and I want to contribute to the project soon. Sorry for the question if it's not appropriate. Do you consider using SonarQube (or something similar) for code quality analysis?

Regards,
Ivan

[Aymeric Augustin]

On 28 Aug 2016, at 20:46, Ivan Sevastoyanov <ivan.sev...@gmail.com> wrote:

> Do you consider using SonarQube (or something similar) for code quality analysis?

Hello Ivan,

Generally speaking, there isn’t a lot of demand for code changes with no impact on functionality, especially as first time contributions. Such patches are tedious to review compared to the value they add. In practice it can be faster for a committer to redo the job that to check that it was done correctly. The coding style tends to improve as a side effect of making other changes in an area.

If SonarQube goes beyond traditional code quality guidelines, for example if it does static analysis and can find bugs with reasonable accuracy, that would be more interesting. In that case you’ll have to tell us a bit more about the kind of results you expect. Many people (including myself) have never heard of SonarQube before and aren’t familiar with what it can do.

I hope this helps,

--
Aymeric.

PS: could you pick a better word than “guys” to address people on this mailing list? Even though “guys” can include people regardless of gender in some cultures, originally “guy” is a synonym for “man”, and you don’t want to imply that you’re only talking to men. Thanks!

[Ivan Sevastoyanov]

My mistake. I had to ask with more details. My question is do you consider using SonarQube for code quality analysis, static analysis and find bugs because it's able to do that. I am asking for the Django project as a whole. Sonar can track the commits and show you if there are added some "code smells". That way we can prioritize some of the findings for fixing in the next releases.

PS: I accept the criticism and won't use guys annymore :)

Regards,
Ivan

[Aymeric Augustin]

On 28 Aug 2016, at 21:43, Ivan Sevastoyanov <ivan.sev...@gmail.com> wrote:

> My question is do you consider using SonarQube for code quality analysis, static analysis and find bugs because it's able to do that.

I guess that depends on the signal / noise ratio in the things SonarQube flags.

Perhaps you could do an initial run and see whether SonarQube spots interesting bugs?

I have no idea what the results could be because I’m not familiar with static analysis of Python code.

--
Aymeric.

[Ivan Sevastoyanov]

OK, I will try to do that on my machine and will post the results here (because frankly speaking I haven't done it before on my own). I don't know when I will have enough time to do it but I guess 3 to 4 days.

Regards,
Ivan

[Ivan Sevastoyanov]

That is the report from the Sonar with all the rules included. Unfortunately, I cannot export it as a PDF or some more convenient format. I can describe all the steps in my blog so some of the Django members could set up Sonar on his/her machine and see a lot more details and figure out if it's worth it to fix some of the issues.

[Tim Graham]

Perhaps you could tell us about some of the critical issues so we could get a sense for that.

[Ivan Sevastoyanov]

All the rules are with a default severity so there might be some major issues that it's worth reviewing them. I will post the critical issues this evening because I'm at work now. Do you want to post them somewhere else because it's a sensitive information? I will try to find out how to export the whole report in a convenient format.

Regards,
Ivan

[Tim Graham]

Any security issues should be reported to secu...@djangoproject.com, otherwise it's fine to share the information here.

[Ivan Sevastoyanov]

I'm posting the the 11 criticals. In my opinion, they are not critical, they are just code smells. I will try to export the report so you can review the major issues by groups.

Regards,
Ivan

[Sergei Maertens]

I kind of like these reports, since they can take away some of the early review work. I would put it on the same level as the `isort` checks we have now. On the other hand, adapting the existing codebase to 'resolve' this code smells will introduce quite some 'stupid' commits, so it might be best to get it done with in one or two go's.

If it can be applied to pull-requests, it would be nice I guess.

One final question: why use sonar instead of something like pylint/pep8 - these tools also provide static analysis and report common violations in the Python world.

[Curtis Maloney]

If there will be sweeping commits to remove six and other py2 concessions, can the cleaning be included then?

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Ivan Sevastoyanov]

Hi,

I'm on a vacation and far from my PC now so it's possible that I'll not be able to answer some of the questions.
@Sergei - Sonar can be applied the same way you have Jenkins. It will be easier to track some issues immediately. Sonar combines all the rules from pylint, pep8 plus some other rules. It's just more convenient.

Regards,
Ivan

[Hanne Moa]

Is there a way to ignore django.utils.dateformat? That code is very straight forward, and it is not supposed to be called manually by humans. I can't see how a "fix" would improve things. Munging the second string in the getattr? Adding "upper" and "lower" or something similar to each method-name?

[Aymeric Augustin]

Hello Ivan,

Given that both Django and Sonar are open-source, anyone should be able to reproduce your results easily… If there are security issues, please email them to secu...@djangoproject.com instead of publishing them. That’ll make them a bit less easy to discover. Otherwise, go ahead and post the issues wherever is most convenient for you.

Thanks,

-- 

Aymeric.

-- 
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f7d07e45-c0a4-4285-9ce8-3605c9885d4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Gaynor]

If these are what qualifies as critical, I don't think this is a good use of our time.

Alex

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7263b3cc-a0b6-4dc6-9a33-204ed3aac9a5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero

GPG Key fingerprint: D1B3 ADC0 E023 8CA6

[James Bennett]

On Wed, Aug 31, 2016 at 10:55 AM, Alex Gaynor <alex....@gmail.com> wrote:

If these are what qualifies as critical, I don't think this is a good use of our time.

Agreed. If those are the critical things, then either Django is really really good, or there are things it's missing. I suspect there are things it's missing. 

[Ivan Sevastoyanov]

Hello,

I'm back from the vacation.

@Hanne Moa - As far as I know, you can skip packages, files and everything can be customized. It's the same with the rules. I did not prioritized the Sonar rules - they are the default ones and Sonar is detecting not only possible bugs and issues but code smells, some ideas for improving the readability and maintainability, etc. So I agree that these "criticals" are, in fact, not real "criticals" - they are not issues, they will not improve the performance, they are just a tip to improve the readability of the code. But you have the full power to customize the rules and choose which of them are blockers, criticals, major, minor and info.

@Aymeric Augustin - Yes, it's easy to reproduce the results. Unfortunately, I installed the latest version of Sonar and some of the plug-ins for exporting into PDF and HTML are still not compatible. I can install some older version and put an old working plug-in into work. But I'm not sure if the rules will be the same or less than now. I will review the rules and will send an e-mail if I think some of them are security issues. Other I can do is to write a blog post how to install SonarQube and some of the plug-ins and how to configure them but I don't know when I will have enough time for doing that.

@Alex Gaynor - You can see what I wrote to Hanne Moa.

@James Bennett - You can see what I wrote to Hanne Moa. The rules should be prioritized but in my honest opinion I'm not the right person for doing that. I can copy/paste the rules here but I'm not sure that some of them are understandable from their short description.

Regards,
Ivan

[Ivan Sevastoyanov]

Hello,

I installed some older versions of SonarQube and unfortunately the rules are not the same and the report generated is not full. But I reviewed the issues and I did not find any security issues or something that is absolutely critical. There are 40 major issues that are marked as bugs. Most common they are of this type - "Having two branches in the same if structure with the same implementation is at best duplicate code, and at worst a coding error. If the same logic is truly needed for both instances, then they should be combined.". So I will write a blog post for setting up SonarQube, sonar-scanner and Python plug-in and post it here. It takes not more than 15 minutes, so you can see the issues yourself.

Regards,
Ivan

[Ivan Sevastoyanov]

Hello,

Here is my blog post about setting up SonarQube. I think it takes about 15 minutes, so you can run it yourself if you want (and if you have time of course). Have a good day!

Regards,
Ivan