GSoC 2017 - Flexible Suppression Model (part 1)

Andrew Selkin

unread,

May 7, 2017, 4:12:29 PM5/7/17

to checkstyle-devel

This thread is a placeholder only for the GSoC 2017 Flexible Suppression Model project.

https://github.com/checkstyle/checkstyle/wiki/Checkstyle-GSoC-2017-Project-Ideas#project-name-flexible-suppression-model

Тимур Тибеев

unread,

Jun 7, 2017, 5:33:51 AM6/7/17

to checkstyle-devel

Current progress

Perform analysis of the current Checkstyle suppression model - Main part of Checkstyle suppression model is suppressing by line and column. Additionally Checkstyle supports suppression by annotations and by markers. Markers can be two types: for one line or for block of code.
Perform analysis of the PMD and Sonar suppression model - PMD is a source code analyzer. PMD supports different types of suppressing code: by annotation, by marker, by regular expressions and by xpath. Sonarqube has suppressions by annotation, by marker and by regular expressions.
Understanding of the current Checkstyle suppression model - In Checkstyle suppressions implemented using different filters: SuppressionFilter, SuppressionWarningsFilter, SuppressionCommentFilter and SuppressWithNearbyCommentFilter. All filters implement “Filter” class, which has only method “boolean accept(AuditEvent event);” AuditEvent is the structural AST unit is used inside filters to check acceptance of the code. Inside "Checker" class, all errors from all modules are collected first, then they are being checked by acceptance using "Filter.accept" method.
Discuss task implementation with mentor - Since current Checkstyle suppression model already includes: markers, annotations and line/column suppressing, there are only possible approaches to implement: XPATH, Regular expression. Both approaches compile one time in the beginning. Since regular expressions sometimes are messy and not clear, XPATH is more suitable for large projects. Another moment, some users can be familiar with PMD suppression model, it will be easier for them to use Checkstyle. Alongside with AST-tree, Xpath-tree will be created. Every node inside Xpath tree will be mapped to corresponding node from AST-tree using "TokenTypes.java" constants. For every xpath query we find corresponding node and suppress element mapped to this node.
Compare JAXEN and SAXON XPath libraries - I have read documentations and created simple applications. Both implementations can work on DOM, XOM, DOM4j and JDOM, but Saxon also supports JDOM2 and AXIOM tree models. Jaxen project is probably not active now, last changes on Git repository were about 2 years ago (https://github.com/jaxen-xpath/jaxenand official site is unavailable (http://jaxen.codehaus.org/, www.jaxen.org/), while Saxon's last release was in 5 April 2017. Jaxen does not support XPATH 2.0, while Saxon does. PMD uses Jaxen for own suppression model. They also allow to select Jaxen/Saxon for writing custom rules. I think Saxon library is more reliable and has more functionalities comparing to Jaxen.

Issues

Pull requests

Design a new format of suppression dtd schema to support XPath queries

Future plans

Design XPath query format - I should suggest format for XPath queries, it can be same as PMD format. Format should be based on constants from "TokenTypes.java"
Write UTs for XPath model
Implement XPath model in code - Implement classes for XPath-tree nodes.
Write initial documentation for XPath query and model

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Jun 10, 2017, 2:21:02 AM6/10/17

to checkstyle-devel

Experimental dtd schema forn the new suppression model was introduced in the scope of the issue #4364 and merged to Checkstyle codebase in PR #4427.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Jun 10, 2017, 3:22:22 AM6/10/17

to checkstyle-devel

Timur, please start working on issue #4369.

Here you can find the basic implementation which is based on the PMD's one. It might be a starting point for you.

Cover the whole new code with UTs (100% coverage is required). For the first step of the implementation, UTs should be the basic source of the documentation;
Extend the implementation with the features described in the issue;
Write javadoc for each class, interface, etc;
Log each step to this thread.

Please, do not try to write the ideal implementation. Send small changes set in each PR. More than one PR can be sent for the issue.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Roman Ivanov

unread,

Jun 12, 2017, 9:55:22 PM6/12/17

to Andrew Selkin, checkstyle-devel

I do not see Monday status report

Тимур Тибеев

unread,

Jun 13, 2017, 4:42:04 PM6/13/17

to checkstyle-devel

Currently implementing basic classes for Xpath tree and unit tests for them

"Node" interface - common interface, that extends "NodeInfo" from saxon library

"AbstractNode" class - abstract class, implements "Node" interface, class contains reference to corresponding element inside AST tree

"ClassDefinition", "MethodDefinition" and etc. - classes describing xpath tree nodes

Steps:

1. Make xpath queries in xpath tree

2. Get corresponding element from AST tree, using xpath queries

3. Start to implement "SuppressionXpathFilter"

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Jun 14, 2017, 10:00:46 AM6/14/17

to checkstyle-devel

Timur, please start working on #3573 in parallel. Your tasks are described in the comment.

Please, provide reports of the current status more frequently.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Тимур Тибеев

unread,

Jun 18, 2017, 5:59:17 AM6/18/17

to checkstyle-devel

1. Uploaded first prototype of mapper to my repository (https://github.com/checkstyle/checkstyle/issues/4369)

Received feedback, working on correcting

2. Started doing Step 1 of https://github.com/checkstyle/checkstyle/issues/3573

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Jun 19, 2017, 5:19:25 AM6/19/17

to checkstyle-devel

Timur, please send PR with your changes for (issue #4369 and issue #3573) to allow us to review the changes. In case you do not follow the plan of the project and we do not see your contribution, you may loose your position in GSoC as a student.

The first evaluation will be opened June 26 and is due before June 30, 16:00 UTC.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Тимур Тибеев

unread,

Jun 19, 2017, 12:32:52 PM6/19/17

to checkstyle-devel

Created two PRs,

https://github.com/checkstyle/checkstyle/pull/4469

Improved implementation, added ability to make query using @Image attribute

Added javadoc

Added references to DetailAst and DetailNode

Finished first step of

https://github.com/checkstyle/checkstyle/issues/3573

https://github.com/checkstyle/checkstyle/pull/4468

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Jun 19, 2017, 1:25:31 PM6/19/17

to checkstyle-devel

Timur, I did the review of two PR you sent (4469 and 4468). Please, look at my comments and fix the problems mentioned.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Jun 20, 2017, 10:25:50 AM6/20/17

to checkstyle-devel

Timur, what is the status of 4469 ? I do not see any fixes.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Тимур Тибеев

unread,

Jun 20, 2017, 1:11:26 PM6/20/17

to checkstyle-devel

Andrew,

Fixed moments you mentioned inside my commit

Currently working on making more complex queries, with several levels

Fixed notes for 4468

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Тимур Тибеев

unread,

Jun 22, 2017, 7:13:38 AM6/22/17

to checkstyle-devel

Updated PR, resolved changes requests

https://github.com/checkstyle/checkstyle/pull/4468

Made new commit to my repository

https://github.com/timurt/checkstyle/commit/9adcce8604a742b16b39a609b3ad6bf829cd91d0

1. Added more UT, with more complex queries

2. Now it is able to iterate whole AST tree and create XPATH tree

Next steps:

1. Add opportunity to query Javadoc elements 4369

2. Work on 4415

3. if Issue 3573 step 1 is accepted and merged, I will work on step 2

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Jun 22, 2017, 11:19:27 AM6/22/17

to checkstyle-devel

Timur, I finished review of PR #4468 and approved it. I also requested the review from your second mentor (rnveach).

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Jun 24, 2017, 3:14:45 PM6/24/17

to checkstyle-devel

Timur, please start working on 3034 as it blocks PR 4468.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Тимур Тибеев

unread,

Jun 25, 2017, 4:46:54 PM6/25/17

to checkstyle-devel

Issue 3034

Pull requested 3034

Issue 4369

My commit 4369

Added context-based queries. Ex. METHOD_DEF[@text='foo']
Resolved problem with //METHOD_DEF[@text='foo'] queries

Next step

Support attribute-axis queries. Ex. *[@text]

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Jun 27, 2017, 4:52:44 PM6/27/17

to checkstyle-devel

SUMMARY (for the first evaluation)

This is the summary of what was done in accordance with the project plan:

The new xsd schema was introduced in the scope of the issue #4364 and merged to the master branch in PR #4427 .
The student started working on the issue #3573 as it blocks the issue #4382. The issue #3573 was not part of GSoC project, but due to the problem in Checkstyle design it became. The student sent PR #4468 to resolve the first step of the issue, but during review we found another problem in Checkstyle design (see issue #3034). Timur sent one more PR (#4525) to resolve the problem. The PR was reviewed and approved by two mentors, but now it is waiting for the regression reports from the student.
The student did a great work to resolve the issue #4369. The issue is one of the main parts of GSoC project. The results are commited to the forked Checkstyle repo. The review of the changes can be found here. As it can be seen from the tests, now Xpath mapper supports the basic xpath expressions. It allows to refer to the nodes in the AST tree nodes and its attributes starting from the root.

The issues which should be resolved in future can be found in the project's dashboard.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Jul 11, 2017, 12:55:18 AM7/11/17

to checkstyle-devel

Timur, please provide Monday status report.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Message has been deleted

Тимур Тибеев

unread,

Jul 11, 2017, 2:36:54 AM7/11/17

to checkstyle-devel

Issue 3034 FileText should not extends AbstractList:

PR merged, issue closed

Issue 4641 Avoid usage of getLines method from FileText:

PR merged, issue closed

Issue 3573 Resolve design problem of FileContentsHolder (step 1);

Issue became active after resolving 3034,

fixed conflicts and points described inside PR,

one thing left: now trying to suppress TeamCity: "Class has many dependencies" violation

Issue 4369 Investigate how to map XPath expressions onto AST nodes:

Added attribute iterator,

Resolved problem with querying symbolic characters like TokenTypes.RCURLY, TokenTypes.LCURLY, TokenTypes.SLIST and etc.

Added UTs to test Attribute query, Parent query, And query and etc

Received review from mentor, working on resolving them

Preparing commit to send PR (mvn clean verify, cobertura coverage)

Future steps:

After finishing first 3573, will start working on step 2

If 4369 will be accepted, I can start working on Xpath filter

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Jul 13, 2017, 5:24:37 PM7/13/17

to checkstyle-devel

Hi, Timur!

Please, see my and romani comments in PR #4468. You need to start working on #4714. In parallel you are able to implement XpathFilter. As TreeWalkerAuditEvent and AstFilter interface will be introduced in #4714, you can create simple XpathFilter which does not implement AstFilter interface and try to implement suppression logic. Take into account, that you should be able to get all information which is required for suppression from TreeWalkerAuditEvent in future.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Тимур Тибеев

unread,

Jul 15, 2017, 3:17:58 AM7/15/17

to checkstyle-devel

Issue 4369 My commit:

Fixed points

Passed mvn clean verify

Passed cobertura checks

Started working on 4714

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Тимур Тибеев

unread,

Jul 24, 2017, 10:19:47 AM7/24/17

to checkstyle-devel

Issue 4714 Make SuppressionCommentFilter and SuppressWithNearbyCommentFilter children of TreeWalker

Sent PR, fixed all points, waiting for approvement

Issue 4369 Investigate how to map XPath expressions onto AST nodes

Sent PR, received TeamCity and other CI violations, working on fixing them

Issue 4422 Implement XpathFilter

Already started implementing new Xpath filter

Issue 4419 Add token type to LocalizedMessage

Starting working on this issue

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Aug 1, 2017, 5:47:33 PM8/1/17

to checkstyle-devel

Hi, Timur!

Please provide status report.

I provided the plan for #4830 to let us to fix the problems in separate PR. Please, start working on the update of the checks which extend AbstractCheck and log line number and column number. They can be updated to log DetailAST.

Review for your changes for #4422 is on the way. Need more time to finish. We should start working on #4421 next week. Be ready for the issue. I will correct task description ASAP.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Тимур Тибеев

unread,

Aug 2, 2017, 9:11:49 AM8/2/17

to checkstyle-devel

Andrew,

Issue #4714: Make SuppressionCommentFilter and SuppressWithNearbyCommentFilter children of TreeWalker

Resolved, PR was merged

Issue #4419: Add token type to LocalizedMessage

Resolved, PR was merged

Issue #3573: resovle design problem of FileContentsHolder

Resolved, PR was merged

Issue #4843: TreeWalkerFilter needs to be added to ModuleReflectionUtils

Resolved, PR was merged

Issue #4830: Update all AbstractChecks to log DetailAST

Had problems during implementation, discussed and received feedback from mentor,

preparing detailed descriptions of the problems

Issue #4369: Investigate how to map XPath expressions onto AST nodes

Working on fixing points from mentors

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Andrew Selkin

unread,

Aug 14, 2017, 8:49:31 AM8/14/17

to checkstyle-devel

Hello, Timur!

Please, provide Monday status report.

воскресенье, 7 мая 2017 г., 23:12:29 UTC+3 пользователь Andrew Selkin написал:

Тимур Тибеев

unread,

Aug 15, 2017, 6:48:05 AM8/15/17

to checkstyle-devel

Andrew,

Issue #4422: Implement XpathFilter

Fixing points from mentors and errors from CI

Issue #4901: Implement xpath query generator

Fixing points from mentors and errors from CI

Issue #4421: Support suppression-xpath element in SuppressionLoader

Finishing part 1

понедельник, 8 мая 2017 г., 2:12:29 UTC+6 пользователь Andrew Selkin написал:

Roman Ivanov

unread,

Aug 25, 2017, 4:50:50 PM8/25/17

to Тимур Тибеев, checkstyle-devel

Hi Andrew,

please figure out why we still have patches wercker

https://github.com/checkstyle/checkstyle/search?l=YAML&q=patch&type=&utf8=%E2%9C%93

all patches should have commend "till <Link to PR to target project>"

thanks,

Roman Ivanov

Тимур Тибеев

unread,

Aug 30, 2017, 2:09:19 AM8/30/17

to checkstyle-devel

Hello,

GSoC is over and its time to summarize results

During the program I got to know different technologies and techniques. The most interesting part for me was understanding Checkstyle, exploring architecture and testing the code. Another interesting thing was working with Saxon library, since this something I have never did before.

From my mentor Andrei, I knew many things such as mutation tests, coding patterns and best practices.

My future plans

First of all I want to complete my project and continue to improve it. My intentions are to continue contributing to the Checkstyle and other opensource projects. I want to use Checkstyle in my daily developer's life. With new suppression model, it maybe easier to suggest this tool to other developers. I have plans to introduce Checkstyle tool in local meetups and etc.

Thanks to Andrei, Richard and Roman for your help and advices

My complete report is here

R Veach

unread,

Oct 30, 2017, 4:07:02 PM10/30/17

to checkstyle-devel

Timur/Mezk:

Are both of you planning to continue working on this project?

If so, what is the plan for finishing this? Can you give a list of unfinished/open Issues and PRs to complete this project, and the order they need to be tackled in? Please also make note of anything half-completed and make sure the issue notes what is done and what is left.

This is what we see on github: https://github.com/checkstyle/checkstyle/projects/2

Timur:
If you have any finished worked but are waiting for other things to be finished first, please push them to the server now and make a note of them here so we have a record of your work and can finish them if you happen to become too busy.

Тимур Тибеев

unread,

Nov 1, 2017, 10:31:29 AM11/1/17

to checkstyle-devel

Hi Richard,

Support suppression-xpath element in SuppressionLoader is done, issue should be marked as completed

I started but not finished 4530 and 102 due my education and work, I can finish 102 this week and 4530 next week

After those issues xpath-suppression model can be already used for some of the checks

Other issues can be finished in any order

вторник, 31 октября 2017 г., 2:07:02 UTC+6 пользователь R Veach написал:

R Veach

unread,

Dec 5, 2017, 2:17:15 PM12/5/17

to checkstyle-devel

Just to keep all parties informed, there was a bug discovered in xpath.
See https://github.com/checkstyle/checkstyle/pull/5321 .

Message has been deleted

Roman Ivanov

unread,

Sep 6, 2018, 8:07:11 AM9/6/18

to khalid...@gmail.com, checkstyle-devel, Lars Ködderitzsch

Hi Khadil,

please do analysis on https://github.com/checkstyle/checkstyle

sources.

you did https://github.com/checkstyle/eclipse-cs .

Lars, FYI.

thanks,

Roman Ivanov

On Thu, 6 Sep 2018 at 04:23, Khalid AlKharabsheh <khalid...@gmail.com> wrote:

Reply all

Reply to author

Forward