GSoC 2017 - Regression Testing Project

[R Veach]

This thread is a placeholder only for the GSoC 2017 Regression Testing project.

https://github.com/checkstyle/checkstyle/wiki/Checkstyle-GSoC-2017-Project-Ideas#project-name-regression-testing-tool-and-html-report-generator-for-pull-request

[骆梁宸]

Hi there,

I am Liangchen Luo, the GSoC student developer for this project.

Here is my proposal submitted to Google: http://www.luolc.com/GSoC2017-playground/proposal.html

There would be some small changes to the timeline I make before.

In the proposal I put "Try the GitHub Pages Deployment service of Travis CI" in July 1 - 28. But now I decide to do it at first. If we could set up a deployment service at first, then all the PRs from then on could test our deployment function. Even though we are not generate test reports right now, we could test the deployment itself immediately. And once we could generate reports, PRs could be nice test source. If I do this at the last, I could only test myself.

So I am reading the Travis-CI's doc and test it in my own repo now. The doc link: https://docs.travis-ci.com/user/deployment/pages/

Notice that to use the deployment service, a personal access token is required. But I do not know whether it is safe to use such info for an organization repo. The token could be set as a global variable on Travis-CI and not be shown in the build.

Please confirm :D

BTW: Does anyone know how to at someone at Google groups? This reply is not emergency but I want to learn it in case a further need.

在 2017年5月6日星期六 UTC+8下午10:04:15，R Veach写道：

[R Veach]

> If we could set up a deployment service at first, then all the PRs from then on could test our deployment function.

It is my opinion that we shouldn't hook it up to the CI system right away. We should wait until it can produce some kind of output, however small, before this.
If we put it as part of the CI and it isn't stable, we may constantly get red/failing CIs and will start to annoy people. We already deal with this problem a bit, and it has become annoying and makes the CI less reliable in it's results.

> If I do this at the last, I could only test myself.

We do test driven development. Since I am expecting us to create this as a Java utility, we will have unit tests and code coverage to showcase that the utility works correctly and we have little unexpected issues when we go to deploy it.

> Notice that to use the deployment service, a personal access token is required. But I do not know whether it is safe to use such info for an organization repo. The token could be set as a global variable on Travis-CI and not be shown in the build.

We may be unable to do anything about this as it is a limitation of travis. Travis only lets us use these tokens for master CIs. It doesn't allow them for PRs. So we will have to have a public token in the code and CI logs until we can find a way around this.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions



We have started a google document with your proposal and some extra notes that I have written on any ideas and specifications on the utility. Feel free to look it over and ask any question or make suggestions.

https://docs.google.com/document/d/1p9d54E7hV9PLkjFaKrNoEltcwFtFSkIntVSTU9BcqQY/edit

[骆梁宸]

We should wait until it can produce some kind of output, however small, before this.

It does make sense. I would create a simplest generator before put it in CI system.

I would try to find other workaround for the PR limitation.

在 2017年5月29日星期一 UTC+8上午7:39:32，R Veach写道：

[骆梁宸]

I am writing a script to generate the simplest config depending on the changed files now(with the help of JGit) and would complete it soon. Should I create a new dir in checkstyle/contribution repo for this or just keep it on my local(or own GitHub repo) currently?

在 2017年5月29日星期一 UTC+8下午8:41:34，骆梁宸写道：

[R Veach]

> I am writing a script to generate the simplest config

Please define `script`. We are expecting this to be a complicated tool so we are currently expecting it to be written in Java, not a scripting language like bash or groovy.
This will try to fix all our needs for regression so it will grow over time.

> Should I create a new dir in checkstyle/contribution repo

https://github.com/checkstyle/regression-tool

Please use this repository as the home for your tool.

[骆梁宸]

We are expecting this to be a complicated tool so we are currently expecting it to be written in Java, not a scripting language like bash or groovy.

I am using clojure to explore the JGit library and the algorithm now as it is easy to implement smth quickly. And it is only for experiment. I will refactor the code into java.

Please use this repository as the home for your tool.

Got it. 

在 2017年5月31日星期三 UTC+8下午9:59:52，R Veach写道：

[骆梁宸]

Having some problems now.

The repo checkstyle/regression-tool is empty now. I cannot fork it as GitHub tells me "Cannot fork because repository is empty." Therefore I cannot send PR to it.

I don't have write access to this repo so I am not able to add smth to make it non-empty.

Would you please create smth(maybe a simple README) and then I could fork it and send PR later. :D

在 2017年5月31日星期三 UTC+8下午10:43:39，骆梁宸写道：

[R Veach]

Done.

[骆梁宸]

Project State Report [2017-06-02 00:25:09 +0800]

Recent work

• Investment JGit to help us get the Git branches diff information.
• It is not convenient to get such information via Git CLI, as we will need many details (i.e. changed type, lines) in the future.
• JGit can provide the changed file info and each edit info. An edit is a change of a continuous code block. For the simplest option generator (Using default option), the file change info is enough.
• Do experiment of Travis-CI deployment
  
• I created a testing repo in my profile to do experiment of Travis-CI deployment.
• Due to the limitation of Travis-CI, PR build cannot use the Travis environment variable. So maybe deploying the regression report to a separate repo and hardcode the access token of the "report-repo" could be a workaround. To-Do: confirm the workaround.
• Implement the simplest generator
• I have implemented a simplest generator (Using default options, without any other specific settings) with the help of JGit, in Clojure language.
• I am refactoring the Clojure script into Java and moving it to checkstyle/regression-tool repo. Clojure is also a JVM based language so the refactoring would not be hard.

Further plan

• Complete the simplest generator in Java and upload it.
• Keep investigating the Travis integration solution.

在 2017年5月31日星期三 UTC+8下午11:57:35，R Veach写道：

[骆梁宸]

Project State Report [2017-06-05 18:36:09 +0800]

Recent work

• Implement the simplest generator
• Completed the refactor from Clojure script to Java, also with some basic settings: .gitignore, .travis.yml etc.
• About the current structure of the project, see https://github.com/checkstyle/regression-tool/pull/1 .

Further plan

• Refine the simplest generator with mentors.
• Keep investigating the Travis integration solution and try to use the simplest generator in Travis-CI.
  

在 2017年6月2日星期五 UTC+8上午12:52:53，骆梁宸写道：

[R Veach]

I merged the PR to start the maven project.
I also started some issues, please look at the ones assigned only to you. The others are things to think about for the future as we more further along.
Feel free to discuss anything with me.

[骆梁宸]

Got it. I've seen the "Add more to CI" issue. I will send the PR of Git component after we refine the CI with more verification. And I am thinking about the test of Git component currently.

Before this, I will finish this https://github.com/checkstyle/checkstyle/issues/4414 as it is emergency.

在 2017年6月6日星期二 UTC+8下午6:31:19，R Veach写道：

[骆梁宸]

Project State Report [2017-06-09 03:28:31 +0800]

Recent work

• Implementing the git component
• I tried to write some UTs for the git component, but not sure if it is appropriate yet.
• PR: https://github.com/checkstyle/regression-tool/pull/17
  
• Codecov branch coverage is 50% with try-with-resource block. I gave up for raising that to 100%. FYI: https://stackoverflow.com/questions/26360245/try-with-resource-unit-test-coverage

Further plan

• Discuss the design of other components and implement them.

• Keep investigating the Travis integration solution and try to use the simplest generator in Travis-CI.

在 2017年6月6日星期二 UTC+8下午6:52:43，骆梁宸写道：

[Roman Ivanov]

> Codecov branch coverage is 50% with try-with-resource block. I gave up for raising that to 100%. 

please use old catch approach. It is issue with cobertura, unfortunately.

if you run into more problems that cobertura is misbehaving - please contact mentors.

​

[骆梁宸]

Got it. Thanks. :)

在 2017年6月9日星期五 UTC+8上午6:27:54，Roman Ivanov写道：

[骆梁宸]

Project State Report [2017-06-13 00:42:57 +0800]

Recent work

• Complete core function of Git Component
• Refining the test of Git Component
• Have a conclusion of the components' responsibilities

Further plan

• Keep refining the test of Git Component
• Start the work of Module Component

在 2017年6月13日星期二 UTC+8上午12:42:40，骆梁宸写道：

[骆梁宸]

Project State Report [2017-06-16 03:28:04 +0800]

Recent work

• Refine the test of Git Component
• Refine the Git Component
• Discuss the solution of grabbing information for branch version

Further plan

• Keep refining the test of Git Component

• Work on Module Component with some workaround

在 2017年6月13日星期二 UTC+8上午12:48:10，骆梁宸写道：

[骆梁宸]

I just had a discussion with rnveach about whether we need a filter interface in DiffParser.

Take default generator as an example, we would just ignore utility class currently. So we would drop utility classes in default generator. There are 2 ways: write the logic of dropping them all in module component; or add a filter interface in DiffParser, and pass the filter as a parameter to DiffParser.

Then we reach an agreement that we would do the filtering in module component. We would start to focus on utility classes as well when we start woking on checkstyle/regression-tool#6. Dropping the utility classes in module component now is fine and git component would not do the work of module component.

在 2017年6月16日星期五 UTC+8上午3:31:40，骆梁宸写道：

[骆梁宸]

Regrading the filter, I forgot one thing. According to the discussion on https://github.com/checkstyle/regression-tool/pull/17#issuecomment-307312459 and the following, we would not only grab the file in path "src/main". But currently for the default generator we need a filter to drop testing files and other non-related files (README etc.). These filters only base on file path. Should we add an interface in DiffParser for these filters?

在 2017年6月16日星期五 UTC+8下午10:27:38，骆梁宸写道：

[R Veach]

> we would not only grab the file in path "src/main"

> Should we add an interface in DiffParser for these filters?

We will eventually need test files to pull in properties for non-default module testing. https://github.com/checkstyle/regression-tool/issues/5
If we add a filter to drop these files from `DiffParser` now, we will need to remove it when we do need to grab those changes, or re-run it with an opposite filter.
This is really no different than the utility classes discussion.

IMO, we shouldn't run the git component more than once.
We should get all the file differences we need in one pass. Filtering specific files should be done in the `module` component.
As right now, we will need 3 different 'filters': Module, utilities, tests.

[骆梁宸]

Project State Report [2017-06-20 01:21:15 +0800]

Recent work

• Work on Module Component
• I introduced Immutables to make it easy of writing POJOs.
• Currently using Json to save the module information.
• There are still many things to discuss about the PR#33.

Further plan

Discuss the questions raised by mentor and keep working on it.

在 2017年6月17日星期六 UTC+8上午1:15:02，R Veach写道：

[骆梁宸]

Project State Report [2017-06-23 02:32:31 +0800]

Recent work

• Work on Module Component
• There should be a middleware returned by module component. Gathering the direct change of a module class, the change of related UT class and the changes of utility classes it used.
• Still thinking about the contents of the middleware. Current opinion: ModuleInfo(represents the data in json), moduleChange, testChange, utilityChanges(a list of change). Refactoring on progress but not very sure if it is appropriate enough. It should be thought carefully in consideration of the future compatibility(including test change and utility changes).  
• Fix some minor problems in PR#33

Further plan

• Discuss with mentor and determine the contents of the middleware, and refactor the ConfigGenerator.

P.S. I was a little busy these days and don't have much progress. :( My summer vacation is coming soon and I would work much harder then!

在 2017年6月20日星期二 UTC+8上午1:28:08，骆梁宸写道：

[骆梁宸]

Recent work

• Module Component
• Changes are split to different parts: dependencies update, adding data object, json related codes and main module functions
• dependencies: Introduced Immutables, Gson and re-order, #38: https://github.com/checkstyle/regression-tool/pull/38. Just some minor changes.
• adding data object: ModuleInfo and ModuleExtractInfo, including some suppression related. #39: https://github.com/checkstyle/regression-tool/pull/39
  
• json related: see below
• main: blocked by the above changes. We have confirm the contents of the middleware(ModuleInfo) and the generation rule in the progress of loop(add depending on production code changes, then test code). Code 
• Extract Component
• Json processor is a copy of the codes in module component before.

Further plan

• Refine the test of git component.
• The diff between a latest master and a branch with rebasing could have problems. I have also tested it in git CLI, it has the same behavior.
• I would confirm the problems in git CLI first and find solutions of our project.
• Configuration Component
• Start this if there is rest time

在 2017年6月23日星期五 UTC+8上午2:44:57，骆梁宸写道：

[R Veach]

If you find yourself waiting on me any, please start looking into other issues coming up and describe in some high level details on how you plan to work the issue. This way we can agree on path before any coding has begun.

[骆梁宸]

Got it.

在 2017年6月29日星期四 UTC+8上午1:20:08，R Veach写道：

[骆梁宸]

Recent work

• Completed module component
• Developing configuration component
• Completed the main logic of generating a config xml, PR #43
• Missing the output to file function currently.
• Added a filter of DELETE type in git component.

Further plan

• Refine the test of git component.
• The diff between a latest master and a branch with rebasing could have problems. I have also tested it in git CLI, it has the same behavior.
• I would confirm the problems in git CLI first and find solutions of our project.

• Starts report component if there is rest time

在 2017年6月29日星期四 UTC+8上午1:55:38，骆梁宸写道：

[骆梁宸]

Recent work

• Developing configuration component
• Fix CRLF problem in test.
• Move all XML resources to the resources folder instead of hardcoding.
• Add a test checking the version of DTD.
• Add "output to file" function.
• Fix diff problems when parsing a branch behind master
• Find a common ancestor of PR branch and master at first, then run diff command between the base commit the the PR head commit.
• Developing report component
• As discussion at https://github.com/checkstyle/regression-tool/issues/21#issuecomment-312438062, I would first clone the contribution project run the groovy CLI directly to see how it works.

Further plan

Keep working on report component.
Work on next part of extract component.

在 2017年6月30日星期五 UTC+8上午5:06:17，骆梁宸写道：

[骆梁宸]

Recent work

• Completed Configuration component.
• Completed fixing diff problems when parsing a branch behind master.
• Completed the refining of DiffParserTest
• There is a issue found and created separately #49
• Developing Report component
• I collected some questions and discussed with mentor.
• Developing Extract component
• Split the work into two parts: #50 and #52

Further plan

• Keep working on Report component and Extract component.

在 2017年7月4日星期二 UTC+8下午12:42:50，骆梁宸写道：

[骆梁宸]

Recent work

• Developing Report component
• Trying the way of integrating Groovy script into our project.
• Waiting for confirm of whether to place the script in our repo directly.
• Developing Extract component
• Use our own Json processing utility instead of other dependencies.
• Add a no exception test of extract info generation in Travis.
• Update the codes in #50, missing Maven invoker currently. 
• Fix some minor bugs.

Further plan

• Decide the Report component's plan and develop on that.

在 2017年7月7日星期五 UTC+8上午3:07:24，骆梁宸写道：

[骆梁宸]

Recent work

• Developing Report component
• Facing some problems of invoking Groovy in Java, maybe consider to use CLI directly.
• Developing Extract component
• Use Maven Invoker instead of Maven-Embedder to avoid strange error.
• Preparing to start `Main` as extract and report would be completed soon.

Further plan

• Completed extract and report component and start on Main.

在 2017年7月11日星期二 UTC+8上午11:57:10，骆梁宸写道：

[骆梁宸]

Recent work

• Waiting for mentor's further suggestion on `report` and `extract` component.
• Testing the deployment function of Travis-CI.
• Working on #4607

Further plan

• Completed extract and report component and start on Main.

在 2017年7月14日星期五 UTC+8上午11:53:25，骆梁宸写道：

[骆梁宸]

Recent work

• Completed extract and report component.
• Working on Main. Now refining the arguments name and the way of handling exceptions.

• Testing the deployment function of Travis-CI.
• Working on #4607

Further plan

• Completed Main.

在 2017年7月18日星期二 UTC+8上午10:45:40，骆梁宸写道：

[骆梁宸]

Recent work

• Completed main part of Main. Need to fix jar generation bug and add docs.

Testing the deployment function of Travis-CI.
Working on #4607

Further plan

• Completed Main.

在 2017年7月21日星期五 UTC+8上午10:58:21，骆梁宸写道：

[R Veach]

Luolc,

We are now approaching the 2nd review for GSoC and I feel we haven't made much progress towards catching up.
We have only recently just gotten a working command line entry point for the application after we have dropped writing tests and code coverage. Our past 24 hours consists of you answering yesterdays questions, pushing a small change that I pointed you to, and me responding to your new changes (https://github.com/checkstyle/regression-tool/pull/57#issuecomment-318122791).

You said you would find more time once out of school, but I don't feel it has improved any. There was a time where I was absent for a few days, and you didn't do any other activity while you were waiting for me to review the current PR. We currently have 16 open issues, any of which could be worked on. I only use assignments to direct where I think we should be focusing but I won't deny work for other issues. Some of the open issues also have open ended questions, where you can give your opinion on and we can come to a decision on how to implement the requested functionality.

I ask you please be more active and push code and do more each day. Even if it is incomplete and fails CI, atleast it will allow me some chunks to review and help guide you instead of doing it after you write it all. If you are stuck on something ask me questions and switch to something else that you know you can do. Ping me in chat if you need me to look at or help with anything quicker.
I would love for us to get more complex stuff done than default configuration generation.

[R Veach]

Luolc,

While you continue to do work in regression-tool, please begin the PR for https://github.com/checkstyle/checkstyle/issues/4607 . You have stated that you have been working on this for atleast 10 days, so I expect you have a good handle on it.
Once we release our tool to checkstyle, we will need this in the next release of checkstyle for regression on using non-default modules. We also ran into a bug with this recently, so it would be better to get it done sooner.

As you finish implementing the regression on checkstyle for our tool, please specify which issue you would like to work on next:
https://github.com/checkstyle/regression-tool/issues/5
https://github.com/checkstyle/regression-tool/issues/6

[骆梁宸]

Sorry for my misunderstanding of the assignment label, I considered I should focus on the labeled ones before. Work in July is harder than what I expected at the beginning and I forgot to ask for help more frequently. I would update my work even it is not completed in the next working period and work hard for the project.

Recent work

• Completed Main.

Further plan

• Create a documentation.
• Add a no exception test.
• Issue #4607
• Further develop on non-default mode.

在 2017年7月28日星期五 UTC+8上午10:32:37，R Veach写道：

[骆梁宸]

Recent work

• Documentation
• A doc about how to use regression-tool is created
• No exception test
• We use the HEAD commit and one specified commit to do the test.
• `--stopAfterGenerateConfig` mode is fine now.
• Report generation still has problems. I am trying to reproduce the error on my local.

Further plan

• Issue #4607
• Seems I misunderstood the approach to solve this before, to some extent. I would check other's work on this and re-think my way.

• Further develop on non-default mode.

在 2017年7月28日星期五 UTC+8下午12:23:44，骆梁宸写道：

[R Veach]

Luolc, we haven't recieved any new updates from you yesterday. Is there a problem?
Is there still problems starting the PR for https://github.com/checkstyle/checkstyle/issues/4607 ?

[骆梁宸]

Sorry I got a toothache yesterday and went hospital. I am at hospital now and would update my progress after getting home at noon.

在 2017年8月3日星期四 UTC+8下午6:55:32，R Veach写道：

[骆梁宸]

Sorry for the late of the summary. Spent much time to get an appointment with doctors...

Recent work

• Documentation
• Refine the doc and spilt the bugs to separate issue.
• No exception test
• Now I could reproduce the bugs on my local. But I am not very familiar with Groovy and still finding the bugs now. I haven't made sure it is a regression-tool bug or a checkstyle-tester bug.
• Issue #4607
• I didn't realize that I need to update `compareTo` method before. I delete all the wrong changes now and send PR of my latest commit.

Further plan

Further develop on non-default mode.

Bug fix of the issues split from docs.

在 2017年8月4日星期五 UTC+8上午9:57:14，骆梁宸写道：

[骆梁宸]

Recent work

• Documentation
• PR merged.
• No exception test
• No much progress on this. I would add a non-zero exit code first if the diff run fails.
• Issue #4607
• Complete most points. Some of discussions remaining.

Further plan

• Further develop on non-default mode.
• Bug fix of the issues split from docs.

在 2017年8月5日星期六 UTC+8上午1:13:44，骆梁宸写道：

[R Veach]

As you finish implementing the regression on checkstyle for our tool, please specify which issue you would like to work on next:
https://github.com/checkstyle/regression-tool/issues/5
https://github.com/checkstyle/regression-tool/issues/6

This question was never answered. Improving regression functionality should be our next focus.

[骆梁宸]

Recent work

• Refine some documentation related code.
• Fix the bug that user can only run the tool in source directory.
• Add a test to verify all injection files are in copy list.
• Finished NoException test with the help of mentor.
• Finished #issue4607

Further plan

• Working on #5 and #6 to add more features for the tool

在 2017年8月11日星期五 UTC+8上午12:36:47，R Veach写道：

[骆梁宸]

Recent work

These days I spent most time on investigating the testing on more properties.

Firstly, both utility regression and non-default regression needs the line change info. Updating `git` to let it be able to detect line changes is the first thing to do.

For finding possible properties, I think from UT is an independent way, to some extent. There are some examples: (searching 'regression' in checkstyle repo)

https://github.com/checkstyle/checkstyle/pull/4905

https://github.com/checkstyle/checkstyle/pull/4902

https://github.com/checkstyle/checkstyle/pull/4853

https://github.com/checkstyle/checkstyle/pull/4770

https://github.com/checkstyle/checkstyle/pull/4736

These PRs all contain module changes along with the corresponding UT changes. By detecting `createModuleConfig`, `addAttribute` and other related methods, we can find what properties to use.

Besides, most UTs are independent from each other. That is an advantage. Processing one file is easier than analyze many files together.

But still, some testing files are complicated and can't be treated easily. But I think the common UT style could cover a large proportion.

Hard-coding the properties to use is a solution. But we have so many modules and some properties type are not enumeration(i.e. regular exp). Hard-coding may help, but we need to writing them little by little. This could be an auxiliary way.

Other way is analyzing the changes of production code directly. We could mark all the fields that represent properties. If the changed lines covered some of them, then we may know which properties to set. But we can't decide the value, we only know the property name. Still we need some hard-coding I think.

I am thinking if we could use the feature of checkstyle to help us detecting the affect lines of a field. For example, implement a check, process the changed file, and print the lines in `visitToken` or `leaveToken`.

To sum up, I think analyzing UT is more independent and clearer. I prefer to start from this.

For the utility regression, I would like to find some PR example. I assume it is not quite often. So currently I think finding properties is the more important task.

We need to investigate the static analysis tech of method reference detection. I think there would be many existing tools and methods to refer.

Meanwhile I worked on the line changes detection, I once tired this in my previous work.

Further plan

• Working on #5 and #6 to add more features for the tool

在 2017年8月11日星期五 UTC+8下午12:31:57，骆梁宸写道：

Recent work

• Refine some documentation related code.
• Fix the bug that user can only run the tool in source directory.
• Add a test to verify all injection files are in copy list.
• Finished NoException test with the help of mentor.
• Finished #issue4607

Further plan

• Choose a point from above to focus on.

[R Veach]

> Hard-coding may help, but we need to writing them little by little. This could be an auxiliary way.

> But we can't decide the value, we only know the property name.

Hard coding is only a temporary solution. We need to be dynamic incase a new property is added in the PR. Hard coding won't see this new property and hence won't test it.
https://github.com/checkstyle/regression-tool/blob/master/src/main/resources/com/github/checkstyle/regression/extract/ExtractInfoGeneratorTest.java should be able to pull in the properties just like we do in checkstyle at https://github.com/checkstyle/checkstyle/blob/master/src/test/java/com/puppycrawl/tools/checkstyle/internal/XdocsPagesTest.java#L931 .
But identifying the exact type is still an issue. See https://github.com/checkstyle/checkstyle/issues/3543 and https://github.com/checkstyle/checkstyle/issues/3254 .

So getting property names and default values shouldn't be an issue, since we do that in Checkstyle already. It should just be simply coping code to our extactor.
We just can't get all possible values unless we can reflectively examine all enumerations and such.

> print the lines in `visitToken` or `leaveToken`.

I'm not sure what you are getting at here, but a property isn't always guaranteed to be used in those areas. Checkstyle likes to split up its code in many methods.

> These PRs all contain module changes along with the corresponding UT changes.

Checkstyle enforces this for code coverage and mutation coverage. One small problem is changes in production code may show up in different test files. Mutation does help prevent this, but it is still a slight possibility, but probably less so for checks and some modules.

> But I think the common UT style could cover a large proportion.

If you plan/want to make all UTs follow a single pattern, make an issue for it in Checkstyle repo and we can discuss it more there.

> I think analyzing UT is more independent and clearer

I agree with this for determining property values. We shouldn't need this for property names like I mentioned above, unless you can say what issue there is with it.

> find some PR example

Can you expand on what type of PR example you are looking for?
We can add a new property, remove a property, change it's type, change it's default value, and change it's enumeration if it was attached to one.

> We need to investigate the static analysis tech of method reference detection. I think there would be many existing tools and methods to refer.

Can you expand on what you mean by `method reference detection`. You specified exact methods we were looking for in the UTs. Why can't we use a custom Checkstyle check for this? Thanks to the git component we are already limiting ourselves to the specific module and test.

> both utility regression and non-default regression needs the line change info. Updating `git` to let it be able to detect line changes is the first thing to do.

> I worked on the line changes detection

I assume you are talking about changes to https://github.com/checkstyle/regression-tool/blob/master/src/main/java/com/github/checkstyle/regression/data/GitChange.java to save the line numbers of changes along with the file name.
For any work you are starting please make a new issue for it, if there isn't one so I can follow and help out.

[R Veach]

One final note. We can't always expect there to be test changes with production code. If it is just a fixing of code and not a logic change or such, we still need to run regression on it and all it's properties to verify nothing changed or is broken.
We can assume the tests will be in a specific location as it should just be a change of `main` to `test` and appending `Test` to the class name.

[骆梁宸]

Recent work

• Add line changes detection in git component.
• We came up an agreement that collect property values in all UTs of a `*Test` class. I am writing a custom checkstyle check to go through the code and try to grab the values now.
• I wrote the property name collection, by referring the code in checkstyle. I am thinking to collect the type of them as well.  

Further plan

• Come up a custom check that be able to grab info from common Tests.
• Add property collection in extract component.

在 2017年8月15日星期二 UTC+8上午4:15:49，R Veach写道：

[骆梁宸]

Recent work

• Implement a unit test processor, including a custom check and a utility class. This could process the UT method and grab property info from the `addAttribute` method call.
• Add property name and type collection in extract component.

Further plan

• Refine the unit test processor and integrate it into our generator.
• Try to find a way to handle the type problem of property.

在 2017年8月18日星期五 UTC+8下午12:16:25，骆梁宸写道：

[骆梁宸]

Recent work

• Unit test processor has some problems to fix now:
• not support module config as class field yet
• not support property value that is not string
• I scanned the UTs of checkstyle modules and check the processed result to find if we are still missing anything.
• I have to look into UTs one by one manually. It does take a lot of time. I am thinking to fix and ensure the two problems are OK and move on now. 

Further plan

• Refine the unit test processor and integrate it into our generator.
• Try to find a way to handle the type problem of property.

• I didn't have time to work on this last few days. I would complete the main part of UT processor and leave the rest for now and focus on this as well.

在 2017年8月22日星期二 UTC+8上午11:36:17，骆梁宸写道：

[骆梁宸]

Recent work

• Problems solved and problems left are list at this post: https://github.com/checkstyle/regression-tool/pull/85#issuecomment-325333038
• We plan to write another custom check in sevntu project to force the format of UT method.

Further plan

• Refine the unit test processor and integrate it into our generator.
• Try to find a way to handle the type problem of property

在 2017年8月25日星期五 UTC+8下午12:08:27，骆梁宸写道：

[R Veach]

Student's final summary: https://docs.google.com/document/d/1gOdpkTxLFjbYLV0cPAnwwKZWJ0jCcwIuIuF1jAddsAA/edit?usp=sharing

[R Veach]

Luolc:

Are you planning to continue working on this project? I have tried contacting you and see read receipts and activity on github from you but you have never responded.

I have still continued working on the project in your absence. I would appreciate if you could still help us out.

I will detail my plan for continuing soon.

[R Veach]

The order of issues are:

1) https://github.com/sevntu-checkstyle/sevntu.checkstyle/issues/610 and https://github.com/checkstyle/checkstyle/issues/5228
(https://github.com/checkstyle/checkstyle/issues/5157 might be covered by the above 2)
2) https://github.com/checkstyle/regression-tool/issues/78
-- https://github.com/checkstyle/regression-tool/pull/85
3) https://github.com/checkstyle/regression-tool/issues/79
-- https://github.com/checkstyle/regression-tool/pull/84
4) https://github.com/checkstyle/regression-tool/issues/5
5) https://github.com/checkstyle/regression-tool/issues/6
6) https://github.com/checkstyle/regression-tool/issues/36

Code coverage issues can be done at any time. Updating project to latest CI can be done at any time.

https://github.com/checkstyle/checkstyle/issues/5156 is important, but I don't see it as a priority until the regression util can print these as errors and even still it may take some work to create these tests. We need a tool to print these areas as errors for us before we can work on them.