Introducing New Gerrit Plugin: Code Review with ChatGPT

[谢龙]

Hello Everyone,

I've been working on a plugin that leverages OpenAI's ChatGPT for code review.

The 'ChatGPT Code Review' plugin introduces a new dimension to the code review process by utilizing the capabilities of OpenAI's powerful language model. Once a patch is submitted, OpenAI will provide review comments in the form of feedback to help improve your code.

For more details, including configuration parameters and testing guidelines, please refer to the plugin README in the Github repository: https://github.com/xielong/chatgpt-code-review-gerrit-plugin


I'm excited to hear your feedback, suggestions, or any issues you encounter when using the plugin :)

[Luca Milanesio]

That’s so cool, I am very tempted to enable this on GerritHub.io :-) with a project-specific configuration for all of those who want to use it.

That would require:

- project-specific configuration for enable/disable the review

- project-specific authentication tokens to talk to OpenAI API

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0d29ab02-c628-4c21-9324-8af26600f4e9n%40googlegroups.com.

[Christian Gagneraud]

We will give it a try sometime next week.
First time with ChatGPT (just went through
https://platform.openai.com/docs/quickstart) and looking at your code,
mainly this:

public static final String OPENAI_DOMAIN = "https://api.openai.com";
public static final String DEFAULT_GPT_MODEL = "gpt-3.5-turbo";
public static final String DEFAULT_GPT_PROMPT = "Act as a Code
Review Helper, please review this patch set: ";

Would be nice if this could be changed without having to rebuild the plugin.

Just saw Luca's answer, I think my comment is similar.

I'm really curious about the results.

Oh! I just realised that the above values are the defaults. They can
be changed in the gerrit config file.

Thanks,
Chris

[Christian Gagneraud]

On Sun, 18 Jun 2023 at 15:29, Christian Gagneraud <chg...@gmail.com> wrote:
[...]

> Oh! I just realised that the above values are the defaults. They can
> be changed in the gerrit config file.

Looks like the ChatGPT "temperature" parameter cannot be changed.

PS: Sorry for the top-posting in the previous message, that's gmail
default behaviour.

[Adrià Vilanova Martínez]

On Sunday, June 18, 2023 at 12:02:41 AM UTC+2 Luca Milanesio wrote:

On 17 Jun 2023, at 09:21, 谢龙 <xielo...@gmail.com> wrote:

Hello Everyone,

I've been working on a plugin that leverages OpenAI's ChatGPT for code review.

The 'ChatGPT Code Review' plugin introduces a new dimension to the code review process by utilizing the capabilities of OpenAI's powerful language model. Once a patch is submitted, OpenAI will provide review comments in the form of feedback to help improve your code.

For more details, including configuration parameters and testing guidelines, please refer to the plugin README in the Github repository: https://github.com/xielong/chatgpt-code-review-gerrit-plugin


I'm excited to hear your feedback, suggestions, or any issues you encounter when using the plugin :)

Thanks for building this! I will also test it when I have some time in the Gerrit instance where I hold my personal projects.

Also, you might find the following blog article interesting (it's another great use of AI in code reviews!): Resolving code review comments with ML – Google AI Blog 

That’s so cool, I am very tempted to enable this on GerritHub.io :-) with a project-specific configuration for all of those who want to use it.

That would require:

- project-specific configuration for enable/disable the review

- project-specific authentication tokens to talk to OpenAI API

 

+1, it would be very helpful even outside GerritHub.io! For instance in my personal instance I have some private projects I'd prefer not to send to OpenAI :)

[谢龙]

Thanks for your suggestion! 

I've been looking into the Gerrit documentation, and it seems like I could create a Plugin Settings Screen. This would allow us to configure ChatGPT tokens for different projects and securely store them within the plugin. I'm now investigating if Gerrit has any user or project-level encryption/decryption api that could be used for this task.

If you have any ideas or specific suggestions in this area, that'd be great! : )

[谢龙]

Thanks for your interest! I will provide a configuration option for the 'temperature' parameter in the upcoming updates. Stay tuned! :)

[谢龙]

No problem, I will start by adding a project switch based on the existing configuration.

I'll also check out the article you provided - thanks for sharing! ~

[谢龙]

gptTemperature, globalEnable, and enabledRepos can now all be configured in gerrit.config.

gptTemperature: The default value is 1. What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.
globalEnable: The default value is true. If set to false, the plugin will only review the repositories specified in the enabledRepos.
enabledRepos : The default value is empty. If globalEnable is set to false, the plugin will operate exclusively on the repositories specified here. The value should be a comma-separated list of repository names, for example: " repo1,repo2,repo3".

[Matthias Sohn]

It would be more useful if project specific configuration would be done in the project.config on the refs/meta/config branch of a given project.
This enables project owners to configure options themselves without bothering central Gerrit admins.

The default would then be defined in project.config of the All-Projects project.

See https://gerrit-review.googlesource.com/Documentation/dev-plugins.html#simple-project-specific-configuration

-Matthias

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/4800b077-7fd6-48f4-bd46-bfc9dde03334n%40googlegroups.com.

[谢龙]

Good suggestion, Thanks. I will look into it.

[谢龙]

The project-specific configuration feature has been provided:  https://github.com/xielong/chatgpt-code-review-gerrit-plugin#configuration-parameters
Thank you for your valuable input and feedback : )  

[Matthias Sohn]

On Mon, Jun 26, 2023 at 2:14 PM 谢龙 <xielo...@gmail.com> wrote:

The project-specific configuration feature has been provided:  https://github.com/xielong/chatgpt-code-review-gerrit-plugin#configuration-parameters
Thank you for your valuable input and feedback : )  

Please avoid top posting on this list, use interleaved posting style instead which makes it easier

to follow the conversation [1]

I think the gptToken should not be stored unencrypted in gerrit.config since this file is typically versioned

in some git repository which doesn't look like a good idea from a security perspective. 

Storing unencrypted secrets in a separate, strictly local, configuration file secure.config

which should be only readable by the OS user running Gerrit would be more secure [2].

A more secure solution involves encrypting the secure.config file, [3] is an example demonstrating

how this can be implemented.

Similarly storing unencrypted secrets on the refs/meta/config branch of a repository is not very secure.

[1] https://en.wikipedia.org/wiki/Posting_style#Interleaved_style 

[2] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#secure.config

[3] https://gerrit.googlesource.com/plugins/secure-config/

-Matthias

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/f5a79b41-b559-47e2-a69a-c9244a9c3d57n%40googlegroups.com.

[谢龙]

Please avoid top posting on this list, use interleaved posting style instead which makes it easier

to follow the conversation [1]

 

Sorry for that. I will switch to interleaved posting style.

I think the gptToken should not be stored unencrypted in gerrit.config since this file is typically versioned

in some git repository which doesn't look like a good idea from a security perspective. 

Storing unencrypted secrets in a separate, strictly local, configuration file secure.config

which should be only readable by the OS user running Gerrit would be more secure [2].

A more secure solution involves encrypting the secure.config file, [3] is an example demonstrating

how this can be implemented.

Similarly storing unencrypted secrets on the refs/meta/config branch of a repository is not very secure.

Thanks for the clear explanation. I'll make some changes to this part based on your instructions. 

[Han-Wen Nienhuys]

If I understand correctly, you send ChatGPT the following prompt,

   "Act as a Code Review Helper, please review this patch set: " + diff_contents

Can you share some examples of the full prompt, and tell us how well this works where in practice? Does ChatGPT have useful feedback?

--

Han-Wen Nienhuys - Google Munich

I work 80%. Don't expect answers from me on Fridays.

--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Liana Sebastian

[Christian Gagneraud]

On Wed, 28 Jun 2023 at 21:04, 'Han-Wen Nienhuys' via Repo and Gerrit
Discussion <repo-d...@googlegroups.com> wrote:
>> I'm excited to hear your feedback, suggestions, or any issues you encounter when using the plugin :)
>
>
> If I understand correctly, you send ChatGPT the following prompt,
>
> "Act as a Code Review Helper, please review this patch set: " + diff_contents
>
> Can you share some examples of the full prompt, and tell us how well this works where in practice? Does ChatGPT have useful feedback?

FYI, as an alternative, i looked into that:

https://arxiv.org/pdf/2203.09095.pdf
https://huggingface.co/microsoft/codereviewer

Unfortunately, it's just a demo, nowhere near ready for production.

Still, the paper is very interesting.

Chris

[谢龙]

Han-Wen Nienhuys <han...@google.com> 于2023年6月28日周三 17:04写道：

On Sat, Jun 17, 2023 at 10:21 AM 谢龙 <xielo...@gmail.com> wrote:

Hello Everyone,

I've been working on a plugin that leverages OpenAI's ChatGPT for code review.

The 'ChatGPT Code Review' plugin introduces a new dimension to the code review process by utilizing the capabilities of OpenAI's powerful language model. Once a patch is submitted, OpenAI will provide review comments in the form of feedback to help improve your code.

For more details, including configuration parameters and testing guidelines, please refer to the plugin README in the Github repository: https://github.com/xielong/chatgpt-code-review-gerrit-plugin


I'm excited to hear your feedback, suggestions, or any issues you encounter when using the plugin :)

If I understand correctly, you send ChatGPT the following prompt,

   "Act as a Code Review Helper, please review this patch set: " + diff_contents

Can you share some examples of the full prompt, and tell us how well this works where in practice? Does ChatGPT have useful feedback?

Just wanted to share my experience with using ChatGPT in code review. The quality of the results really hinges on the model you choose and the prompts you provide. For example, if you opt to use the GPT-3.5 model with default plugin prompts, the level of suggestions you get would be approximately the same as current static analysis tools, such as Sonar or Findbugs. This means you could use this plugin as a replacement, or in conjunction with these tools for a more robust code review experience.

However, if you choose more specific prompts, you'll find that the suggestions provided by ChatGPT are usually more precise. For instance, a common prompt I use is: "Review this patchset according to the principles of 'Clean Code'."

And if you employ the GPT-4 model along with specific prompts, you will find that the review results from ChatGPT can sometimes be quite impressive.

Since our business code cannot be directly demonstrated, I'm willing to take the time to create some business-unrelated examples to vividly showcase the effects and recommended prompts.

Furthermore, we have found that using ChatGPT as an assistant in code review usually results in better outcomes. Therefore, our plugin has also been updated to support the @ function. You can @ your configured ChatGPT bot in comments and ask specific questions, like: "The xx function seems a bit complex, is there any possibility to optimize it?" or "Could the naming of this piece of code be more concise?" Conducting reviews through this interactive Q&A approach makes the entire process more flexible and precise. 

[Luca Milanesio]

Wow, that’s impressive.

Are you willing to present this at the next forthcoming Gerrit User Summit 2023?

Luca.

[谢龙]

Luca Milanesio <luca.mi...@gmail.com> 于2023年6月29日周四 15:50写道：

Hi, thank you so much! Really appreciate it.

May I know when and where the summit is, and I assume the presentation could be online? I live in Beijing, China. 

[Luca Milanesio]

We are finalising the dates/times and an announcement will come at the beginning of July.

Of course you can attend and present remotely, no issues with that :-)

Thanks again for sharing this with the community, much appreciated.

Luca.

[谢龙]

Luca Milanesio <luca.mi...@gmail.com> 于2023年6月29日周四 19:10写道：

Thank you! I’m looking forward to it. 

[Christian Gagneraud]

Spent some time looking deeper into that, and this is my take:

From a legal point of view, this is interesting for several reasons:
- model was trained exclusively on permissive licensed code
- You can host the service on premise, so no need to send your code to
third parties
I still need to figure out licensing terms for generated code.

A definite advantage of this project is that it is tailored for the
job of code review.
Inputs of the model are the original source file, a patchset, and the
set of comments.

Unfortunately, this project is "just" a prototype for the paper. Lack
of documentation and code quality make it hard to do a real world
evaluation.
According to the latest message from Han-Wen, the ChatGPT model looks
more promising.
Unfortunately (for us), I think there are licensing issues.

Thanks Han-Wen for sharing your experience, and thanks 谢龙 for
contributing your plugin.
I will definitely give it a try.

Chris

[Michael Nazzareno Trimarchi]

Hi all

First of all thank you for starting. We were looking for in the company for a nice use case.

We went a big further on the plugin an create a fork on project. We are still base our experience on chatgpt. You can find more information in [1] and source code [2]. We will like to try using different llama review codebase but at the moment we don't have enough resources in our infrastructure for such setup.

Michael

[1] https://wiki.amarulasolutions.com/opensource/chatgpt-gerrit.html

[2] https://github.com/amarula/chatgpt-code-review-gerrit-plugin/

Thanks Han-Wen for sharing your experience, and thanks 谢龙 for
contributing your plugin.
I will definitely give it a try.

Chris

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CABxGUTi-CQBrtrkdSykNi6rEmyEUdofHWvx4ED_cEhDxQbgiDQ%40mail.gmail.com.