Turning on AI code review assistant's in OVN-Kubernetes project

[Surya Seetharaman]

Hello all!

We wanted to raise this topic in yesterday's community meeting but didn't get a chance to do so.

Turning on AI code review assistant's in OVN-Kubernetes project can bring about added benefits like fixing typos or smaller mistakes, adding clear code comments or function docs, over time perhaps with consistency and developing standardized patterns and also help have an initial set of eyes on a PR. Note that this in no way means any changes to how reviews are done - We still will have humans review a PR and ensure it goes through its course.

Currently we have two AI tools in mind that can assist in code reviews on PRs opened in OVN-Kubernetes repo:

1. Copilot
   1. We can simply add a workflow into our project that says when a PR is opened for copilot to be a reviewer: https://docs.github.com/en/copilot/using-github-copilot/code-review/using-copilot-code-review#enabling-automatic-reviews
   2. The docs mention https://docs.github.com/en/copilot/using-github-copilot/code-review/using-copilot-code-review#requesting-a-review-from-copilot by adding Copilot as reviewer its possible to get a review done automatically and we used to be able to do this in the past - https://github.com/ovn-kubernetes/ovn-kubernetes/pull/5030#pullrequestreview-2760060203 but perhaps something changed later?
   3. However with my limited experience with copilot - I have not seen that much promising results - but we'd need to turn it ON for a few days to see how it performs with our code base to get a baseline
2. CodeRabbit
   1. https://www.coderabbit.ai/ also allows us to do the exact same aspects
   2. https://docs.coderabbit.ai/platforms/github-com/ from the docs integration seems easy
   3. I have personally not used coderabbit, wondering if anyone in else has and can share their experience?

Either ways - we'd also need to open a ticket with CNCF staff to ensure whichever tool we pick is compliant to their recommendations.

If anyone has other ideas please feel free to share them here. More importantly if there is a strong preference on using one versus the other based on experience please let us know.

I would like to also keep this thread open for some working days for the community to raise any concerns/opinions and by the next community meeting we can have one of the tools turned on

Cheers,

Surya.

[Tim Rozet]

Thanks Surya for sending this out. Here is an example of coderabbit:

https://github.com/flightctl/flightctl/pull/1230

I think it looks pretty cool. We could always try it and then just disable it if it doesn't work for us try something else instead.

Tim Rozet

Red Hat OpenShift Networking Team

--
You received this message because you are subscribed to the Google Groups "ovn-kubernetes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ovn-kubernete...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ovn-kubernetes/48b3535c-c92c-4ef7-a83f-f411f347f080n%40googlegroups.com.

[Surya Seetharaman]

On Tue, Jun 24, 2025 at 5:07 PM Tim Rozet <tro...@redhat.com> wrote:

Thanks Surya for sending this out. Here is an example of coderabbit:

https://github.com/flightctl/flightctl/pull/1230

I think it looks pretty cool. We could always try it and then just disable it if it doesn't work for us try something else instead.

Thanks for sharing this Tim. +1 that it looks more cooler than sample copilot review: https://github.com/ovn-kubernetes/ovn-kubernetes/pull/5313#discussion_r2163248575

It seems to be telling us:

1. code diff
2. suggested code diff
3. mermaid diagram of code call which is awesome specially for bigger PRs
4. PR summary

I'm sold on trying out coderabbit.ai. I will start with requesting an open source tier account for it as indicated in the docs.

Meanwhile let's see what other maintainers think.

Cheers,

Surya

[Nadia Pinaeva]

Hey all, thanks for sharing the examples!

+1 on trying coderabbit, it does make the comments a bit noisy, but let's see how it works in our case.

---

Nadia

[Surya Seetharaman]

Hello all, 

Given we have majority maintainers vote and no concerns so far and in this case - unless we try it out we won't know the concerns (example like how Nadia mentioned - it might be more noisier) - I am going to turn on coderabbit.ai only for ovn-kubernetes repo in our org. Let's let it run for a week and then in next upstream meeting we can talk about whether its good and we should keep it OR if we should try something else.

This means coderabbitai has been authorized to have:

• Read access to actions, checks, discussions, members, and metadata
• Read and write access to code, commit statuses, issues, and pull requests

 so that it can do its job. These rights are mandatory for it to be installed.

Side-note: I spoke with CNCF Staff and we got the green light: They said CNCF does not have any specific guidelines as of now, and to use these tools "responsibly" and to let them know if it works out well for us so maybe in the future if it can benefit CNCF projects in general. Currently there is no official agreement for CNCF with coderabbit.ai.

Cheers,

Surya

[Surya Seetharaman]

ANNOUNCEMENT: coderabbit.ai has been turned on.

Some helpful notes:

1. Only new commits added after the app was installed will get coderabbit's attention
2. If you want reviews on your older PRs please use the command @coderabbitai review on your older PR

Let's see how this turns out :)

Again if someone has concerns, please don't hesitate to raise them.

[Dumitru Ceara]

On 6/26/25 10:41 AM, 'Surya Seetharaman' via ovn-kubernetes wrote:
> Hello all, 

Hi Surya,

> Given we have majority maintainers vote and no concerns so far and in
> this case - unless we try it out we won't know the concerns (example
> like how Nadia mentioned - it might be more noisier) - I am going to
> turn on coderabbit.ai only for ovn-kubernetes repo in our org. Let's let
> it run for a week and then in next upstream meeting we can talk about
> whether its good and we should keep it OR if we should try something else.
>

I'm no maintainers but I just happened to see this email thread.

> This means coderabbitai has been authorized to have:
>

> * Read access to actions, checks, discussions, members, and metadata
> * Read and write access to code, commit statuses, issues, and pull

> requests
>
>  so that it can do its job. These rights are mandatory for it to be
> installed.
>

At a first glance "write access to code, commit statuses, issues, and
pull requests" seems very intrusive. My concerns are along the lines
of, is there any form of gatekeeping in place preventing coderabbit.ai
from any of the following?

- pushing ANY kind of code to ANY ovn-kubernetes branch
- editing/deleting PR/issue comments that were not added by coderabbit.ai

I tried to skim the coderabbit.ai documentation and I can't easily find
any detailed information about what this "write access" entails. I
guess you can see exactly what the integration is allowed to "write"
when you enable it in the repo.

It would be nice if none of the two scenarios I mentioned above are allowed.

> Side-note: I spoke with CNCF Staff and we got the green light: They said
> CNCF does not have any specific guidelines as of now, and to use these
> tools "responsibly" and to let them know if it works out well for us so
> maybe in the future if it can benefit CNCF projects in general.
> Currently there is no official agreement for CNCF with coderabbit.ai.
>
> Cheers,
> Surya
>

Regards,
Dumitru

> On Wednesday, June 25, 2025 at 7:11:13 PM UTC+2 n.m.p...@gmail.com wrote:
>
> Hey all, thanks for sharing the examples!
> +1 on trying coderabbit, it does make the comments a bit noisy, but
> let's see how it works in our case.
>
> ---
> Nadia
>
> On Tuesday, 24 June 2025 at 18:06:27 UTC+2 Surya Seetharaman wrote:
>
> On Tue, Jun 24, 2025 at 5:07 PM Tim Rozet <tro...@redhat.com> wrote:
>
> Thanks Surya for sending this out. Here is an example of
> coderabbit:
>

> https://github.com/flightctl/flightctl/pull/1230 <https://
> github.com/flightctl/flightctl/pull/1230>

>
> I think it looks pretty cool. We could always try it and
> then just disable it if it doesn't work for us try something
> else instead.
>
>
>
> Thanks for sharing this Tim. +1 that it looks more cooler than
> sample copilot review: https://github.com/ovn-kubernetes/ovn-

> kubernetes/pull/5313#discussion_r2163248575 <https://github.com/
> ovn-kubernetes/ovn-kubernetes/pull/5313#discussion_r2163248575>

> It seems to be telling us:
>

> 1. code diff
> 2. suggested code diff
> 3. mermaid diagram of code call which is awesome specially for
> bigger PRs
> 4. PR summary
>
> I'm sold on trying out coderabbit.ai <http://coderabbit.ai>. I

> will start with requesting an open source tier account for it as
> indicated in the docs.
> Meanwhile let's see what other maintainers think.
>
> Cheers,
> Surya
>

> --
> You received this message because you are subscribed to the Google
> Groups "ovn-kubernetes" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to ovn-kubernete...@googlegroups.com <mailto:ovn-
> kubernetes+...@googlegroups.com>.

> To view this discussion visit https://groups.google.com/d/msgid/ovn-

> kubernetes/6ae080a1-c2e2-4eaf-9ef9-8e7c1a586608n%40googlegroups.com
> <https://groups.google.com/d/msgid/ovn-kubernetes/6ae080a1-
> c2e2-4eaf-9ef9-8e7c1a586608n%40googlegroups.com?
> utm_medium=email&utm_source=footer>.

[Or Mergi]

Hi Suyra, 

I would like to share a couple of practices for more efficient usage of such tooling in Github, 

following my experience with similar AI tools (Dosue, Sourcery) in other projects (KubeVirt).

1. As mentioned earlier, these bots tend to be very noisy and distracting as they spit their 

opinion on the PR as soon as it's posted. Especially for PRs with proper descriptions and / or

 proper commit messages.

To avoid that, please consider making the bot review optional and not automatic 

(i.e.: reviewers will have to explicitly request for feedback @coderabbitai review)

And the same goes for issues (not sure coderabbit.ai provide such functionality)

2. The bots feedback adds almost zero value (if anything) for backports / cherry-picks and doc 

changes PRs. 

Please consider turning off the bot automatic feedback for such PRs.

I hope you will find it useful :) 

Thanks,

--
You received this message because you are subscribed to the Google Groups "ovn-kubernetes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ovn-kubernete...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/ovn-kubernetes/35e55d36-1f87-44cc-be85-2d7b21bec1adn%40googlegroups.com.

--

Or Mergi

[Surya Seetharaman]

Thanks Dumitru for raising these questions. very valid++

> - pushing ANY kind of code to ANY ovn-kubernetes branch

I know coderabbit can't do this because we have a branch protection rule in place (for sure for master, but need to check for 1.0) where one has to be either a org admin, repo admin, or user with "Maintain role" or one of the members of "ovn-kubernetes-committers" group -> these are the only users with push access from what I can see on the settings.

Based on reading docs (and you are write I didn't find a section where they call out each perm requirement) here is my understanding:

pull-requests: write

is required for posting review comments that has perms for "suggesting changes" which can be committed by the PR owner if they want, but also given coderabbit.ai is enabled through github actions to auto-post on PR when its opened it needs to read the PR but also needs write to comment on the PR. The github actions/automation perms are more explicit than for example when a user logs in.

- editing/deleting PR/issue comments that were not added by coderabbit.ai

This is a great point and something I will look into. As of now as per roles docs for sure anyone with "Write" role in an org/repo they can do these actions. Coderabbit being a github action/auto bot its not a memeber of the org or having a specific repo/org role assigned and is similar to our PR labeller: https://github.com/ovn-kubernetes/ovn-kubernetes/blob/abc2b838aa3d876ddbf7fa0a33bc83c4b17884be/.github/workflows/pr-labeler.yml#L9. But it does have PR:write and read acess. I will open a ticket to make sure we understand this aspect because even in github docs I can't find much about PR perms for workflow, I can only find the information about roles in a repo. However I know that when someone/thing deletes,edits it does leave a trace on github.

[Surya Seetharaman]

Hello Or!

Thank you for sharing your experience.

On Thursday, June 26, 2025 at 12:02:51 PM UTC+2 Or Mergi wrote:

Hi Suyra, 

I would like to share a couple of practices for more efficient usage of such tooling in Github, 

following my experience with similar AI tools (Dosue, Sourcery) in other projects (KubeVirt).

1. As mentioned earlier, these bots tend to be very noisy and distracting as they spit their 

opinion on the PR as soon as it's posted. Especially for PRs with proper descriptions and / or

 proper commit messages.

To avoid that, please consider making the bot review optional and not automatic 

(i.e.: reviewers will have to explicitly request for feedback @coderabbitai review)

And the same goes for issues (not sure coderabbit.ai provide such functionality)

I think this is something to consider for sure.

Like mentioned in my email - note that this is only an experimental phase and based on feedback we can turn it off or makeit optional.

I like the making it optional idea where maintainers or owners of the PR can enabled it explicitly instead of on each PR.

I can raise this point in the next upstream meeting.

The data I am looking for is - is it catching something useful at all or is it all noise. If you have any feedback from your PRs with coderabbit, LMK

 

2. The bots feedback adds almost zero value (if anything) for backports / cherry-picks and doc 

changes PRs. 

Please consider turning off the bot automatic feedback for such PRs.

I hope you will find it useful :) 

yes indeed good points, thanks for sharing these. 

[Surya Seetharaman]

On Thursday, June 26, 2025 at 12:11:27 PM UTC+2 Surya Seetharaman wrote:

Thanks Dumitru for raising these questions. very valid++

- editing/deleting PR/issue comments that were not added by coderabbit.ai

This is a great point and something I will look into. As of now as per roles docs for sure anyone with "Write" role in an org/repo they can do these actions. Coderabbit being a github action/auto bot its not a memeber of the org or having a specific repo/org role assigned and is similar to our PR labeller: https://github.com/ovn-kubernetes/ovn-kubernetes/blob/abc2b838aa3d876ddbf7fa0a33bc83c4b17884be/.github/workflows/pr-labeler.yml#L9. But it does have PR:write and read acess. I will open a ticket to make sure we understand this aspect because even in github docs I can't find much about PR perms for workflow, I can only find the information about roles in a repo. However I know that when someone/thing deletes,edits it does leave a trace on github.

so code rabbit does and can edit comments from others looks like:

Patryk mentioned his PR didn't have description initially and coderabbit added it:

and what it edited is also visible:

as for deleting a comment: Patryk did a test here: https://github.com/ovn-kubernetes/ovn-kubernetes/pull/5324#issuecomment-3008024372 but let's just say we don't trust it can't delete something :)

I have opened helpdesk support tickets with both github and coderabbit to understand permission side of things better.

Cheers,

Surya

 

[Surya Seetharaman]

On Thursday, June 26, 2025 at 12:02:51 PM UTC+2 Or Mergi wrote:

Hi Suyra, 

I would like to share a couple of practices for more efficient usage of such tooling in Github, 

following my experience with similar AI tools (Dosue, Sourcery) in other projects (KubeVirt).

1. As mentioned earlier, these bots tend to be very noisy and distracting as they spit their 

opinion on the PR as soon as it's posted. Especially for PRs with proper descriptions and / or

 proper commit messages.

To avoid that, please consider making the bot review optional and not automatic 

(i.e.: reviewers will have to explicitly request for feedback @coderabbitai review)

Looking more into this there are some knobs we can fine tune to reduce noise and also add

linting and other rules if we want:

Example: https://docs.coderabbit.ai/getting-started/configure-coderabbit/ and https://docs.coderabbit.ai/reference/yaml-template/

I can turn off high_level_summary, commit_status,changed_files_summary, chat:auto_reply etc...

to make it more sober...

[Surya Seetharaman]

Hello All! I got some info from coderabbit support team regarding the questions we had asked around security:

On Thursday, June 26, 2025 at 12:11:27 PM UTC+2 Surya Seetharaman wrote:

Thanks Dumitru for raising these questions. very valid++

> - pushing ANY kind of code to ANY ovn-kubernetes branch

I know coderabbit can't do this because we have a branch protection rule in place (for sure for master, but need to check for 1.0) where one has to be either a org admin, repo admin, or user with "Maintain role" or one of the members of "ovn-kubernetes-committers" group -> these are the only users with push access from what I can see on the settings.

Based on reading docs (and you are write I didn't find a section where they call out each perm requirement) here is my understanding:

pull-requests: write

is required for posting review comments that has perms for "suggesting changes" which can be committed by the PR owner if they want, but also given coderabbit.ai is enabled through github actions to auto-post on PR when its opened it needs to read the PR but also needs write to comment on the PR. The github actions/automation perms are more explicit than for example when a user logs in.

- editing/deleting PR/issue comments that were not added by coderabbit.ai

This is a great point and something I will look into. As of now as per roles docs for sure anyone with "Write" role in an org/repo they can do these actions. Coderabbit being a github action/auto bot its not a memeber of the org or having a specific repo/org role assigned and is similar to our PR labeller: https://github.com/ovn-kubernetes/ovn-kubernetes/blob/abc2b838aa3d876ddbf7fa0a33bc83c4b17884be/.github/workflows/pr-labeler.yml#L9. But it does have PR:write and read acess. I will open a ticket to make sure we understand this aspect because even in github docs I can't find much about PR perms for workflow, I can only find the information about roles in a repo. However I know that when someone/thing deletes,edits it does leave a trace on github.

This is what they said:


CodeRabbit Team <xxx@coderabbit.ai>

4:11 AM (7 hours ago)

to suryaseetharaman.9

Hi,

Thank you for reaching CodeRabbit!

Let me clarify your security concerns below:

1. Can CodeRabbit delete or edit other people’s comments?

No, CodeRabbit cannot delete or edit comments made by other users. It can only post, edit its own comments on Pull Requests (PRs). CodeRabbit is explicitly restricted from modifying any content generated by other contributors or team members. This ensures that your review discussions remain transparent and unaltered.

2. What does "write permission for pull request and code" mean?

The “write permission” requested by CodeRabbit enables it to:

• Post comments on PRs

• Suggest code improvements

• Update its own review messages

However, this does not mean CodeRabbit can push code to your repository or merge changes. CodeRabbit:

• Does not commit or push changes directly to any branch

• Does not merge pull requests

• Respects your repository's permission settings and branching protections

In short, CodeRabbit operates with minimal and scoped access, limited to suggesting and reviewing code only. It follows strict SOC 2-compliant practices to ensure code security and auditability at all times.

If you have any more specific queries, you can reach out to our security team directly at  x...@coderabbit.ai.

Regards,

Team CodeRabbit