Robot Comments Deprecation

179 views
Skip to first unread message

Ben Rohlfs

unread,
Mar 16, 2023, 6:58:58 AM3/16/23
to Repo and Gerrit Discussion
Hi all,

We are planning to deprecate robot comments in favor of Checks. Please let us know if you have any questions or concerns!

The timeline would be: Deprecation in release 3.8. Removal in release 3.9.

Why are we doing this?
  • There is a huge overlap between Check Results and Robot Comments. It is costly to maintain both and difficult to create a great consistent user experience for both.
  • We are not aware of any use-cases that Check Results cannot cover. Check Results are much more powerful (for example by offering “action” callbacks).
  • From what we hear, most Gerrit servers that use robot comments have also already started integrating with the Checks API.
  • The amount of data produced by CI systems and analyzers is ever growing, and typically these systems already have databases for their results. Gerrit is not the place to store (or even duplicate) this data.
  • The nature of robot comments and checks is that they are rapidly evolving and mostly transient. They can also be produced in very large amounts for every patchset of all changes. As such there is no benefit in persisting the data in Gerrit forever, while quickly becoming stale at the same time. 

Thanks, Ben

Antoine Musso

unread,
Mar 16, 2023, 7:10:51 AM3/16/23
to Ben Rohlfs, Repo and Gerrit Discussion

Hello Ben,

Are robot comments reported under the Findings tab?

I am guessing the equivalent in Checks API is the CheckResult.Fix interface?

-- 
Antoine "hashar" Musso
Release Engineering

Ben Rohlfs

unread,
Mar 16, 2023, 7:32:34 AM3/16/23
to Antoine Musso, Repo and Gerrit Discussion
Correct, "Findings" and "Robot Comments" are synonymous in Gerrit.

I am guessing the equivalent in Checks API is the CheckResult.Fix interface?

And yes, the equivalent of a finding in the Checks API is a CheckResult with a CodePointer and (optionally) a Fix.

Luca Milanesio

unread,
Mar 16, 2023, 8:04:28 AM3/16/23
to Repo and Gerrit Discussion, Luca Milanesio
Hi Ben,
Thanks for sharing this plan.

On 16 Mar 2023, at 11:32, 'Ben Rohlfs' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

On Thu, 16 Mar 2023 at 12:10, Antoine Musso <amu...@wikimedia.org> wrote:
Le 16/03/2023 à 11:58, 'Ben Rohlfs' via Repo and Gerrit Discussion a écrit :
Hi all,

We are planning to deprecate robot comments in favor of Checks. Please let us know if you have any questions or concerns!

The timeline would be: Deprecation in release 3.8. Removal in release 3.9.

Why are we doing this?
  • There is a huge overlap between Check Results and Robot Comments. It is costly to maintain both and difficult to create a great consistent user experience for both.
  • We are not aware of any use-cases that Check Results cannot cover. Check Results are much more powerful (for example by offering “action” callbacks).
  • From what we hear, most Gerrit servers that use robot comments have also already started integrating with the Checks API.
  • The amount of data produced by CI systems and analyzers is ever growing, and typically these systems already have databases for their results. Gerrit is not the place to store (or even duplicate) this data.

100% agree with the amount of data produced: the use of robot comments by the code analyser can generate so much data that is overloading the Git repository and, honestly, should live somewhere else.

I have personally seen changes with *millions of robot comments* (before the introduction of limits in Gerrit) and the challenges were huge.
I’d rather have them integrated with Gerrit but stored somewhere else.
  • The nature of robot comments and checks is that they are rapidly evolving and mostly transient. They can also be produced in very large amounts for every patchset of all changes. As such there is no benefit in persisting the data in Gerrit forever, while quickly becoming stale at the same time.
… and there is only slowdown and pain associated with storing them in NoteDb.

+1 from my side to this move.

Luca.

Hello Ben,

Are robot comments reported under the Findings tab?

Correct, "Findings" and "Robot Comments" are synonymous in Gerrit.

I am guessing the equivalent in Checks API is the CheckResult.Fix interface?

And yes, the equivalent of a finding in the Checks API is a CheckResult with a CodePointer and (optionally) a Fix.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAEWS%2BaOW1gwkmDCncQhMOFMiJYmtBMfmY2QoRF3aNB0co%3D_NAQ%40mail.gmail.com.

Xiao Jia

unread,
Mar 20, 2023, 3:11:41 AM3/20/23
to Repo and Gerrit Discussion
On Thursday, March 16, 2023 at 6:58:58 PM UTC+8 Ben Rohlfs wrote:
Hi all,

We are planning to deprecate robot comments in favor of Checks. Please let us know if you have any questions or concerns!

The timeline would be: Deprecation in release 3.8. Removal in release 3.9.

Why are we doing this?
  • There is a huge overlap between Check Results and Robot Comments. It is costly to maintain both and difficult to create a great consistent user experience for both.
  • We are not aware of any use-cases that Check Results cannot cover. Check Results are much more powerful (for example by offering “action” callbacks).
  • From what we hear, most Gerrit servers that use robot comments have also already started integrating with the Checks API.
Where can we find the documentation of how to integrate with the Checks API?

If I google "gerrit checks api" I only find https://gerrit-review.googlesource.com/Documentation/pg-plugin-checks-api.html which basically forces me to dig into Gerrit source code.

In comparison, robot comments have documented REST APIs which I can (and already) try and see what happens with just curl or simple Python scripts.

Ben Rohlfs

unread,
Mar 20, 2023, 3:29:10 AM3/20/23
to Xiao Jia, Repo and Gerrit Discussion
Hi Xiao,

On Mon, 20 Mar 2023 at 08:11, Xiao Jia <xj...@naivesystems.com> wrote:
On Thursday, March 16, 2023 at 6:58:58 PM UTC+8 Ben Rohlfs wrote:
Hi all,

We are planning to deprecate robot comments in favor of Checks. Please let us know if you have any questions or concerns!

The timeline would be: Deprecation in release 3.8. Removal in release 3.9.

Why are we doing this?
  • There is a huge overlap between Check Results and Robot Comments. It is costly to maintain both and difficult to create a great consistent user experience for both.
  • We are not aware of any use-cases that Check Results cannot cover. Check Results are much more powerful (for example by offering “action” callbacks).
  • From what we hear, most Gerrit servers that use robot comments have also already started integrating with the Checks API.
Where can we find the documentation of how to integrate with the Checks API?

If I google "gerrit checks api" I only find https://gerrit-review.googlesource.com/Documentation/pg-plugin-checks-api.html which basically forces me to dig into Gerrit source code.

Sorry. I was assuming that the general documentation page for how to write JavaScript plugins is referenced from the Checks API docs. Here it is: https://gerrit-review.googlesource.com/Documentation/pg-plugin-dev.html.

You can paste this code snippet into the JavaScript console of your browser to get you started and let you play around with the API a little:

window.Gerrit.install(plugin => {

    const run = {

      checkName: 'Fake Run',

      status: 'COMPLETED',

      results: [

        {category:'ERROR', summary:'Test failed.'}

      ],

    };

    const fetch = () => Promise.resolve({responseCode:'OK', runs:[run]});

    const checksApi = plugin.checks();

    checksApi.register({fetch});

    checksApi.announceUpdate();

  }, undefined, '/plugins/fake-checks-plugin');


The fundamental difference between Robot Comments and Checks is that Gerrit does not store check results. Check results are fetched on-the-fly from other systems when Gerrit loads the change page. So the migration involves 3 tasks:
  • Setup a database to store analyzer results, or find an existing one to integrate with.
  • Create an API that Gerrit can call to retrieve these results.
  • Write a Gerrit TypeScript Plugin that fetches the results and converts them into CheckRuns and CheckResults.
-Ben

In comparison, robot comments have documented REST APIs which I can (and already) try and see what happens with just curl or simple Python scripts.
  • The amount of data produced by CI systems and analyzers is ever growing, and typically these systems already have databases for their results. Gerrit is not the place to store (or even duplicate) this data.
  • The nature of robot comments and checks is that they are rapidly evolving and mostly transient. They can also be produced in very large amounts for every patchset of all changes. As such there is no benefit in persisting the data in Gerrit forever, while quickly becoming stale at the same time. 

Thanks, Ben

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Xiao Jia

unread,
Mar 31, 2023, 2:27:53 AM3/31/23
to Repo and Gerrit Discussion
Hi Ben,

Thanks for your information! I haven't got the time to go through the process of creating a plugin but I'll definitely try in the near future.

On the other hand, considering that our company's product is trying to integrate with our customers' existing Gerrit installations, the robot comments mechanism still has its value in potentially lowering the bar of integration, especially in terms of cognitive overhead, for our customers. Today they just need to create a new service user dedicated to our product and it's done. We lack enough data points to see if their sysadmins are comfortable with installing plugins that are not there out of the box.

You mentioned that it's difficult to keep both features (checks api and robot comments). Could you elaborate on that? I'm very interested in figuring out if there is some middle ground somewhere if any.

Thanks,
Xiao

Ben Rohlfs

unread,
Mar 31, 2023, 2:40:07 AM3/31/23
to Repo and Gerrit Discussion
On Fri, 31 Mar 2023 at 08:28, Xiao Jia <xj...@naivesystems.com> wrote:
Hi Ben,

Thanks for your information! I haven't got the time to go through the process of creating a plugin but I'll definitely try in the near future.

On the other hand, considering that our company's product is trying to integrate with our customers' existing Gerrit installations, the robot comments mechanism still has its value in potentially lowering the bar of integration, especially in terms of cognitive overhead, for our customers. Today they just need to create a new service user dedicated to our product and it's done. We lack enough data points to see if their sysadmins are comfortable with installing plugins that are not there out of the box.

You mentioned that it's difficult to keep both features (checks api and robot comments). Could you elaborate on that? I'm very interested in figuring out if there is some middle ground somewhere if any.

An option that service users will still have is posting normal comments and change messages. That will work just fine, and we also see many systems that prefer that approach for the simplicity that you mentioned.

We think that either using normal comments or check results are good choices, and that there is no need for something additional in between. Maintaining robot comments means that we have to consider it in every refactoring, cleanup, or migration that will happen over the next few years. It will also make UX design more difficult, if we need to keep thinking about where and how we present robot comments to the user. And also users may get confused about what the difference is between comments, robot comments and check results.

We generally believe that it is vital for product and code health to deprecate and remove features from time to time. Huge and feature rich products are harder to maintain and change.

-Ben

 

Clark Boylan

unread,
Mar 31, 2023, 11:03:24 AM3/31/23
to Ben Rohlfs, Repo and Gerrit Discussion
On Thu, Mar 30, 2023 at 11:40 PM 'Ben Rohlfs' via Repo and Gerrit
Discussion <repo-d...@googlegroups.com> wrote:
>
> On Fri, 31 Mar 2023 at 08:28, Xiao Jia <xj...@naivesystems.com> wrote:
>>
>> Hi Ben,
>>
>> Thanks for your information! I haven't got the time to go through the process of creating a plugin but I'll definitely try in the near future.
>>
>> On the other hand, considering that our company's product is trying to integrate with our customers' existing Gerrit installations, the robot comments mechanism still has its value in potentially lowering the bar of integration, especially in terms of cognitive overhead, for our customers. Today they just need to create a new service user dedicated to our product and it's done. We lack enough data points to see if their sysadmins are comfortable with installing plugins that are not there out of the box.
>>
>> You mentioned that it's difficult to keep both features (checks api and robot comments). Could you elaborate on that? I'm very interested in figuring out if there is some middle ground somewhere if any.
>
>
> An option that service users will still have is posting normal comments and change messages. That will work just fine, and we also see many systems that prefer that approach for the simplicity that you mentioned.

What about inline comments from CI systems? Zuul, for example, has the
ability to post linter results inline in Gerrit on the lines that
linters are unhappy about. I'm fairly certain these comments are
posted as robot comments today. Zuul also reports directly about its
own configuration errors in a similar manner. Would we need to report
those as normal comments going forward? Maybe the checks API has some
sort of method for retrieving this information and Zuul would need to
store the results in its own DB for querying by a Checks API plugin?

Ben Rohlfs

unread,
Mar 31, 2023, 11:25:02 AM3/31/23
to Clark Boylan, Repo and Gerrit Discussion
On Fri, 31 Mar 2023 at 17:03, Clark Boylan <clark....@gmail.com> wrote:
On Thu, Mar 30, 2023 at 11:40 PM 'Ben Rohlfs' via Repo and Gerrit
> On Fri, 31 Mar 2023 at 08:28, Xiao Jia <xj...@naivesystems.com> wrote:
>> On the other hand, considering that our company's product is trying to integrate with our customers' existing Gerrit installations, the robot comments mechanism still has its value in potentially lowering the bar of integration, especially in terms of cognitive overhead, for our customers. Today they just need to create a new service user dedicated to our product and it's done. We lack enough data points to see if their sysadmins are comfortable with installing plugins that are not there out of the box.
>>
>> You mentioned that it's difficult to keep both features (checks api and robot comments). Could you elaborate on that? I'm very interested in figuring out if there is some middle ground somewhere if any.
>
>
> An option that service users will still have is posting normal comments and change messages. That will work just fine, and we also see many systems that prefer that approach for the simplicity that you mentioned.

What about inline comments from CI systems? Zuul, for example, has the
ability to post linter results inline in Gerrit on the lines that
linters are unhappy about. I'm fairly certain these comments are
posted as robot comments today. Zuul also reports directly about its
own configuration errors in a similar manner. Would we need to report
those as normal comments going forward? Maybe the checks API has some
sort of method for retrieving this information and Zuul would need to
store the results in its own DB for querying by a Checks API plugin?

Yeah, most robot comments are "inline", so you have to either post normal comments or use check results. For check results to show up inline in diffs you have to set the `codePointer` property: https://cs.opensource.google/gerrit/gerrit/gerrit/+/master:polygerrit-ui/app/api/checks.ts;l=419;drc=84fa59ebd45ddde78a6dfedbe3c159ea04860492

-Ben
Reply all
Reply to author
Forward
0 new messages