index.maxTerms Value Considerations and Starred Changes

[Clark Boylan]

Hello,

I have a user that has starred more than 1024 items. When they try to
list their starred changes they get this error "Error 400 (Bad
Request): too many terms in query: 1193 terms (max = 1024)". The main
issue for the user is that they cannot find the starred items in order
to unstar them to get back under the limit.

It looks like we can increase index.maxTerms from the default of 1024
to some value over 1193. The documentation warns that search terms are
expensive in terms of performance. Is there a general recommendation
for setting this value? Is it memory or cpu limited? Maybe both? I
think we're likely to increase this configuration value at least
temporarily to allow the user to clean up their stars.

Are there any other ways for the user to list stars in a more limited
fashion in order to see which items are starred to unstar them? Sounds
like the only way they can see if a change is starred is by opening a
change and seeing it on the change page. All change listing summaries
lack this information.

Finally, is this something that can be improved with the star system?
It does seem odd that it would completely break down when you reach
this arbitrary limit. Maybe the system should prevent you from adding
a 1025'd star? Or can the star system be rebuilt to rely less on this
limit?

Thank you,
Clark

[Daniele Sassoli]

Hi Clark,

On Tuesday, 15 August 2023 at 13:14:42 UTC-7 Clark Boylan wrote:

Hello,

I have a user that has starred more than 1024 items. When they try to
list their starred changes they get this error "Error 400 (Bad
Request): too many terms in query: 1193 terms (max = 1024)". The main
issue for the user is that they cannot find the starred items in order
to unstar them to get back under the limit.

It looks like we can increase index.maxTerms from the default of 1024
to some value over 1193. The documentation warns that search terms are
expensive in terms of performance. Is there a general recommendation
for setting this value? Is it memory or cpu limited? Maybe both? I
think we're likely to increase this configuration value at least
temporarily to allow the user to clean up their stars.

 

I'm not  sure of the answer to the specific question, but sounds like a potential

workaround would be for the Gerrit admin to delete at least some of the stars

for this user. Assuming you're on a noteDB version of gerrit, you can do this by

going to `All-Users.git` and doing a
`git update-ref -d refs/starred-changes/<shard-id>/<change-num>/<account number>`.

What I'm calling `shard-id` is just the last 2 digits of the `change-num`. 

Are there any other ways for the user to list stars in a more limited
fashion in order to see which items are starred to unstar them? Sounds
like the only way they can see if a change is starred is by opening a
change and seeing it on the change page. All change listing summaries
lack this information.

Finally, is this something that can be improved with the star system?
It does seem odd that it would completely break down when you reach
this arbitrary limit. Maybe the system should prevent you from adding
a 1025'd star? Or can the star system be rebuilt to rely less on this
limit?

Thank you,
Clark

Hope this helps,

Dani 

[Han-Wen Nienhuys]

Thanks for the report. We never thought someone would star this many
changes (and have been debating on and off if we could simply remove
the feature.)

For context: this data used to be stored as part of the change in the
change index, but the source of truth is the refs that the other
poster alluded to. This caused infrastructural problems, so this was
recently removed from the index.

(Perhaps we should dump the offending query in the error message; that
would also show which changes are causing the problem.)

--
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Liana Sebastian

[Clark Boylan]

On Tue, Aug 15, 2023 at 1:50 PM Daniele Sassoli
<daniele...@gmail.com> wrote:
>
> Hi Clark,
>
> On Tuesday, 15 August 2023 at 13:14:42 UTC-7 Clark Boylan wrote:
>
> Hello,
>
> I have a user that has starred more than 1024 items. When they try to
> list their starred changes they get this error "Error 400 (Bad
> Request): too many terms in query: 1193 terms (max = 1024)". The main
> issue for the user is that they cannot find the starred items in order
> to unstar them to get back under the limit.
>
> It looks like we can increase index.maxTerms from the default of 1024
> to some value over 1193. The documentation warns that search terms are
> expensive in terms of performance. Is there a general recommendation
> for setting this value? Is it memory or cpu limited? Maybe both? I
> think we're likely to increase this configuration value at least
> temporarily to allow the user to clean up their stars.
>
>
> I'm not sure of the answer to the specific question, but sounds like a potential
> workaround would be for the Gerrit admin to delete at least some of the stars
> for this user. Assuming you're on a noteDB version of gerrit, you can do this by
> going to `All-Users.git` and doing a
> `git update-ref -d refs/starred-changes/<shard-id>/<change-num>/<account number>`.

If you go this route do you make the change externally to Gerrit then
push it in order to have Gerrit pick up on the update? Or do you make
it in place and reindex changes?

In our case we bumped the index.maxTerms value up 50% and that seems
to be working so far. That won't prevent someone from starring even
more changes, but we'll cross that bridge when we get there.