Learnings from the feedback about autonag
Suhaib Mujahid
Hello,
Following up on the survey we sent some weeks ago to collect feedback and suggestions about autonag (the friendly bot nagging you on Bugzilla), we want to share our learnings and what we are doing in response. We are glad to hear about all the positive feedback where autonag is already helping. In this email, we will focus on the main areas where we will improve.
Identify important bugs. In order to reduce the backlog, autonag tries to bring attention to old bugs that could be unblocked. However, some of the old bugs could be redundant or not important. We will apply machine learning to identify important and duplicate/similar bugs. This way, we could prioritize important bugs from the backlog and identify common problems and areas for improvement.
Setting the priority level. We got feedback that priority should be up to the team to set, not be automated. Previously, autonag was automatically setting high priority for tracked bugs, following the triage rules; in response to the feedback and given that many teams are not using priority in the “official” way, we disabled this behaviour. Currently, autonag will ask the team to set the priority when there are indications that the bug should have higher priority.
Too much bugmail. Autonag could be noisy when it triggers bugmail for a minor change or unactionable event. Also, it is annoying to receive multiple needinfos\bugmails for the same bug in a row. Thus, we plan to disable bugmail when autonag is performing unactionable changes (e.g., fixing metadata inconsistency). In addition, we will be aggregating actions performed by autonag where possible.
Awareness about autonag. We constantly introduce improvements to autonag; however, the reasoning and the motivation of any new behaviour are not always clear for everybody. In some cases, the new behaviour could be seen as an annoying unnecessary change. To mitigate such problems, we will be adding more justification when autonag comments or sends needinfo requests. Also, where suitable, we will announce significant changes to dev-platform and firefox-dev, with clear reasoning and asking for feedback before enabling new autonag features.
Addressing the correct person. Autonag sometimes needinfoes the triage owners for things out of their responsibilities. To distribute the needinfo requests to the correct people, autonag will adjust to address the team managers when their attention is needed (e.g., bug needs to be assigned). Also, for teams that have internal triage owner rotations (which we are making easier to set up), autonag will address the people in the rotations.
Responsiveness to needinfo requests. Some needinfo requests (set by autonag or other people) wait for a long time without a response. In many cases, these requests are pending on inactive users. To avoid having bugs blocked by needinfo requests that most likely will never be answered, autonag now identifies and handles such pending requests (with various rules to avoid missing important bugs but at the same time be as silent as possible).
We also received additional ideas to implement, we are not listing them all in this email but you can follow the ongoing improvements to autonag by checking the Autonag Improvements project on GitHub.
Thank you,
Suhaib, on behalf of the CI and Quality Tools team.