As an alternative, I recommend you write a script that can perform these deletions, grant sudo permissions to that script for the "go" user, and then trigger that script via an explicit task command in a job. This is the approach that we tend to take for this specific use case (file deletion).
Are there other actions that your plugin is taking up beyond the deletion of files? You could also consider moving all such specialisations into version-controlled script files. This way, you will not be coupled with the CI system, any user with suitable system privileges can work on and can debug the scripts, and the scripts can be changed (vs task plugins being changed, recompiled and redeployed).