Azure Service Discovery for scraping Virtual Machine Scale Sets

[William Glasse]

Hi,

I'm trying to make use of Prometheus's Azure Service Discovery to monitor VMs that live on an Azure Description. I've configured Prometheus to use a managed identity to find all the VMs within a description, however I can't see the VMs with the scrape target list. 

My managed identity is configured with the Reader permission. This seems to be sufficient for scraping VMs that aren't generated from a scale set. My VMs were correctly being scraped by Prometheus.

 However I've recently introduced a scale set, and now the service discovery operation is erroring.

I am getting the following:

ts=2022-09-28T13:36:23.595Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=azure msg="Unable to refresh target groups" err="could not get virtual machine scale set vms: could not list virtual machine scale set vms: compute.VirtualMachineScaleSetVMsClient#List: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code=\"OperationNotAllowed\" Message=\"Operation 'VirtualMachineScaleSets.virtualMachines.GET' is not allowed on Virtual Machine Scale Set 'vmss-my-scale-set'.\""

Any thoughts? I've played around with adding further permissions to the Managed Identity, but to no avail. 

A side question, is the "Reader" permission on the subscription the strictest permission I can apply to accommodate VM scraping with azure service discovery, or can I restrict this down further?