Issues with service-discovery ui in prometheus >=3

[Meier]

I am looking into upgrading to prometheus 3 and stumbled across an issue with the service discovery UI. For some scrape pools i get the error:

Error: Received target information for an unknown scrape pool, likely the list of scrape pools has changed. Please reload the page.

After digging around a bit in the java-script I suspect that the issue is caused by me setting a `job` label in file_sd configs breaking the `s[u]` lookup in this code since my custom job label is not a valid key.

```
    for (const c of n) {
        const {job: u} = c.discoveredLabels
          , d = s[u];
        if (!d)
            throw new Error("Received target information for an unknown scrape pool, likely the list of scrape pools has changed. Please reload the page.");
        d.total++
    }
    const o = i.length !==
```

Am I correct with this? If so, is there any way I can use a custom `job` label or is that something that should not be done?

Cheers

Christoph

[Meier]

So I am wondering if this can be considered as a bug in prometheus 3.x

The relabel configuration says:
> Initially, aside from the configured per-target labels, a target's job label is set to the job_name value of the respective scrape configuration. 

Which could imply that it is valid to set a custom job-label per job. 

Also the TODO comments in https://github.com/prometheus/prometheus/blob/main/web/ui/mantine-ui/src/pages/service-discovery/ServiceDiscoveryPoolsList.tsx#L115 and https://github.com/prometheus/prometheus/blob/main/web/ui/mantine-ui/src/pages/service-discovery/ServiceDiscoveryPoolsList.tsx#L84 indicate that there maybe is an issue here