Moving the PromQL editor to prometheus/prometheus

Augustin Husson

unread,

Aug 10, 2021, 6:36:12 AM8/10/21

to Prometheus Developers

Hello fellow Prometheus developers :),

As you probably know, in Prometheus, you have since a couple month a great PromQL editor (with autocomplete, linter, highlight feature) which is for the moment maintained in two separate repositories:

prometheus-community/codemirror-promql that contains all the autocomplete / linter / highlight logic.
promlabs/lezer-promql that contains the PromQL grammar (web version)

When a new feature enriched PromQL, the PR on Prometheus' side is usually modifying the backend and the documentation. But it doesn't change the PromQL editor since it's in two different repositories.

It's usually Julius or/and me that are putting back this feature, creating multiple PRs in these repositories, then releasing each to finally be able to create a single PR in prometheus/prometheus which usually just changes the version of codemirror-promql.

This way worked for a couple of times because I was quite reactive on the PromQL features. And now we have the new function present_over_time that is going to be released in v2.29, and the editor is not yet aligned.

So it's proof (at least for me) that this model doesn't work / scale.

What I'm proposing (which is not new, actually Julien already proposed a long time ago), is to merge these two repositories in prometheus/prometheus.

Like that when a PR is changing PromQL it will actually change:

the backend
the docs
the frontend

codemirror-promql is released as a npm package, and it is currently used by some third parties like Victoria Metrics for example.

I think we should keep it as a separate npm package. Which means it won't follow the same release process as Prometheus even if it's in the same repository.

What we are proposing with Julius is to add a special tag like codemirror-promql-0.18.0 that then will trigger a special pipeline to release this npm package.

Finally, the npm package is owned by me, so if you are ok to do what is proposed above, then I will transfer the ownership to Prometheus.

WDYT ? Do you have any particular blocking point that would be against this repository migration ?

Cheers,

Augustin.

Julius Volz

unread,

Aug 10, 2021, 7:16:45 AM8/10/21

to Augustin Husson, Prometheus Developers, Julien Pivotto

I like the idea. I want to make sure that having multiple tag formats for differently-versioned subprojects (Prometheus itself and one or multiple npm packages) doesn't cause any problems I don't foresee. It would be great if people more familiar with the current Prometheus CI / build system could give an opinion on that. CC-ing Julien as I think he has a decent overview over that part, and he is also the default Prometheus server repo maintainer.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAOJizGebmGjD3%2Bde%3Dzb3dGUSSoprV0zk3JdobpAmpQ%2BhFD7uiQ%40mail.gmail.com.

--

Julius Volz

PromLabs - promlabs.com

Augustin Husson

unread,

Aug 10, 2021, 8:17:16 AM8/10/21

to Julius Volz, Julien Pivotto, Prometheus Developers

From my point of view, to have a different tag wasn't because I didn't want to wait for a Prometheus release.

In fact, currently these repositories are for the moment quite in a maintenance mode. It just follows the changes of PromQL basically. So it's quite fine to wait for the Prometheus release to unleash any bugfixes / features.

On my side, my concern regarding following the tag version of Prometheus is more we will release the npm package quite often with no changes. That's something weird to release a library with no changes.

It is still interesting to create UI module to be able to share code between Thanos and Prometheus (I have made a proposal in this sense here, which I think can be improved), but in that particular case, I think the changes will appear quite often and it will be one npm package that would contain all Prometheus module. ( a bit like angular is doing for example). So in that particular case, it makes sense to follow the tag of Prometheus.

In this perspective, I could imagine that the PromQL editor is actually a Prometheus module, but then it will be a different npm package. I could leave with that, as long as it won't be the unique UI module.

Another idea would be to release the npm package during the release process of Prometheus, but the version won't follow the tag, it will follow what is written in the npm package. So if the version didn't change between 2 Prometheus versions, then it won't release the npm package.

Like that we don't have extra git tag, we don't release any extra version with no changes.

WDYT about this last proposition ?

Le mar. 10 août 2021 à 13:29, Julien Pivotto <roidel...@inuits.eu> a écrit :

Hello,

I like the idea to combine them in one repository.

I would rather see if we can use it "unversioned" inside
prometheus/prometheus and release it together with the Prometheus
releases for the outside world.

My concerns are:

- It would add an extra burden to release management if we add extra steps or
more packages
- I expect that some people actually build Prometheus from the tags and
adding extra tags could break quite a few workloads. I do not think
that building tags is a xkcd 1172 case https://xkcd.com/1172/

Additionally, there has been interests in the past to have even more
UI modules available, e.g. for thanos.

I know that it would be quite inconvenient to wait for a Prometheus
release to publish bugfixes for these, but:
1) we release Prometheus quite often
2) we should still try to minimize the code *not used* by Prometheus
itself, so that bugfixes will more likely hit Prometheus as well.

Regards,

On 10 Aug 13:16, Julius Volz wrote:
> I like the idea. I want to make sure that having multiple tag formats for
> differently-versioned subprojects (Prometheus itself and one or multiple
> npm packages) doesn't cause any problems I don't foresee. It would be great
> if people more familiar with the current Prometheus CI / build system could
> give an opinion on that. CC-ing Julien as I think he has a decent overview
> over that part, and he is also the default Prometheus server repo
> maintainer.
>
> On Tue, Aug 10, 2021 at 12:36 PM Augustin Husson <husson....@gmail.com>
> wrote:
>
> > Hello fellow Prometheus developers :),
> >
> > As you probably know, in Prometheus, you have since a couple month a great
> > PromQL editor (with autocomplete, linter, highlight feature) which is for
> > the moment maintained in two separate repositories:
> >

> > - prometheus-community/codemirror-promql
> > <https://github.com/prometheus-community/codemirror-promql> that

> > contains all the autocomplete / linter / highlight logic.

> > - promlabs/lezer-promql <https://github.com/promlabs/lezer-promql>

> > that contains the PromQL grammar (web version)
> >
> > When a new feature enriched PromQL, the PR on Prometheus' side is usually
> > modifying the backend and the documentation. But it doesn't change the
> > PromQL editor since it's in two different repositories.
> > It's usually Julius or/and me that are putting back this feature, creating
> > multiple PRs in these repositories, then releasing each to finally be able
> > to create a single PR in prometheus/prometheus which usually just changes
> > the version of codemirror-promql.
> >
> > This way worked for a couple of times because I was quite reactive on the
> > PromQL features. And now we have the new function present_over_time that is
> > going to be released in v2.29, and the editor is not yet aligned.
> > So it's proof (at least for me) that this model doesn't work / scale.
> >
> > What I'm proposing (which is not new, actually Julien already proposed a
> > long time ago), is to merge these two repositories in prometheus/prometheus.
> > Like that when a PR is changing PromQL it will actually change:
> >

> > - the backend
> > - the docs
> > - the frontend

> >
> > codemirror-promql is released as a npm package, and it is currently used
> > by some third parties like Victoria Metrics for example.
> > I think we should keep it as a separate npm package. Which means it won't
> > follow the same release process as Prometheus even if it's in the same
> > repository.
> >

> > What we are proposing with Julius is to add a special tag like *codemirror-promql-0.18.0
> > *that then will trigger a special pipeline to release this npm package.

> >
> > Finally, the npm package is owned by me, so if you are ok to do what is
> > proposed above, then I will transfer the ownership to Prometheus.
> >
> > WDYT ? Do you have any particular blocking point that would be against
> > this repository migration ?
> >
> > Cheers,
> > Augustin.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Prometheus Developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to prometheus-devel...@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-developers/CAOJizGebmGjD3%2Bde%3Dzb3dGUSSoprV0zk3JdobpAmpQ%2BhFD7uiQ%40mail.gmail.com

> > <https://groups.google.com/d/msgid/prometheus-developers/CAOJizGebmGjD3%2Bde%3Dzb3dGUSSoprV0zk3JdobpAmpQ%2BhFD7uiQ%40mail.gmail.com?utm_medium=email&utm_source=footer>

> > .
> >
>
>
> --
> Julius Volz
> PromLabs - promlabs.com

--
Julien Pivotto
@roidelapluie

Rob Skillington

unread,

Aug 10, 2021, 8:32:55 AM8/10/21

to Augustin Husson, Julien Pivotto, Julius Volz, Prometheus Developers

You could also follow the Kubernetes model where subdirectories of the repository is mirrored to a second repository (either by CI or some other infrastructure) and there the code is tagged.

That way you still have a monorepo of all the code and can make single changes across layers, but the releasing and other versioning aspect is done in a separate repo (and potentially handling issues, etc too).

This is how the k8s client is released separately even though the code lives in the main k8s central repo alongside k8s API server, kubelet, etc.

Rob

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAOJizGfq8PD3CzonpyOMs%2B4O%3Dd65CtSd_3Jr%2ByT-ppk8Q-V_KQ%40mail.gmail.com.

Augustin Husson

unread,

Aug 10, 2021, 9:10:21 AM8/10/21

to Rob Skillington, Julien Pivotto, Julius Volz, Prometheus Developers

Oh yeah, I like this idea ! Thanks Rob :).

I think that would cover all concerns raised for the moment, right Julien and Julius ?

I forgot to mention it, but yes of course Julien the prometheus web ui will use the local version of the codemirror-promql and won't use the npm package.

Julius Volz

unread,

Aug 10, 2021, 10:31:09 AM8/10/21

to Augustin Husson, Rob Skillington, Julien Pivotto, Prometheus Developers

I would be fine with that as well, yeah. It's more overall complexity having to sync things, but then at least that extra complexity doesn't concern the main Prometheus repo :)