Moving Elasticsearch High Level REST client to Quarkiverse

[Guillaume Smet]

Hi,

Just to be clear: this email only concerns the high level client and NOT the low level one.

As you may know, Elastic has decided to make Elasticsearch proprietary software. The low level client is kept Open Source but the High Level one (which depends on most of Elasticsearch due to an architecture issue) has been moved to the new license.

For now, we haven't upgraded nor the low level client nor the high level client as they are kept in sync and we don't want to include the non-Open Source version.

Problem is: I want to upgrade the low level one to the latest version for 2.7 and we have absolutely no idea if the old version of the high level client will work with it (and I really don't want us to somehow guarantee it given it won't be fully tested).

Thus I would like to move the high level client to the Quarkiverse hub (and people using it can then downgrade the low level client version if they want or do their own testing on the particular features they use). I would like to do this for 2.7 so very soon.

Elastic has developed a new Open Source high level client that we could include in a later version (or for 2.7 if we have an extension ready). It has a different name so I would expect the extension to have a different name too.

I already discussed this with @Yoann Rodiere so @Loïc MATHIEU this is mostly for your awareness and discussing it with you.

Thanks.

--

Guillaume

[George Gastaldi]

+1

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALt0%2Bo8GG4kCfz9k1zBX6T3%2B3uVQyAGJXhaLunNSt1Xef1hXbQ%40mail.gmail.com.

[Georgios Andrianakis]

+1

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJigAfA00iKiBPbTxrHOqPwPPZaAR7LnpuqO3vx1gRbb_w%40mail.gmail.com.

[Max Rydahl Andersen]

Sounds reasonable to me.

Just for due diligence - Do we know of anything actually using the high-level client ?

/max

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALt0%2Bo8GG4kCfz9k1zBX6T3%2B3uVQyAGJXhaLunNSt1Xef1hXbQ%40mail.gmail.com.

/max
https://xam.dk/about

[Guillaume Smet]

On Mon, Dec 20, 2021 at 6:29 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

Sounds reasonable to me.

Just for due diligence - Do we know of anything actually using the high-level client ?

Not really. That doesn't mean it's not used. Loïc might use it for instance.

But in any case, we have proven we can move things to the Quarkiverse with very little perturbation for the users. Also note that the High Level client is gone from Elasticsearch 7.16 and it's very tied to Elasticsearch versions so people will have to move to the new client.

We can't really keep it around.

--

Guillaume

[Max Rydahl Andersen]

>> Sounds reasonable to me.
>>
>> Just for due diligence - Do we know of anything actually using the
>> high-level client ?
>>
> Not really. That doesn't mean it's not used. Loïc might use it for instance.
>
> But in any case, we have proven we can move things to the Quarkiverse with
> very little perturbation for the users.

wait - so maybe I misunderstood your ask - we only can make it easy to move to quarkiverse
when staying in platform. I thought the ask here is to move to quarkiverse AND outside the platform ?

> Also note that the High Level
> client is gone from Elasticsearch 7.16 and it's very tied to Elasticsearch
> versions so people will have to move to the new client.
>
> We can't really keep it around.

I'm not asking to keep it around - just trying to grok where the move have impact and if need to make it more visible.

If its just to quarkiverse it doesn't matter much - its only if out of platform it becomes a more impactful thing;
again in this specific case if you don't know of much usage and with the annoying elasticsearch license + possible upgrade issue its better it moves out ASAP.

/max
https://xam.dk/about

[Guillaume Smet]

On Mon, Dec 20, 2021 at 6:45 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

wait - so maybe I misunderstood your ask - we only can make it easy to move to quarkiverse
when staying in platform. I thought the ask here is to move to quarkiverse AND outside the platform ?

That's not true. We installed relocations for all the moved artifacts. And we keep versions for these artifacts in the Quarkus Core BOM as long as we will keep the relocations.

Thus, when you move to 2.6.0.Final, you have a warning asking you to move to the Quarkiverse artifacts but your application still works as Maven handles the relocation.

 

> Also note that the High Level
> client is gone from Elasticsearch 7.16 and it's very tied to Elasticsearch
> versions so people will have to move to the new client.
>
> We can't really keep it around.

I'm not asking to keep it around - just trying to grok where the move have impact and if need to make it more visible.

If its just to quarkiverse it doesn't matter much - its only if out of platform it becomes a more impactful thing;
again in this specific case if you don't know of much usage and with the annoying elasticsearch license + possible upgrade issue its better it moves out ASAP.

Again, we cannot keep it. It's dead from an Elastic POV. So I'm not proposing to move it to the Platform. I'm proposing to move it out to the Quarkiverse Hub in case some users need it until they can migrate to the new one, that we might support one day.

--

Guillaume

[Max Rydahl Andersen]

On 20 Dec 2021, at 19:07, Guillaume Smet wrote:

> On Mon, Dec 20, 2021 at 6:45 PM Max Rydahl Andersen <mand...@redhat.com>
> wrote:
>
>> wait - so maybe I misunderstood your ask - we only can make it easy to
>> move to quarkiverse
>> when staying in platform. I thought the ask here is to move to quarkiverse
>> AND outside the platform ?
>>
>
> That's not true. We installed relocations for all the moved artifacts. And
> we keep versions for these artifacts in the Quarkus Core BOM as long as we
> will keep the relocations.

hmm - I didn't realise we kept these in quarkus core bom - how is that not creating
some weird funky cyclic dependency preventing us from releasing one without the other ?

The original idea was to remove them from Quarkus core repo (and Quarkus core bom) and
add them both in the Quarkus Platform not the Quarkus Core BOM as otherwise it create a funky cyclic dependency.
If we don't remove them from core bom we don't really move closer to be able to split/rearrange
Quarkus core more freely.

How does keeping it in core actually work release wise ? :)

> Thus, when you move to 2.6.0.Final, you have a warning asking you to move
> to the Quarkiverse artifacts but your application still works as Maven
> handles the relocation.
>
>
>>> Also note that the High Level
>>> client is gone from Elasticsearch 7.16 and it's very tied to
>> Elasticsearch
>>> versions so people will have to move to the new client.
>>>
>>> We can't really keep it around.
>>
>> I'm not asking to keep it around - just trying to grok where the move have
>> impact and if need to make it more visible.
>>
>> If its just to quarkiverse it doesn't matter much - its only if out of
>> platform it becomes a more impactful thing;
>> again in this specific case if you don't know of much usage and with the
>> annoying elasticsearch license + possible upgrade issue its better it moves
>> out ASAP.
>>
>
> Again, we cannot keep it. It's dead from an Elastic POV. So I'm not
> proposing to move it to the Platform. I'm proposing to move it out to the
> Quarkiverse Hub in case some users need it until they can migrate to the
> new one, that we might support one day.

But if they are still in the platform they aren't moved out and still need to as compatible.

Thats what I don't get - if you want it out to upgrade to higher version without bumping elastic high level
you need to remove it from the BOM (both core and platform).

This is not a case where keeping the relocations in place is useful as the platform publishing
will force align dependencies of extensions in the platform.

Or am I missing some other trick we found out lately ? :)

/max

> --
> Guillaume
>
> --
> You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALt0%2Bo-Gv-700ZTjTyndz991fJ5gpfmkVvoPP0p0swovUvGLeA%40mail.gmail.com.

/max
https://xam.dk/about

[Loïc MATHIEU]

Hi,

I don't think moving to Quarkiverse an extension to then deprecate it and archive the repository a few weeks after is a good thing, it gives a false sense of what is the Quarkiverse, this is not a cemetery of dead extensions.

If we don't want to support the high level client, and I agree we shouldn't, we must deprecate it and remove it in a few release cycles where it is (so in the core).

Speaking of who uses the high level client: me and my team (in production) and I assume almost all applications that connect to Elasticsearch directly as using the low level client is not convenient and the high level client is the recommended way by Elastic since a long time.

There is at least one extension that use it: the reactive elasticsearch client: https://github.com/quarkiverse/quarkus-elasticsearch-reactive

Moreover, the low level and the high level client are tightly coupled: they share the same version, the same dependencies, the same config options, the same config class, the same health check, the same native support, ... The high level extension is just a simple CDI producer that constructs a high level client based on a low level one. So if you move one to the quarkiverse you should move the other or the maintenance would be way more complex and risky. Both versions will always need to be aligned.

What I propose is to :

- Move to Elasticsearch 7.16

- Mark the high level client as deprecated with a notice that it will be removed on a subsequent release and that the Java client is the way to go

- Create an extension for the Java client in core, it'll be trivial to build as it can be created with a low level client (see https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/connecting.html), I can have a look later (next week).

- Create an extension for Opensearch (it can be in the Quarkiverse): it's easy to build based on the one for Elasticsearch (copy/paste and change the packages), it doesn't need to be there for 2.7. I can work on it (I use an Opensearch cluster so I can easily test it).

- I'll also discuss with the Vert.x team to have a wrapper based on the Java client but this can wait a little.

Keep in mind that the 7.16 Elasticsearch client will not allow you to communicate with an Opensearch server, to mitigate the issue while sorting things out we can use the 7.13.4 instead as it'll work with both (but is not Apache 2 licensed), or users would need to do this themselves, we can document it or not as it'll be a transient situation while waiting for the missing clients.

I'm on PTO so it may take me some time to answer, please don't hurry on this.

Regards,

Loïc

[George Gastaldi]

+1000 to this

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJLxjVFqmA6W2Z_MDJMbD-rBhSsCpkn1A6VazSQD-3LygE_K%3Dw%40mail.gmail.com.

[Guillaume Smet]

On Tue, Dec 21, 2021 at 9:54 AM Max Rydahl Andersen <mand...@redhat.com> wrote:

hmm - I didn't realise we kept these in quarkus core bom - how is that not creating
some weird funky cyclic dependency preventing us from releasing one without the other ?

The original idea was to remove them from Quarkus core repo (and Quarkus core bom) and
add them both in the Quarkus Platform not the Quarkus Core BOM as otherwise it create a funky cyclic dependency.
If we don't remove them from core bom we don't really move closer to be able to split/rearrange
Quarkus core more freely.

Hmmm, maybe you misunderstood but given I came up with this whole idea of moving the extensions, I suppose I would be the one knowing what the original idea was. And Alexey and I were on the same page on this.

We don't add the Quarkiverse extensions in the BOM. We keep the artifacts created by the relocations in the BOM - exactly as we did for the extensions we already moved:

https://github.com/quarkusio/quarkus/blob/main/bom/application/pom.xml#L5309

This way, if you have in your application pom:

io.quarkus:quarkus-tika

it's resolved to:

io.quarkus:quarkus-tika:2.6.0.Final because of the BOM

which is a relocation artifact

which then gets redirected to:

io.quarkiverse.tika:quarkus-tika:1.0.2 because of the relocation.

These elements are in a special section of the BOM and will be removed when we drop the relocations.

How does keeping it in core actually work release wise ? :)

The only problem you can have is if we have something in Core that will break the Quarkiverse extensions. In this case, I will base the extension build on the CR1 and the relocation will be updated to this version before the Final Core release.

 

> Again, we cannot keep it. It's dead from an Elastic POV. So I'm not
> proposing to move it to the Platform. I'm proposing to move it out to the
> Quarkiverse Hub in case some users need it until they can migrate to the
> new one, that we might support one day.

But if they are still in the platform they aren't moved out and still need to as compatible.

Thats what I don't get - if you want it out to upgrade to higher version without bumping elastic high level
you need to remove it from the BOM (both core and platform).

This is not a case where keeping the relocations in place is useful as the platform publishing
will force align dependencies of extensions in the platform.

You are talking about the Elasticsearch high level client dependency, I'm talking about the Quarkus extension for it.

The  Elasticsearch high level client will be entirely gone from the BOM - be it the Core BOM or the Platform BOM, we don't want it there. What will be in the Core BOM for a while is the corresponding Quarkus extension so that the relocation can work properly. It will just define the version used to resolve the relocation artifact.

Until we get rid of the relocation.

-- 

Guillaume

[Guillaume Smet]

On Tue, Dec 21, 2021 at 11:11 AM Loïc MATHIEU <loik...@gmail.com> wrote:

I don't think moving to Quarkiverse an extension to then deprecate it and archive the repository a few weeks after is a good thing, it gives a false sense of what is the Quarkiverse, this is not a cemetery of dead extensions.

Seriously, can we stop with this?

 

If we don't want to support the high level client, and I agree we shouldn't, we must deprecate it and remove it in a few release cycles where it is (so in the core).

No. It's in the way for us to upgrade the low level client so we cannot keep it in Core. And I want the low level client updated for 2.7 as I want to make sure Hibernate Search uses the latest version of the low level client.

 

Speaking of who uses the high level client: me and my team (in production) and I assume almost all applications that connect to Elasticsearch directly as using the low level client is not convenient and the high level client is the recommended way by Elastic since a long time.

There is at least one extension that use it: the reactive elasticsearch client: https://github.com/quarkiverse/quarkus-elasticsearch-reactive

Moreover, the low level and the high level client are tightly coupled: they share the same version, the same dependencies, the same config options, the same config class, the same health check, the same native support, ... The high level extension is just a simple CDI producer that constructs a high level client based on a low level one. So if you move one to the quarkiverse you should move the other or the maintenance would be way more complex and risky. Both versions will always need to be aligned.

What I propose is to :

- Move to Elasticsearch 7.16

- Mark the high level client as deprecated with a notice that it will be removed on a subsequent release and that the Java client is the way to go 

- Create an extension for the Java client in core, it'll be trivial to build as it can be created with a low level client (see https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/connecting.html), I can have a look later (next week).

- Create an extension for Opensearch (it can be in the Quarkiverse): it's easy to build based on the one for Elasticsearch (copy/paste and change the packages), it doesn't need to be there for 2.7. I can work on it (I use an Opensearch cluster so I can easily test it).

- I'll also discuss with the Vert.x team to have a wrapper based on the Java client but this can wait a little.

Please read my email, you're missing the point:

- we don't want to keep using the old version of the low level client for 2.7

- we cannot upgrade the high level client to 7.16.2 in Quarkus: its license is far too problematic for us to do that, anyone not paying Elastic would be in serious troubles

So that leaves us no choice than to move the existing high level client out of Quarkus.

Note that I'm not the one deprecating it or declaring it's a dead end:

https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.15/java-rest-high.html

The two options are:

- move it to the Quarkiverse and hope it works for a while with an old high level client version and the updated low level client version

- drop it entirely (which is what I'm trying to avoid)

The third option that I don't want to consider is to keep both in Core with different versions. Because when it breaks, I don't want it to prevent me from releasing the Core.

 

Keep in mind that the 7.16 Elasticsearch client will not allow you to communicate with an Opensearch server, to mitigate the issue while sorting things out we can use the 7.13.4 instead as it'll work with both (but is not Apache 2 licensed), or users would need to do this themselves, we can document it or not as it'll be a transient situation while waiting for the missing clients.

That's not what Yoann told me. He's using the low level client to test Hibernate Search with OpenSearch.

What will NOT work probably is the new Java "high level" client. But that's another subject entirely and that's not the point of this discussion. We will need new clients for sure.

Also note that I'm not the one having created this mess, I'm just trying to find an acceptable solution given the license mess Elastic created for themselves and also given our constraints to have an updated client for Hibernate Search.

-- 

Guillaume

[Guillaume Smet]

Also, please, folks, try to understand the big picture here and the problem we are trying to solve.

I'm not proposing this for fun, I will be the one doing the work and it's definitely not fun work. I know it's not ideal but, unfortunately, I don't see any other solutions that would be acceptable given the constraints we have:

- update the low level client to the latest version for 2.7

- do NOT put companies using the high level client at risk of legal issues -> we can't upgrade the high level client

- do NOT break the Core release when the new low level client will break the old high level client - which might or might not happen but I really don't want to take this risk

That's what we are trying to solve. Anything not solving this whole set of issues is not an acceptable solution.

-- 

Guillaume

[Max Rydahl Andersen]

On 21 Dec 2021, at 12:48, Guillaume Smet wrote:

> On Tue, Dec 21, 2021 at 9:54 AM Max Rydahl Andersen <mand...@redhat.com>
> wrote:
>
>> hmm - I didn't realise we kept these in quarkus core bom - how is that not
>> creating
>> some weird funky cyclic dependency preventing us from releasing one
>> without the other ?
>>
>> The original idea was to remove them from Quarkus core repo (and Quarkus
>> core bom) and
>> add them both in the Quarkus Platform not the Quarkus Core BOM as
>> otherwise it create a funky cyclic dependency.
>> If we don't remove them from core bom we don't really move closer to be
>> able to split/rearrange
>> Quarkus core more freely.
>>
>
> Hmmm, maybe you misunderstood but given I came up with this whole idea of
> moving the extensions, I suppose I would be the one knowing what the
> original idea was. And Alexey and I were on the same page on this.

I'm talking about the original idea of listing relocations and the new ones in the same platform
to allow warnings + use of new one.

Anyhow I hadn't expected relocations to show up in core bom (even if only via relocations) but
thats the reality now so thats where we are :)

> We don't add the Quarkiverse extensions in the BOM. We keep the artifacts
> created by the relocations in the BOM - exactly as we did for
> the extensions we already moved:
> https://github.com/quarkusio/quarkus/blob/main/bom/application/pom.xml#L5309
>
> This way, if you have in your application pom:
> io.quarkus:quarkus-tika
> it's resolved to:
> io.quarkus:quarkus-tika:2.6.0.Final because of the BOM
> which is a relocation artifact
> which then gets redirected to:
> io.quarkiverse.tika:quarkus-tika:1.0.2 because of the relocation.
>
> These elements are in a special section of the BOM and will be removed when
> we drop the relocations.

Yeah - I just expected that to have been done in the Quarkus Platform not core.

> How does keeping it in core actually work release wise ? :)
>>
>
> The only problem you can have is if we have something in Core that will
> break the Quarkiverse extensions. In this case, I will base the extension
> build on the CR1 and the relocation will be updated to this version before
> the Final Core release.

Yeah - that would work. I just hoped we started to "break" extensions relying on quarkus core bom
and just keep users that should be using published quarkus platform would get the benefit
of friendly handholding.

Advantage is more will get the warnings - but does mean we haven't seen the actual impact
yet.

>>> Again, we cannot keep it. It's dead from an Elastic POV. So I'm not
>>> proposing to move it to the Platform.

>>> I'm proposing to move it out to the
>>> Quarkiverse Hub in case some users need it until they can migrate to the
>>> new one, that we might support one day.
>>
>> But if they are still in the platform they aren't moved out and still need
>> to as compatible.
>>
>> Thats what I don't get - if you want it out to upgrade to higher version
>> without bumping elastic high level
>> you need to remove it from the BOM (both core and platform).
>
>
>> This is not a case where keeping the relocations in place is useful as the
>> platform publishing
>> will force align dependencies of extensions in the platform.
>>
>
> You are talking about the Elasticsearch high level client dependency, I'm
> talking about the Quarkus extension for it.
>
> The Elasticsearch high level client will be entirely gone from the BOM -
> be it the Core BOM or the Platform BOM, we don't want it there. What will
> be in the Core BOM for a while is the corresponding Quarkus extension so
> that the relocation can work properly. It will just define the version used
> to resolve the relocation artifact.

Yes I get that - I'm just not seeing how that avoids the issue you are trying
to avoid - that the high level client won't work with the aligned dependency for the
low level client and thus break the quarkus high level extension.

If its not used and it is known liability just seems clearer/better to remove it
and move it to be truly external for the platform?

/max

>
> Until we get rid of the relocation.
>
> --
> Guillaume

/max
https://xam.dk/about

[Max Rydahl Andersen]

On 21 Dec 2021, at 13:19, Guillaume Smet wrote:

Also, please, folks, try to understand the big picture here and the problem
we are trying to solve.

I'm not proposing this for fun, I will be the one doing the work and it's
definitely not fun work. I know it's not ideal but, unfortunately, I don't
see any other solutions that would be acceptable given the constraints we
have:
- update the low level client to the latest version for 2.7
- do NOT put companies using the high level client at risk of legal issues
-> we can't upgrade the high level client
- do NOT break the Core release when the new low level client will break
the old high level client - which might or might not happen but I really
don't want to take this risk

...for that last item to work don't we need to remove the high-level client extension from the POM ?
It might not break the core release but it will mean the core bom and the platform released is inconsistent.

/max
https://xam.dk/about

[Max Rydahl Andersen]

On 21 Dec 2021, at 13:05, Guillaume Smet wrote:

> Also note that I'm not the one having created this mess, I'm just trying to find an acceptable solution given the license mess Elastic created for themselves and also given our constraints to have an updated client for Hibernate Search.

Very much aware of this - hence why we want to fix it in Quarkus and see what we can do for effected extensions (if anything).

Loic - you are right in saying the high level client is referenced in https://github.com/quarkiverse/quarkus-elasticsearch-reactive/blob/main/deployment/pom.xml#L14 which it would be interesting to understand how we upgrade that if quarkus core still lock in versions of the high level client dependencies.

quarkus-elasticsearch-reactive should start using quarkiverse high level client or being moved to the lower one directly ?

/max
https://xam.dk/about

[Guillaume Smet]

On Tue, Dec 21, 2021 at 1:30 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

I'm talking about the original idea of listing relocations and the new ones in the same platform
to allow warnings + use of new one.

Anyhow I hadn't expected relocations to show up in core bom (even if only via relocations) but
thats the reality now so thats where we are :)

The decision about the Platform is orthogonal to the relocations.

What being in the Platform brings is that you don't need to define a version for the moved extensions and that you are sure the extensions are consistent and tested.

That's entirely different from getting the relocations to actually work. The relocations wouldn't work at all without keeping the extensions in the BOM. That's what we did for the ones we previously moved.

There are no reasons to break people if we can avoid it at no cost - note that we have a lot of early users using the Core BOM: for a long time, we haven't promoted the Platform BOM that much and we weren't releasing the platform for CRs, so people not needing it moved to the Core BOM. As for these non-platform extensions, they get a proper warning to handle the update. And things will actually get broken in a few releases when we remove the relocations.

But keep in mind that Platform extensions will also be broken once we remove the relocations.

 

> We don't add the Quarkiverse extensions in the BOM. We keep the artifacts
> created by the relocations in the BOM - exactly as we did for
> the extensions we already moved:
> https://github.com/quarkusio/quarkus/blob/main/bom/application/pom.xml#L5309
>
> This way, if you have in your application pom:
> io.quarkus:quarkus-tika
> it's resolved to:
> io.quarkus:quarkus-tika:2.6.0.Final because of the BOM
> which is a relocation artifact
> which then gets redirected to:
> io.quarkiverse.tika:quarkus-tika:1.0.2 because of the relocation.
>
> These elements are in a special section of the BOM and will be removed when
> we drop the relocations.

Yeah - I just expected that to have been done in the Quarkus Platform not core.

That makes no sense to knowingly break all applications/extensions using the Core BOM (and there are a lot).

People will have a few releases to fix things. But yes, I expect some will miss it and will discover it in a few releases when we remove the relocations. But at least, some users will have migrated with no pain and that's a win.

Yeah - that would work. I just hoped we started to "break" extensions relying on quarkus core bom
and just keep users that should be using published quarkus platform would get the benefit
of friendly handholding.

Advantage is more will get the warnings - but does mean we haven't seen the actual impact
yet.

Well, you can't have smooth upgrades and know what breaks. From all the discussions we had, people wanted smooth upgrades and I think that's the best for our user base.

But if you want to discuss this further, let's do it in another thread.

Yes I get that - I'm just not seeing how that avoids the issue you are trying
to avoid - that the high level client won't work with the aligned dependency for the
low level client and thus break the quarkus high level extension.

If its not used and it is known liability just seems clearer/better to remove it
and move it to be truly external for the platform?

It might work for a while. I thought it was better to have a possibly workable alternative for a while rather than dropping it entirely.

-- 

Guillaume 

[Max Rydahl Andersen]

On 21 Dec 2021, at 13:45, Guillaume Smet wrote:

> On Tue, Dec 21, 2021 at 1:30 PM Max Rydahl Andersen <mand...@redhat.com>
> wrote:
>
>> I'm talking about the original idea of listing relocations and the new
>> ones in the same platform
>> to allow warnings + use of new one.
>>
>> Anyhow I hadn't expected relocations to show up in core bom (even if only
>> via relocations) but
>> thats the reality now so thats where we are :)
>>
>
> The decision about the Platform is orthogonal to the relocations.

We'll have to agree to disagree on that as it creates these cyclic dependencies
that gets harder to maintain over time.

> What being in the Platform brings is that you don't need to define a
> version for the moved extensions and that you are sure the extensions are
> consistent and tested.

Yes.

> That's entirely different from getting the relocations to actually work.
> The relocations wouldn't work at all without keeping the extensions in the
> BOM. That's what we did for the ones we previously moved.
>
> There are no reasons to break people if we can avoid it at no cost - note
> that we have a lot of early users using the Core BOM: for a long time, we
> haven't promoted the Platform BOM that much and we weren't releasing the
> platform for CRs, so people not needing it moved to the Core BOM.

Yes, I know - thats the reason why we have to do extra gymnastics and
it gets tricky to move things - because the Quarkus core BOM is used to both
align Quarkus core build dependencies as well as extension writers and endusers.

The introduction of platform in 1.0 was (partly) to at least get endusers to use
the platform by default rather than Quarkus core BOM.

In 2.0 thats been the default and we now have the setup to do core and platform
releases close together even for CRs.

> As for
> these non-platform extensions, they get a proper warning to handle the
> update. And things will actually get broken in a few releases when we
> remove the relocations.
> But keep in mind that Platform extensions will also be broken once we
> remove the relocations.

Yes, so what I originally suggested was to do the above but break core bom,
protect users using the Quarkus platforms.

Anyhow - now we protect all the way.

>>> We don't add the Quarkiverse extensions in the BOM. We keep the artifacts
>>> created by the relocations in the BOM - exactly as we did for
>>> the extensions we already moved:
>>>
>> https://github.com/quarkusio/quarkus/blob/main/bom/application/pom.xml#L5309
>>>
>>> This way, if you have in your application pom:
>>> io.quarkus:quarkus-tika
>>> it's resolved to:
>>> io.quarkus:quarkus-tika:2.6.0.Final because of the BOM
>>> which is a relocation artifact
>>> which then gets redirected to:
>>> io.quarkiverse.tika:quarkus-tika:1.0.2 because of the relocation.
>>>
>>> These elements are in a special section of the BOM and will be removed
>> when
>>> we drop the relocations.
>>
>> Yeah - I just expected that to have been done in the Quarkus Platform not
>> core.
>>
>
> That makes no sense to knowingly break all applications/extensions using
> the Core BOM (and there are a lot).

Well, the original move of extensions (as I understood it) would have broken all
of them if we didn't keep them in platform - but now I see that you might then been suggesting
to keep them both in BOM and platform with the cyclic dependency. That's what
I did not expect - and still don't like as we then really don't fix the issue of
everything is over-reliant on quarkus core bom.

> People will have a few releases to fix things. But yes, I expect some will
> miss it and will discover it in a few releases when we remove the
> relocations. But at least, some users will have migrated with no pain and
> that's a win.
>
> Yeah - that would work. I just hoped we started to "break" extensions
>> relying on quarkus core bom
>> and just keep users that should be using published quarkus platform would
>> get the benefit
>> of friendly handholding.
>>
>> Advantage is more will get the warnings - but does mean we haven't seen
>> the actual impact
>> yet.
>>
>
> Well, you can't have smooth upgrades and know what breaks. From all the
> discussions we had, people wanted smooth upgrades and I think that's the
> best for our user base.
>
> But if you want to discuss this further, let's do it in another thread.

sure.

> Yes I get that - I'm just not seeing how that avoids the issue you are
>> trying
>> to avoid - that the high level client won't work with the aligned
>> dependency for the
>> low level client and thus break the quarkus high level extension.
>>
>> If its not used and it is known liability just seems clearer/better to
>> remove it
>> and move it to be truly external for the platform?
>>
>
> It might work for a while. I thought it was better to have a possibly
> workable alternative for a while rather than dropping it entirely.

Okey - so in this case of high-level client you are suggesting that the platform won't
make sure the extensions are consistent and tested.

If that is the middle ground we do then sure - still good if we can outline to get
extensions like https://github.com/quarkiverse/quarkus-elasticsearch-reactive to
move to the quarkiverse extensions when the quarkus core BOM still by default locks the dependencies.

They would need to override everything related to the high-level client.

/max

[Guillaume Smet]

On Tue, Dec 21, 2021 at 1:37 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

Loic - you are right in saying the high level client is referenced in https://github.com/quarkiverse/quarkus-elasticsearch-reactive/blob/main/deployment/pom.xml#L14 which it would be interesting to understand how we upgrade that if quarkus core still lock in versions of the high level client dependencies.

quarkus-elasticsearch-reactive should start using quarkiverse high level client or being moved to the lower one directly ?

It's important you get the distinction between the low level client and the high level one. They are not interchangeable really.

The low level client is a glorified HTTP client, really not much more. You send all requests manually and you deal with responses the same way. It's a good option for low level stuff or if you want to support several Elasticsearch versions so using it for libraries makes sense.

But that's pretty much it. If you have to talk with Elasticsearch in your application, it's really a pain to use this one as it's too low level. The high level one is far more attractive thus why Loïc based his reactive client on the high level one. The name of the extension is not ideal though. And also the reason why people talking to Elasticsearch probably use the high level client extension. It's for an end user the best alternative.

The big issue with the high level one is that it brings nearly all Elasticsearch with it as dependencies, thus why it was licensed with the new license when they changed the license. It was also one of the reasons why several of us were against including it as its architecture was really bad, especially for Quarkus where we try to reduce the dependencies. We decided to include it as Elastic said they would fix it. Well, they fixed it but with a whole new client.

The new high level client (which is a totally different code base) doesn't have this issue and this is the one we should include now. My guess is that it probably won't work with OpenSearch (or at least not guaranteed to work) though so we will have to have different clients.

I have no idea what OpenSearch will do regarding this high level stuff.

Did I say it's a mess?

--

Guillaume

[Loïc MATHIEU]

Hi,

I agree it's a mess ;)

I think I misunderstand what you wanted to do, you want to keep the high level on version 7.10 and make it use a low level on version 7.16 ?

I'm not sure it'll fully work, and as we cannot test hundreds of API methods I'm not sure it's a good idea to mix versions. I think it'll preferable to keep the high level and the low level version aligned, so if you want to upgrade the low level and move the high level to the Quarkiverse, you'll need to duplicate the low level (and the common extension) in the Quarkiverse so the high level will use a low level client on the same version. It is a lot of copy pasting but it should not be difficult.

For the record, other microservices frameworks decided to upgrade the high level to 7.16 despite the new license.

Regarding the reactive client, it is experimental and not in the platform, let's keep it aside for now, I'll talk to Julien and we will see. 

The version of the high level client it uses comes from the version of the Vert.x client, it cannot use the low level client.

I volunteer to :

- Create an extension for the new Java client, should it be in core with the low level ? (yes for me as they have the same lifecycle but it's up to you).

- Create an extension (or 2 or 3 it's not yet clear) for Opensearch : currently it's copy/paste of the Elasticsearch one with package change but it can be more complex than that in the future. This can be done in the Quarkiverse.

Regards,

Loïc

[Max Rydahl Andersen]

On 22 Dec 2021, at 14:30, Loïc MATHIEU wrote:

> Hi,
>
> I agree it's a mess ;)
>
> I think I misunderstand what you wanted to do, you want to keep the high
> level on version 7.10 and make it use a low level on version 7.16 ?
>
> I'm not sure it'll fully work, and as we cannot test hundreds of API
> methods I'm not sure it's a good idea to mix versions. I think it'll
> preferable to keep the high level and the low level version aligned, so if
> you want to upgrade the low level and move the high level to the
> Quarkiverse, you'll need to duplicate the low level (and the common
> extension) in the Quarkiverse so the high level will use a low level client
> on the same version. It is a lot of copy pasting but it should not be
> difficult.
>
> For the record, other microservices frameworks decided to upgrade the high
> level to 7.16 despite the new license.

which are those ? that is incredibly bad if they do - the license prevents free open use of
any application using it.

/max

/max
https://xam.dk/about

[Loïc MATHIEU]

https://github.com/spring-projects/spring-data-elasticsearch/pull/2036

https://github.com/micronaut-projects/micronaut-elasticsearch/pull/254

Both use the high level rest client.

[Max Rydahl Andersen]

On 22 Dec 2021, at 15:45, Loïc MATHIEU wrote:

> https://github.com/spring-projects/spring-data-elasticsearch/pull/2036
> https://github.com/micronaut-projects/micronaut-elasticsearch/pull/254
>
> Both use the high level rest client.

That I must say is surprising and would be very curious how they defend
the implications of the elastic search license on their users and contributors.

IANAL but been advised by license lawyers to NOT depend/include on it
as its just not an open source license.

But I'll do what I can to investigate it ASAP.

/max

/max
https://xam.dk/about

[Guillaume Smet]

On Wed, Dec 22, 2021 at 3:45 PM Loïc MATHIEU <loik...@gmail.com> wrote:

https://github.com/spring-projects/spring-data-elasticsearch/pull/2036

https://github.com/micronaut-projects/micronaut-elasticsearch/pull/254

Both use the high level rest client.

Yeah, I don't want to do that for a good reason: the high level client is not really supposed to support versions of Elasticsearch different from the matching one. So if we upgrade the high level client to 7.16.2, we are basically requiring our user base to upgrade their servers to 7.16.x or potentially end up with potentially weird incompatibilities and this is not acceptable.

For some the Elastic license is OK but for others providing managed services on Elasticsearch, they can't use the Elastic license, which means they have two options: pay Elastic or end up with the SSPL - which IMHO is the worst license ever written with a special mention for the very vague wording open to all interpretations.

That is definitely not an approach we can have in Quarkus, especially since a lot of people upgrading Quarkus won't notice this change and that the SSPL is a very dangerous beast.

What I'm going to try is upgrade the low level client and see if the high level client 7.10 continues to work. If so, we can keep it until it stops working:

https://github.com/quarkusio/quarkus/pull/22486

But be aware that the very day the high level client stops working with an updated low level client, we will have to drop it. Thus why I wanted to move it to the Quarkiverse Hub in advance as I can't be sure I will have the time to move it then.

--

Guillaume