Notify all replicas?

[George Gastaldi]

Hey!

I have a scenario where I'm storing information in memory (a timestamp) in my application which is invalidated when an endpoint is hit. This works great when the application runs in a single pod/instance, but how can I make that work if I need to run multiple replicas? How can I make sure this information is updated across all replicas? 

I have a DB backend, but I'm afraid that querying the data on every request is not going to scale well.  I could also query the data in a specific timeframe, but I wonder if there are better solutions for this. 

How is that problem solved in a microservices world? I'd love to hear your thoughts,

Best Regards,

George Gastaldi

Principal Software Engineer

Red Hat

[Emmanuel Bernard]

The solution I've recommended as the most cloud native friendly solution is to offload that state to some in memory store like Infinispan or Redis. That way, you can scale your app independently from the state and it massively simplifies the notion of network config etc.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJhY-80K%2BhAGRFtdKQvGou%2BR3hqregA-QruB0uuBNLAVgQ%40mail.gmail.com.

[Sanne Grinovero]

On Fri, 26 Nov 2021, 14:05 Emmanuel Bernard, <eber...@redhat.com> wrote:

The solution I've recommended as the most cloud native friendly solution is to offload that state to some in memory store like Infinispan or Redis. That way, you can scale your app independently from the state and it massively simplifies the notion of network config etc.

+1

Infinispan would be my favourite option as well, however if this is your only need a simple DB table might suffice.

Remember a DB table can be created using an in-memory storage engine, so you can get the in-memory part for free and still access it via simple JDBC. This is useful if you already need a DB for other needs as well.

You'd switch to Infinispan when a single node can no longer take the load. 

On Fri, Nov 26, 2021 at 2:48 PM George Gastaldi <ggas...@redhat.com> wrote:

Hey!

I have a scenario where I'm storing information in memory (a timestamp) in my application which is invalidated when an endpoint is hit. This works great when the application runs in a single pod/instance, but how can I make that work if I need to run multiple replicas? How can I make sure this information is updated across all replicas? 

I have a DB backend, but I'm afraid that querying the data on every request is not going to scale well.  I could also query the data in a specific timeframe, but I wonder if there are better solutions for this. 

How is that problem solved in a microservices world? I'd love to hear your thoughts,

Best Regards,

George Gastaldi

Principal Software Engineer

Red Hat

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJhY-80K%2BhAGRFtdKQvGou%2BR3hqregA-QruB0uuBNLAVgQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANYWk7MHp1wV82s1uv6sEGsoG8xLw7O2qSP4Yh-MAp4%2BRi8BgQ%40mail.gmail.com.

[George Gastaldi]

On Fri, Nov 26, 2021 at 11:11 AM Sanne Grinovero <sa...@hibernate.org> wrote:

Remember a DB table can be created using an in-memory storage engine, so you can get the in-memory part for free and still access it via simple JDBC. This is useful if you already need a DB for other needs as well.

You'd switch to Infinispan when a single node can no longer take the load. 

What would be an in-memory storage engine using JDBC in this case? Infinispan Embedded? 

[Sergey Beryozkin]

On Fri, Nov 26, 2021 at 2:12 PM Sanne Grinovero <sa...@hibernate.org> wrote:

On Fri, 26 Nov 2021, 14:05 Emmanuel Bernard, <eber...@redhat.com> wrote:

The solution I've recommended as the most cloud native friendly solution is to offload that state to some in memory store like Infinispan or Redis. That way, you can scale your app independently from the state and it massively simplifies the notion of network config etc.

+1

Infinispan would be my favourite option as well, however if this is your only need a simple DB table might suffice.

Remember a DB table can be created using an in-memory storage engine, so you can get the in-memory part for free and still access it via simple JDBC. This is useful if you already need a DB for other needs as well.

You'd switch to Infinispan when a single node can no longer take the load. 

This is the same dilemma Cem and myself were looking at, as we need to have an option in OIDC to store the tokens not as cookies due to the cookie size restrictions.

So we thought we'd start with a simple DB table and progress to using Infinispan (later in another extension) - my understanding was, with DB one can still make sure it works across multiple nodes (via the DB level configuration).

George - is it how you are thinking of making it work for your project at the moment (before considering a move to Infinispan/Redis) ?

Thanks, Sergey

 

On Fri, Nov 26, 2021 at 2:48 PM George Gastaldi <ggas...@redhat.com> wrote:

Hey!

I have a scenario where I'm storing information in memory (a timestamp) in my application which is invalidated when an endpoint is hit. This works great when the application runs in a single pod/instance, but how can I make that work if I need to run multiple replicas? How can I make sure this information is updated across all replicas? 

I have a DB backend, but I'm afraid that querying the data on every request is not going to scale well.  I could also query the data in a specific timeframe, but I wonder if there are better solutions for this. 

How is that problem solved in a microservices world? I'd love to hear your thoughts,

Best Regards,

George Gastaldi

Principal Software Engineer

Red Hat

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJhY-80K%2BhAGRFtdKQvGou%2BR3hqregA-QruB0uuBNLAVgQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANYWk7MHp1wV82s1uv6sEGsoG8xLw7O2qSP4Yh-MAp4%2BRi8BgQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO2RdTsUYFLuwoMt379dcV6TmSuMHfNx0S%2Bxyteq1PxbKQ%40mail.gmail.com.

[Sanne Grinovero]

You either use Infinispan OR an in-memory engine of a traditional RDBMS, accessing over JDBC.

For example MySQL has a dedicated engine, which you can choose in your "create table" script:

 - https://dev.mysql.com/doc/refman/5.7/en/memory-storage-engine.html

PostgreSQL doesn't have a dedicated engine but if you configure it with fsync disabled you'll get essentially the same semantics, except it applies for the whole DB, so the MySQL approach is a little more flexible as you can opt-in for particular tables only.

Other databases have similar options, or can get similar effects.

 

[Loïc MATHIEU]

Hi George,

You can also create a headless k8s service (clusterIp: None), then make a DNS resolution of the name of this service, this should give you the list of the IP of the pods.

But yeah, a distributed memory cache may also work, but it can be a little overwhelming if it's just for a simple use case.

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJhfOLJDZrsb0ScWPhuLcHOQXE7r%2BSY8FCVUBGFSfgfqpw%40mail.gmail.com.

[Sanne Grinovero]

On Fri, 26 Nov 2021 at 15:13, Sergey Beryozkin <sbia...@redhat.com> wrote:

On Fri, Nov 26, 2021 at 2:12 PM Sanne Grinovero <sa...@hibernate.org> wrote:

On Fri, 26 Nov 2021, 14:05 Emmanuel Bernard, <eber...@redhat.com> wrote:

The solution I've recommended as the most cloud native friendly solution is to offload that state to some in memory store like Infinispan or Redis. That way, you can scale your app independently from the state and it massively simplifies the notion of network config etc.

+1

Infinispan would be my favourite option as well, however if this is your only need a simple DB table might suffice.

Remember a DB table can be created using an in-memory storage engine, so you can get the in-memory part for free and still access it via simple JDBC. This is useful if you already need a DB for other needs as well.

You'd switch to Infinispan when a single node can no longer take the load. 

This is the same dilemma Cem and myself were looking at, as we need to have an option in OIDC to store the tokens not as cookies due to the cookie size restrictions.

So we thought we'd start with a simple DB table and progress to using Infinispan (later in another extension) - my understanding was, with DB one can still make sure it works across multiple nodes (via the DB level configuration).

Sure but with Infinispan as well. It's designed for *distributed* and fast storage ;)

 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMsYBfVacEZnuXWcHKAv%2BodE3a-v_HuW8WHhdjcRhMCc1spvjw%40mail.gmail.com.

[Max Rydahl Andersen]

Maybe i'm missing something here - we are talking about storing a single timestamp
and have it invalidated to know when it is necessary to go rebuild/refetch data
from a database that gets updated at irregular times.

A separate infinispan process which will be asked to run with 3 nodes too is
what we say is the solution for that ?

George, I would measure if that single query for a single value against the DB will even be detectable as a performance overhead.

/max

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO0XEQ3werHBfhU0pAJNdipFVp9K_Wd92HgVW_ZEJrZ42g%40mail.gmail.com.

/max
https://xam.dk/about

[Emmanuel Bernard]

On Sat 27 Nov 2021 at 01:22, Max Rydahl Andersen <mand...@redhat.com> wrote:

Maybe i'm missing something here - we are talking about storing a single timestamp
and have it invalidated to know when it is necessary to go rebuild/refetch data
from a database that gets updated at irregular times.

A separate infinispan process which will be asked to run with 3 nodes too is
what we say is the solution for that ?

It does not have to be 3 nodes if you’re ok for the possible and temporary absence of that state.

[Max Rydahl Andersen]

On 29 Nov 2021, at 10:49, Emmanuel Bernard wrote:

> On Sat 27 Nov 2021 at 01:22, Max Rydahl Andersen <mand...@redhat.com>
> wrote:
>
>> Maybe i'm missing something here - we are talking about storing a single
>> timestamp
>> and have it invalidated to know when it is necessary to go rebuild/refetch
>> data
>> from a database that gets updated at irregular times.
>>
>> A separate infinispan process which will be asked to run with 3 nodes too
>> is
>> what we say is the solution for that ?
>>
>
> It does not have to be 3 nodes if you’re ok for the possible and temporary
> absence of that state.

well, the 3 nodes isn't something we got under control - but lets say we can get that requirement removed the suggested solution is still to run a full separate container to keep the state of a single timestamp to decide wether relevant to go fetch new set of data.

Still feels like shooting birds with a cannon :)

...at that stage having it more be done just scaling down and up again would be simpler.

/max

>>>>>> Red Hat <https://www.redhat.com/>
>>>>>>
>>>>>> <https://www.redhat.com/>

>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Quarkus Development mailing list" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>> an email to quarkus-dev...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJhY-80K%2BhAGRFtdKQvGou%2BR3hqregA-QruB0uuBNLAVgQ%40mail.gmail.com

>>>>>> <https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJhY-80K%2BhAGRFtdKQvGou%2BR3hqregA-QruB0uuBNLAVgQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>> .

>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Quarkus Development mailing list" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to quarkus-dev...@googlegroups.com.
>>>>> To view this discussion on the web visit

>>>>> https://groups.google.com/d/msgid/quarkus-dev/CANYWk7MHp1wV82s1uv6sEGsoG8xLw7O2qSP4Yh-MAp4%2BRi8BgQ%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/quarkus-dev/CANYWk7MHp1wV82s1uv6sEGsoG8xLw7O2qSP4Yh-MAp4%2BRi8BgQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .

>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Quarkus Development mailing list" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to quarkus-dev...@googlegroups.com.
>>>> To view this discussion on the web visit

>>>> https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO2RdTsUYFLuwoMt379dcV6TmSuMHfNx0S%2Bxyteq1PxbKQ%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO2RdTsUYFLuwoMt379dcV6TmSuMHfNx0S%2Bxyteq1PxbKQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .

>>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Quarkus Development mailing list" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to quarkus-dev...@googlegroups.com.
>>> To view this discussion on the web visit

>>> https://groups.google.com/d/msgid/quarkus-dev/CAMsYBfVacEZnuXWcHKAv%2BodE3a-v_HuW8WHhdjcRhMCc1spvjw%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/quarkus-dev/CAMsYBfVacEZnuXWcHKAv%2BodE3a-v_HuW8WHhdjcRhMCc1spvjw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .

>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Quarkus Development mailing list" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to quarkus-dev...@googlegroups.com.
>>
>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO0XEQ3werHBfhU0pAJNdipFVp9K_Wd92HgVW_ZEJrZ42g%40mail.gmail.com
>> <https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO0XEQ3werHBfhU0pAJNdipFVp9K_Wd92HgVW_ZEJrZ42g%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>> /max
>> https://xam.dk/about
>>

[Sanne Grinovero]

Hi,

If that's all you need, indeed I wouldn't bother with a shared store: have each application instance store their own invalidation timestamps - assuming I understood correctly that you only want a time based expiry and don't  need the ability to perform a synchronised invalidation.

You could set the expiry low enough to not have a noticeable effect on consistency, but at least it can prevent to DOS the DB with queries if load ever gets too high to handle. I would highly recommend to have this safety layer rather than going with plain DB hits on each need.

But if you need node state coordination for stronger consistency, you'll need a shared storage of some sort with basic coordination primitives, or at the very least a bus to send broadcasting control messages. Loic's suggestion sounds good in this case.

But it also depends of you're going to have good use of Infinispan/ a DB for other reasons - starting one just for this need might be overkill, but I wouldn't say it's odd to use them if you're going to benefit from these in other ways as well.. e.g. Infinispan is great to collect usage metrics as well.

Sanne

[Alexey Loubyansky]

On Mon, Nov 29, 2021 at 2:26 PM Sanne Grinovero <sa...@hibernate.org> wrote:

Hi,

If that's all you need, indeed I wouldn't bother with a shared store: have each application instance store their own invalidation timestamps - assuming I understood correctly that you only want a time based expiry and don't  need the ability to perform a synchronised invalidation.

I think we need synchronized invalidation. A single (Maven) client request translates to multiple requests: the JSON artifact, the metadata.xml, the checksums. AFAIU, there is no guarantee all of them will be handled by a single instance of the app.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO3CPprTr87bKqGTjw38aCXk_NAmSewGrXy%3DnQVjootNUw%40mail.gmail.com.

[George Gastaldi]

On Mon, Nov 29, 2021 at 10:30 AM Alexey Loubyansky <alexey.l...@redhat.com> wrote:

On Mon, Nov 29, 2021 at 2:26 PM Sanne Grinovero <sa...@hibernate.org> wrote:

Hi,

If that's all you need, indeed I wouldn't bother with a shared store: have each application instance store their own invalidation timestamps - assuming I understood correctly that you only want a time based expiry and don't  need the ability to perform a synchronised invalidation.

I think we need synchronized invalidation. A single (Maven) client request translates to multiple requests: the JSON artifact, the metadata.xml, the checksums. AFAIU, there is no guarantee all of them will be handled by a single instance of the app.

 

Exactly, this is the use case I'm facing at the moment.   

[Sanne Grinovero]

On Mon, 29 Nov 2021, 13:30 Alexey Loubyansky, <alexey.l...@redhat.com> wrote:

On Mon, Nov 29, 2021 at 2:26 PM Sanne Grinovero <sa...@hibernate.org> wrote:

Hi,

If that's all you need, indeed I wouldn't bother with a shared store: have each application instance store their own invalidation timestamps - assuming I understood correctly that you only want a time based expiry and don't  need the ability to perform a synchronised invalidation.

I think we need synchronized invalidation. A single (Maven) client request translates to multiple requests: the JSON artifact, the metadata.xml, the checksums. AFAIU, there is no guarantee all of them will be handled by a single instance of the app.

But are all these different things published atomically? How do you expect readers to perform a consistent read?

Unless you have a solid answer to such questions, it's fine to cache things and introduce a little delay.

Also I'd expect that it's ok for a new component release to only become fully visible after some seconds or even minutes, but once things are listed you can rely on precise version tags to ensure a consistent read among components that require a critical match, such as a jar and its checksum. Such matches rely on precise versions so caching "discovery" related metadata shouldn't be a problem.

You need to think it in terms of what order of new elements being published is fine to not break consumers - but once you've established which valid orders you want to enforce, it's OK if such steps get published with some effective delay. 

[Sanne Grinovero]

In other words, nobody ever needs synchronised invalidation. 

What you need is to clarify and establish required ordering rules that will result in [some form of] consistent read.

[Guillaume Smet]

A bit late to the party but I think you're presuming a bit too much of how much of a problem it will be to do a simple indexed query on a database.

My advice would be: just use the PostgreSQL database and only consider something else if it becomes an issue (and I'm pretty sure it won't).

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJiOiLmWdtwADTmDGQYhvWxn3_t2xZAx7qke_6n1ppsNwA%40mail.gmail.com.

[Georgios Andrianakis]

On Mon, Nov 29, 2021 at 6:49 PM Guillaume Smet <guillau...@gmail.com> wrote:

A bit late to the party but I think you're presuming a bit too much of how much of a problem it will be to do a simple indexed query on a database.

My advice would be: just use the PostgreSQL database and only consider something else if it becomes an issue (and I'm pretty sure it won't).

+1

On Mon, Nov 29, 2021 at 2:35 PM George Gastaldi <ggas...@redhat.com> wrote:

On Mon, Nov 29, 2021 at 10:30 AM Alexey Loubyansky <alexey.l...@redhat.com> wrote:

On Mon, Nov 29, 2021 at 2:26 PM Sanne Grinovero <sa...@hibernate.org> wrote:

Hi,

If that's all you need, indeed I wouldn't bother with a shared store: have each application instance store their own invalidation timestamps - assuming I understood correctly that you only want a time based expiry and don't  need the ability to perform a synchronised invalidation.

I think we need synchronized invalidation. A single (Maven) client request translates to multiple requests: the JSON artifact, the metadata.xml, the checksums. AFAIU, there is no guarantee all of them will be handled by a single instance of the app.

 

Exactly, this is the use case I'm facing at the moment.   

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJiOiLmWdtwADTmDGQYhvWxn3_t2xZAx7qke_6n1ppsNwA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALt0%2Bo_n4v8tHuo2K99Q-cqpE64PKJt%2BFdVwiWNh4Mf%3Dqw5A3Q%40mail.gmail.com.

[George Gastaldi]

So you're saying to perform indexed queries directly in the database (on every request) without caching anything in the application side?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALt0%2Bo_n4v8tHuo2K99Q-cqpE64PKJt%2BFdVwiWNh4Mf%3Dqw5A3Q%40mail.gmail.com.

[Max Rydahl Andersen]

On 29 Nov 2021, 17:56 +0100, George Gastaldi <ggas...@redhat.com>, wrote:

So you're saying to perform indexed queries directly in the database (on every request) without caching anything in the application side?

Yes - or at least just query the db to see if it had an update before doing it fully.

You can query *alot* from a well tuned data base :)
/max 

[Stuart Douglas]

Do you really need this to be visible effectively immediately? If not you could just check the DB every minute and invalidate them. If you are worried about one node being invalidated while others aren't you can just make them all check at the same time, and don't actually perform the invalidation on the node that handles the request.

Otherwise I would say that if you really wanted to invalidate locally cached state and did not want to add any per-request overhead then it seems like something you would want to solve with a message broker. Just have a topic that they all subscribe to and send an invalidation message when it is required.

Stuart

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAMmsnJhxjEu%2BgCS3MAKmOmbd7VH8ZJo-PR7mx1zDd3237KROoQ%40mail.gmail.com.