User code deployment & application software code upgrade

[ramani...@gmail.com]

Folks,

We are planning to use the open source HZ drop library (3.8) for IMDG. There are questions around how the HZ IMDG manages upgrades of application code. So you know, its not the HZ library version upgrade but the application code that is using HZ library upgrade process use case.

To give you the background, the IMDG is spread across multiple nodes in the cluster (say 8 for instance). The application node uses distributed map & data is persisted / maintained eternally in the memory. Now during application software upgrade, each node is upgraded one at a time (the upgrade process by itself can take a while & can span across days, in other words, nodes in the HZ cluster will be using the same version of HZ drop library, however the application software will / can belong to two different versions & should co-exist). The application software upgrade, will start from one of the non-coordinating node. Each application software upgrade will introduce new distributed map / object. Post upgrade in that node, the node will request to get the handle to the NEW added distributed map (or object / entity), which the oldest node / coordinating HZ node (that continues to run the OLD application software) would not be aware of at that point in time. As the coordinating node will be the last node that will be updated ALWAYS. 

Questions

1. Does HZ user code deployment allows (I see the feature is still in BETA in 3.8) to handle this use case?
   
2. How does enabling the user code deployment (meaning remote class loader feature) impacts the usual / normal working of the HZ cluster?
   
3. Is there a better way to handle application code upgrade, without downtime, assuming each upgrade introduces a new distributed object?
   

Thank you for your time & help.

Ganesh

[ihsan demir]

Hello,

Are you planning to use Hazelcast client lib or will you embed your application into the members? If you are using as client, and since you are not upgrading Hazelcast, you should not need anything special, since client is backward compatible for minor versions starting version 3.6. I do not see why you would need user code deployment for your case\ are you sure that you need such a feature at all?

Regards,

ihsan

19 Haziran 2017 Pazartesi 09:33:14 UTC+3 tarihinde ramani...@gmail.com yazdı:

[ramani...@gmail.com]

Thank you Ihsan for your reply. We are not planning to use HZ client.

I will try to share the use case that I have in my mind.

The application is embedded into each member. To put it right, the application runs / maintains its own proprietary cluster. However, going forward we are planning to use IMDG to cache the cluster data. With IMDG in place, each data instance will be a distributed MAP. For the version 1 (let's assume this is starting version, where we will be introducing HZ to the existing application as a drop library), it's perfectly alright. The first node being upgraded will become the HZ coordinating node & the node following will be able to access / get the distributed MAP (as co-ordinating node knows the same set of distributed objects as the other nodes in the cluster). Basically the code block hHazelcastInstance.getMap("my_map_name"); will work on other nodes, as the co-ordinating node understands & knows the map name / distributed object.

However, post this version, say in version 2, upgrade will follow the same procedure. One node will be updated at a time. In this case, the node being upgraded first will not be the co-ordinating node. Moreover, the upgrade will introduce a new distributed map or object to be cached in IMDG. In other words, the new node application code will have a code block that will refer to a map which the co-ordinating node will not be aware of. Meaning, hHazelcastInstance.getMap("my_NEW_map_name") will be executed when the newly upgraded node is up & running. This will result in an error, as the co-ordinating node is not aware of this NEW distributed map or object my_NEW_map_name?

I was of the opinion User code deployment feature, basically caters to such scenario.

Did I misunderstood?

Hope am clear.

Best Regards,

Ganesh

[Fuad Malikov]

Hi,

This is not going to be an in depth answer, but wanted to make sure that you have seen the following chapter in Hazelcast Deployment and Operations Guide:

https://hazelcast.org/deployment-operations-guide/#maintenance-and-software-updates

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/d79ba5c1-35e3-422b-8b81-f742d51958e7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ramani...@gmail.com]

Thank you Fuad, I did read the "maintenance-and-software-updates" section, it does talk about member node rolling upgrade, however, doesn't share the behavior of the cluster, when one member has newer application code (that access newer distributed object) & other  members (including the coordinating node) are running old application code. 

Am looking @ a switch / mode when HZ coordinating node knows it's in maintenance mode. A mode where the HZ coordinating node did not understand the distributed object access call, but the other upgraded node does.

May be am looking @ the wrong side of the problem. Just want to make sure am not missing a obvious API that HZ provides.

Thank you!

Best,

Ganesh

On Monday, 19 June 2017 13:33:23 UTC-4, Fuad Malikov wrote:

Hi,

This is not going to be an in depth answer, but wanted to make sure that you have seen the following chapter in Hazelcast Deployment and Operations Guide:

https://hazelcast.org/deployment-operations-guide/#maintenance-and-software-updates

On Sun, Jun 18, 2017 at 10:27 AM, <ramani...@gmail.com> wrote:

Folks,

We are planning to use the open source HZ drop library (3.8) for IMDG. There are questions around how the HZ IMDG manages upgrades of application code. So you know, its not the HZ library version upgrade but the application code that is using HZ library upgrade process use case.

To give you the background, the IMDG is spread across multiple nodes in the cluster (say 8 for instance). The application node uses distributed map & data is persisted / maintained eternally in the memory. Now during application software upgrade, each node is upgraded one at a time (the upgrade process by itself can take a while & can span across days, in other words, nodes in the HZ cluster will be using the same version of HZ drop library, however the application software will / can belong to two different versions & should co-exist). The application software upgrade, will start from one of the non-coordinating node. Each application software upgrade will introduce new distributed map / object. Post upgrade in that node, the node will request to get the handle to the NEW added distributed map (or object / entity), which the oldest node / coordinating HZ node (that continues to run the OLD application software) would not be aware of at that point in time. As the coordinating node will be the last node that will be updated ALWAYS. 

Questions

1. Does HZ user code deployment allows (I see the feature is still in BETA in 3.8) to handle this use case?
   
2. How does enabling the user code deployment (meaning remote class loader feature) impacts the usual / normal working of the HZ cluster?
   
3. Is there a better way to handle application code upgrade, without downtime, assuming each upgrade introduces a new distributed object?
   

Thank you for your time & help.

Ganesh

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.

[ihsan demir]

Hello Ganesh,

What do you mean by this:

"hHazelcastInstance.getMap("my_NEW_map_name") will be executed when the newly upgraded node is up & running. This will result in an error"

This call will NOT cause any error, it will create the new map if not exist and just create a proxy for it.

Regards,

ihsan

19 Haziran 2017 Pazartesi 18:36:35 UTC+3 tarihinde Ganesh Ramani yazdı:

[ramani...@gmail.com]

Hi Ihsan,

I realize now I wasn't clear.

The application (OSGi based) allows to construct distributed objects dynamically in a controlled manner. In other words, for the services (new OSGi bundles) that can participate in IMDG are controlled by static final constants managed in multiple service specific Interfaces (a new interface / class is introduced for each new service introduced across upgrades). Meaning to get the distributed object, the actual code would be hHazelcastInstance.getMap(NEWIMDGServices.my_NEW_map_name). NEWIMDGServices is a new interface that was introduced in the new version of the application software. The updated interface will not be available for the old / existing coordinator node, that is still not updated & will ONLY be updated LAST.

Now this new class / interface introduced (NEWIMDGServices) is available in the remote class loader, however, the coordinator class loader is not aware of this new Constant or Interface that was introduced as part of the upgrade. 

Trying to invoke getMap with this new constant, fails with a ClassLoaderUtil failing (@ coordinating node) to load the new interface remotely with "argument 'className' can't be null".

argument 'className' can't be null
        at com.hazelcast.util.Preconditions.isNotNull(Preconditions.java:88)
        at com.hazelcast.nio.ClassLoaderUtil$ClassLoaderWeakCache.get(ClassLoaderUtil.java:219)
        at com.hazelcast.nio.ClassLoaderUtil.newInstance(ClassLoaderUtil.java:70)
        at com.hazelcast.map.impl.mapstore.StoreConstructor.getStoreFromClassOrNull(StoreConstructor.java:82)
        at com.hazelcast.map.impl.mapstore.StoreConstructor.createStore(StoreConstructor.java:46)
        at com.hazelcast.map.impl.mapstore.BasicMapStoreContext.create(BasicMapStoreContext.java:124)
        at com.hazelcast.map.impl.mapstore.MapStoreContextFactory.createMapStoreContext(MapStoreContextFactory.java:48)
        at com.hazelcast.map.impl.MapContainer.<init>(MapContainer.java:111)
        at com.hazelcast.map.impl.MapServiceContextImpl$1.createNew(MapServiceContextImpl.java:113)
        at com.hazelcast.map.impl.MapServiceContextImpl$1.createNew(MapServiceContextImpl.java:109)
        at com.hazelcast.util.ConcurrencyUtil.getOrPutSynchronized(ConcurrencyUtil.java:93)
        at com.hazelcast.map.impl.MapServiceContextImpl.getMapContainer(MapServiceContextImpl.java:235)
        at com.hazelcast.map.impl.PartitionContainer.createRecordStore(PartitionContainer.java:98)
        at com.hazelcast.map.impl.PartitionContainer.access$000(PartitionContainer.java:39)
        at com.hazelcast.map.impl.PartitionContainer$1.createNew(PartitionContainer.java:49)
        at com.hazelcast.map.impl.PartitionContainer$1.createNew(PartitionContainer.java:45)
        at com.hazelcast.util.ConcurrencyUtil.getOrPutSynchronized(ConcurrencyUtil.java:93)
        at com.hazelcast.map.impl.PartitionContainer.getRecordStore(PartitionContainer.java:134)
        at com.hazelcast.map.impl.operation.MapOperation.getRecordStoreOrNull(MapOperation.java:156)
        at com.hazelcast.map.impl.operation.MapOperation.innerBeforeRun(MapOperation.java:76)
        at com.hazelcast.map.impl.operation.MapOperation.beforeRun(MapOperation.java:72)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:180)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:401)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:117)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:102)
        at ------ submitted from ------.(Unknown Source)
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:114)
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:75)
        at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:155)
        at com.hazelcast.map.impl.proxy.MapProxySupport.waitUntilLoaded(MapProxySupport.java:575)
        at com.hazelcast.map.impl.proxy.MapProxyImpl.waitUntilLoaded(MapProxyImpl.java:102)
        at com.hazelcast.map.impl.proxy.MapProxySupport.initializeMapStoreLoad(MapProxySupport.java:220)
        at com.hazelcast.map.impl.proxy.MapProxySupport.initialize(MapProxySupport.java:212)
        at com.hazelcast.map.impl.proxy.MapProxyImpl.initialize(MapProxyImpl.java:102)
        at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.doCreateProxy(ProxyRegistry.java:177)
        at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.createProxy(ProxyRegistry.java:167)
        at com.hazelcast.spi.impl.proxyservice.impl.ProxyRegistry.getOrCreateProxy(ProxyRegistry.java:136)
        at com.hazelcast.spi.impl.proxyservice.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:147)
        at com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:380)
        at com.hazelcast.instance.HazelcastInstanceImpl.getMap(HazelcastInstanceImpl.java:186)
        at com.hazelcast.instance.HazelcastInstanceProxy.getMap(HazelcastInstanceProxy.java:96)

Hope am clear now?

That's the reason, I asked (originally), if enabling user code deployment & setting the PROVIDER MODE to LOCAL_AND_CACHED_CLASSES, will fix this issue? More importantly, if this is the right use case for user code deployment, if yes what are the repercussions of using this BETA feature? 

Thank you.

Best regards,

Ganesh

[ihsan demir]

Default value for provider mode is LOCAL_AND_CACHED_CLASSES, hence I do not think setting it will make a change.

20 Haziran 2017 Salı 17:21:24 UTC+3 tarihinde Ganesh Ramani yazdı:

[ramani...@gmail.com]

Guys,

Just want to circle back & check if User code deployment is intended for the use case I described? Is the use case valid for HZ to resolve the remote class?

Please let me know.

Thanks, Ganesh

[M. Sancar Koyunlu]

Hi Ganesh, 

Your use case is quite different than user code deployment that we provide. User code deployment is to load new classes coming with new members to other old members. Examples to classes : EntryProcessor's used on Map.values(), Runnable's on IExecutorService.submit or Objects put via Map.put. 

I think, you are gonna need to craft your own solution on top of hazelcast.

Regards,

Sancar. 

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/3fb632cf-e09d-42cb-8b06-a3b1e97ea33d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Sancar Koyunlu

Software Engineer, Hazelcast

san...@hazelcast.com

[ramani...@gmail.com]

Thank you Sancar for confirming!

Best,

Ganesh

[Jelly Bomb]

Thanks for the info 

четверг, 29 июня 2017 г. в 19:49:59 UTC+3, ramani...@gmail.com:

[Андрей Бурлаченко]

I really want you to tell me how to become a tester or tell me where to start. I even agree if you throw a useful article here so that I understand what to look for

суббота, 28 мая 2022 г. в 01:33:22 UTC+3, silon...@gmail.com:

[Jelly Bomb]

I can recommend an excellent site, with many interesting articles on various topics. For example, here is a link that shows how a tester should write a bug report for a programmer, and therefore, it seems to me that even a person who is not related to programming will understand everything that is written here

воскресенье, 29 мая 2022 г. в 01:15:51 UTC+3, burlach...@gmail.com: