"Unable to obtain the OperationalStringEntry" and weird redeployment behaviour

[Dawid Loubser]

Hi,

As part of a rather large deployment, I'm deploying a persistent Outrigger JavaSpace. It used to deploy just fine, but after updating to latest Rio-5.0 SNAPSHOT from Git today (first time in 3 weeks) it deploys with this warning in the log:

WARNING 2012.11.21 14:20:22,433 ServiceAdvertiser.advertise - Unable to obtain the OperationalStringEntry for com.travellinck/Space

This is the only service I have that uses inline Jini configuration file syntax to deploy, it looks like:

    service(name: 'com.travellinck/Space')

    {

        interfaces

        {

            classes 'net.jini.space.JavaSpace05'

            artifact ref: 'outrigger-dl'

        }

        implementation(class: 'com.sun.jini.outrigger.PersistentOutriggerImpl')

        {

            artifact ref: 'outrigger-store'

        }

        // Basic persistent Outrigger

        configuration '''

        import com.sun.jini.outrigger.snaplogstore.LogStore;

        import net.jini.id.UuidFactory;

        import net.jini.jeri.BasicILFactory;

        import net.jini.jeri.BasicJeriExporter;

        import net.jini.jeri.tcp.TcpServerEndpoint;

        com.sun.jini.outrigger {

            persistenceDirectory = "${RIO_HOME}/system/external/outrigger/data";

            store = new LogStore(this);

            serverExporter = new BasicJeriExporter(

                TcpServerEndpoint.getInstance(0),

                new BasicILFactory(),

                false,

                true);

        }

        '''

        maintain 1

    }

Furthermore, I have very unusual redeployment behaviour across all my services: I deploy first time fine (other than the warning above, and a mysteriously broken TransactionManger association), but re-deploying the same (with no opstring or code changes) about half of my services fail with ClassNotFound exceptions. Restarting Rio and re-deploying is then fine again, until I try to re-deploy. The error I get looks like:

SEVERE  2012.11.21 14:25:06,384 JSBDelegate$1.run - Failed to load the ServiceBean [com.travellinck/ShortMessageEvaluatorProxy] artifact [com.travellinck.approval:approval-engine-components:1.0]

org.rioproject.deploy.ServiceBeanInstantiationException: Service Instantiation Exception

at org.rioproject.jsb.JSBLoader.create(JSBLoader.java:112)

at org.rioproject.cybernode.ServiceBeanLoader.load(ServiceBeanLoader.java:416)

at org.rioproject.cybernode.JSBDelegate$1.run(JSBDelegate.java:499)

Caused by: java.lang.ClassNotFoundException: com.travellinck.approval.engine.impl.evaluation.evaluator.sms.rio.ShortMessageEvaluatorProxyService

at edu.emory.mathcs.util.classloader.URIClassLoader$1.run(URIClassLoader.java:145)

at java.security.AccessController.doPrivileged(Native Method)

at edu.emory.mathcs.util.classloader.URIClassLoader.findClass(URIClassLoader.java:133)

at java.lang.ClassLoader.loadClass(ClassLoader.java:423)

at java.lang.ClassLoader.loadClass(ClassLoader.java:356)

at org.rioproject.jsb.JSBLoader.create(JSBLoader.java:63)

at org.rioproject.cybernode.ServiceBeanLoader.load(ServiceBeanLoader.java:416)

at org.rioproject.cybernode.JSBDelegate$1.run(JSBDelegate.java:499)

Both of these problems make the current state of Rio in git rather unusable for us... 

Any advice would be much appreciated.

[Dawid Loubser]

Just FYI, I can replicate this every time with my (large) deployment - redeployment always causes several services to not load on the second time. On the other hand, in 10 redeployments, I could not cause the (admittedly simpler) Rio Workflow or Hospital samples to fail in the same manner. Who knows what configuration difference could be causing this... I will have to try and isolate the problem.

Some of the services that don't redeploy have no strange configuration, e.g. the specific failure that I quoted above is defined thusly:

    service(name: 'com.travellinck/ShortMessageEvaluatorProxy')

    {

        interfaces

        {

            classes 'org.rioproject.resources.servicecore.Service'

            artifact ref: 'approval-engine'

        }

        implementation(class:'com.travellinck.approval.engine.impl.evaluation.evaluator.sms.rio.ShortMessageEvaluatorProxyService')

        {

            artifact ref: 'approval-engine'

        }

        parameters

        {

            // Various...

        }

        association( type: "requires", serviceType: 'net.jini.core.transaction.server.TransactionManager', property: 'transactionManager', name: 'com.travellinck/TransactionManager')

        {

            management inject: 'eager', serviceDiscoveryTimeout: 15, serviceDiscoveryTimeUnits : 'minutes'

        }

        association( type: "requires", serviceType: 'net.jini.space.JavaSpace05', property: 'space', name: 'com.travellinck/Space')

        {

            management inject: 'eager', serviceDiscoveryTimeout: 15, serviceDiscoveryTimeUnits : 'minutes'

        }

        maintain 1

    }

[Dennis Reedy]

Sorry you're having troubles. If I follow, you're saying that if I create an opstring with the following 2 services I should be able to reproduce the errors you find while redeploying?

[Dawid Loubser]

I think so, I'm also busy investigating it at this moment. I've not been able to isolate a specific set of individual services that *cause* the problem, I'm simply seeing the strange re-deployment behaviour, where the initial deployment is fine (other than the OperationalStringEntry warning), but doing a

mvn rio:undeploy; mvn rio:deploy

again on the same opstring project fails to deploy a second time (lots of ClassNotFounds). I hope to have some more helpful information soon also, sorry if my posts are a bit short on concrete details. I've tried my best to get the same behaviour with any of the included Rio examples, and failed - this might be an obscure issue indeed!

[Dennis Reedy]

On Nov 21, 2012, at 1002AM, Dawid Loubser wrote:

I think so, I'm also busy investigating it at this moment. I've not been able to isolate a specific set of individual services that *cause* the problem, I'm simply seeing the strange re-deployment behaviour, where the initial deployment is fine (other than the OperationalStringEntry warning), but doing a

mvn rio:undeploy; mvn rio:deploy

Is it rather:

mvn rio:deploy; mvn rio:undeploy; mvn rio:deploy

?

again on the same opstring project fails to deploy a second time (lots of ClassNotFounds). I hope to have some more helpful information soon also, sorry if my posts are a bit short on concrete details. I've tried my best to get the same behaviour with any of the included Rio examples, and failed - this might be an obscure issue indeed!

Can you highlight any more environment setup? How many machines? Any possible way you have older 5.0-SNAPSHOTs on any of the participating machines?

Dennis

[Dawid Loubser]

On Wednesday, November 21, 2012 5:10:10 PM UTC+2, Dennis Reedy wrote:

Is it rather:

mvn rio:deploy; mvn rio:undeploy; mvn rio:deploy

Your command will also work, yes :-) I meant I did an initial deploy, and then later an undeploy / redeploy.

 

Can you highlight any more environment setup? How many machines? Any possible way you have older 5.0-SNAPSHOTs on any of the participating machines?

Oh, just testing on my workstation - one machine, which has a proper install (in maven repo, and in $RIO_HOME) of latest Rio from SVN.

I've made an opstring with a small subset of services (4 or so) and I can confirm that the Warning of being unable to obtain the OperationalStringEntry appears to be totally random, happens on some deployments, others not. With the small subset of services (including the specific one that religiously failed with ClassNotFound the second time around) I'm not seeing the ClassNotFound yet (after about 5 tries). So something strange is going on.

I've noticed that often when I deploy, there is a bit of delay, and I notice network activity. No chance that the Rio resolver is pulling the (older) rio-5.0-SNAPSHOT artifacts from the Rio repo even though I've built (newer) locally perhaps? I'm going to do some offline testing also to try and eliminate that as an option.

thanks for the assistance, Dennis.

[Dawid Loubser]

Oi, I don't know what to do. I have total chaos here with multi-opstring deployments that used to work on the version of a month ago. Sometimes things are not deploying (i.e. sitting forever in an unresolved state, no errors - the deployer just ominously prints things like " There are [1] service provisioning requests outstanding").

If I simply re-deploy an opstring that is already deployed (with mvn rio:deploy) it seems to work fine, but if I once rio:undeploy, I forever get random ClassNotFound for the services when I redeploy them until I restart Rio.

Lastly, it's as if some aspects of association injection have changed. I use eager injection + service discovery timeout quite a bit, but I have cases (unchanged opstrings since a month ago) where the setter to inject the property is simply not called any more.

Dennis, if you could perhaps give me an idea what you've (conceptually) been changing over the past couple of weeks, I could help de-bug this thing? I'm a bit overwhelmed at the moment, but I'm trying to isolate particular problems bit by bit.

Dawid

[Dennis Reedy]

First, can you rollback to the version that was stable for you?

Second, you may have something with older version of the SNAPSHOT being pulled down from the Rio repository. I'll have to look into that.

[Dawid Loubser]

I am rolling back to an earlier version, yes (thanks). I do want to spend a bit of time exploring this problem and hope to give you some meaningful data, since I'll have to cross this bridge sometime in anyway (don't want to be stuck on a certain snapshot forever).