Twiddle errors

[Richard Good]

Hi

I call twiddle from scripts to de-activate and un-install deployable units.  Since upgrading to JSLEE-2.5.0 I have been having problems with this though I see the twiddle API has not changed.

If I do:
./twiddle.sh -s $IP_ADDRESS service -dServiceID[name=TEST_Service,vendor=test.vendor,version=0.1]

I get:
'success'

But if immediately afterwards I do:
./twiddle.sh -s $IP_ADDRESS deploy --un-install=DeployableUnitID[url=file:/opt/JSLEE/jboss-5.1.0.GA/server/default/deploy/TestDU.jar/]

I get:
ERROR [Twiddle] Command failure
org.jboss.console.twiddle.command.CommandException: Failed to invoke "uninstall" due to: ; - nested throwable: (javax.management.MBeanException)
        at org.mobicents.tools.twiddle.op.AbstractOperation.invoke(AbstractOperation.java:250)
        at org.mobicents.tools.twiddle.AbstractSleeCommand.execute(AbstractSleeCommand.java:118)
        at org.jboss.console.twiddle.Twiddle.main(Twiddle.java:306)
Caused by: javax.management.MBeanException
        at org.jboss.mx.interceptor.ReflectedDispatcher.handleInvocationExceptions(ReflectedDispatcher.java:184)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:165)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.jmx.connector.invoker.InvokerAdaptorService.invoke(InvokerAdaptorService.java:263)
        at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
        at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
        at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
        at org.jboss.jmx.connector.invoker.SerializableInterceptor.invoke(SerializableInterceptor.java:74)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
        at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:180)
        at sun.reflect.GeneratedMethodAccessor288.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
        at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:855)
        at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:422)
        at sun.reflect.GeneratedMethodAccessor287.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
        at sun.rmi.transport.Transport$1.run(Transport.java:159)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: javax.slee.InvalidStateException: ServiceID[name=Test_Service,vendor=com.smilecoms,version=0.1] is not inactive
        at org.mobicents.slee.container.management.ServiceManagementImpl.uninstallService(ServiceManagementImpl.java:732)
        at org.mobicents.slee.container.management.jmx.DeploymentMBeanImpl.uninstall(DeploymentMBeanImpl.java:389)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
        ... 45 more


If I wait a bit and execute the second command again then I get 'success'. 

It seems that the first command does not immediately de-activate the service, though it does return 'success'.

I did not have this problem when running JSLEE 2.4.0.FINAL.

Any ideas?

Regards
Richard.

[Bartosz Baranowski]

Twiddle does simple JMX calls, it only hides what/how to make call.

The deactivate methods.

These methods deactivate the Service or Services specified by the method argument.

o It can only be invoked when the specified Service or Services are in the Active state.

o If it returns successfully, the specified Service or Services have transitioned to the Stopping

state. Eventually each deactivated Service will transition to the Inactive state after

all SBB entities executing for the Service have completed processing (see Section

2.2.17).

and for uninstall

An SBB jar file cannot be uninstalled while the SBB defined in the SBB jar file is referenced

by other SBBs or Services installed in the SLEE. The getReferringComponents

method may be used to determine the set of SBBs and Services that reference an

SBB.

Im not quite sure how to interprete this - while services are still active - in process of transition, you cant uninstall DU. 

Bartosz Baranowski
JBoss R & D
==================================
Word of criticism meant to improve is always step forward.

[Eduardo Martins]

Is the service doing any active service logic when deactivated? As
Bartosz mentioned the service deactivation is only complete when all
sbb entities end, gracefully or not (Mobicents sets a timeout and
kills any sbb entity still existent). There were many changes since
2.4.0, but should not affect greatly the time to deactivate!

-- Eduardo
..............................................
http://emmartins.blogspot.com
http://redhat.com/solutions/telco

[Richard Good]

Hi,

Thanks for the quick responses.

Is the service doing any active service logic when deactivated?

I have this problem across all of my SBB's (all of which worked fine in 2.4.0) so I don't think it is a problem with the SBB still doing service logic.

(Mobicents sets a timeout and kills any sbb entity still existent). There were many changes since 2.4.0, but should not affect greatly the time to deactivate!

the un-install command works fine if I try it after about 10 seconds - is there a chance that the Mobicents timeout you mentioned is longer in 2.5.0?

Is there a twiddle command (I can not find one..) that can somehow wait for a service to become de-activated? This is my use case:

• DEACTIVE SERVICE

./twiddle.sh -s $IP_ADDRESS service -dServiceID[name=Test_Service,vendor=test.vendor,version=0.1]

• SOMEHOW WAIT FOR SERVICE TO MOVE FROM STOPPED STATE TO INACTIVE STATE

??

• UNINSTALL DU

./twiddle.sh -s $IP_ADDRESS deploy --un-install=DeployableUnitID[url=file:/opt/JSLEE/jboss-5.1.0.GA/server/default/deploy/TestDU.jar/]

• REINSTALL DU

cp TestDU.jar /opt/JSLEE/jboss-5.1.0.GA/server/default/deploy


Regards
Richard.

[Richard Good]

Hi

One further comment:

If I deactivate the service and confirm that nothing is referencing the service with:
./twiddle.sh -s $IP_ADDRESS deploy -rServiceID[name=Test_Service,vendor=test.vendor,version=0.1]
I still get the error on un-install unless I wait 15 seconds.

A quick and dirty fix is to sleep for 15 seconds between de-activate and un-install, though this is obivously not ideal.

Regards
Richard.

[Alexandre Mendonça]

Can you show the logs from the JSLEE server during/after the call to deactivate ?

--

Alexandre Mendonça // JBoss R&D
http://ammendonca.blogspot.com/

[Richard Good]

Thanks for the suggestion - from looking at the JSLEE logs I can see that the un-install call is waiting for the root SBB entities to end.

All of my SBBs use the timer facility - I suspected that this might be the entity that is waiting to end.  If I remove the timer facility code everything works fine.  (I am not sure why this didn't cause a problem in 2.4.0.FINAL?)

So my question now is: If I am using the timer facility and I want to de-activate the service do I have to cancel all running timers individually (Means keeping track of each timer_id somehow...), or is there some other way to deactivate all timers?

I already do:

public void unsetSbbContext() {

        this.timerFacility = null;
        this.nullACIFactory = null;
        this.nullActivityFactory = null;
       
....

Is that not enough?

Regards
Richard.

2011/10/27 Alexandre Mendonça <brai...@gmail.com>

[Bartosz Baranowski]

Inline.

Bartosz Baranowski
JBoss R & D
==================================
Word of criticism meant to improve is always step forward.

On Thu, Oct 27, 2011 at 5:20 PM, Richard Good <richar...@smilecoms.com> wrote:

Thanks for the suggestion - from looking at the JSLEE logs I can see that the un-install call is waiting for the root SBB entities to end.

All of my SBBs use the timer facility - I suspected that this might be the entity that is waiting to end.  If I remove the timer facility code everything works fine.  (I am not sure why this didn't cause a problem in 2.4.0.FINAL?)

So my question now is: If I am using the timer facility and I want to de-activate the service do I have to cancel all running timers individually (Means keeping track of each timer_id somehow...), or is there some other way to deactivate all timers?

Yup you do - or simply kill all activities, this will cancel all timers. However this is a bit tricky. Event which indicates service going down(ActivityEnd) will be received only by entities attached to service activity.

I already do:

public void unsetSbbContext() {

        this.timerFacility = null;
        this.nullACIFactory = null;
        this.nullActivityFactory = null;
       
....

Is that not enough?

Nope. This is just a simple reference. SLEE does not track such things. 'null' ing wont do any harm to timers :)

[Richard Good]

Hi

With the latest SLEE release (2.6.0) I am having further twiddle
problems.

My use case is for re-deploying SBBs to a running SLEE during
development. So I:
Deactivate SBB
Uninstall SBB
Copy new SBB into deploy folder

My problem is that when I de-activate the SBB with:
./twiddle.sh -s 10.0.0.30 service -
dServiceID[name=Example_Service,vendor=domain,version=0.1]

The state changes to Stopping and never gets out of that state no
matter how long I wait -
./twiddle.sh -s 10.0.0.30 service -
oServiceID[name=Example_Service,vendor=domain,version=0.1]
Stopping

My SBB has a timer that expires every second, apart from that there is
no active service logic.

My twiddle log is below.

Any idea what is happening? Is there a way to force an SBB to
deactivate?

Regards
Richard.

11:47:47,707 DEBUG [Twiddle] args[0]=-c
11:47:47,709 DEBUG [Twiddle] args[1]=file:////var/smile/install/
scripts/JSLEE/twiddle/lib/slee-twiddle.properties
11:47:47,709 DEBUG [Twiddle] args[2]=-s
11:47:47,709 DEBUG [Twiddle] args[3]=10.0.0.30
11:47:47,709 DEBUG [Twiddle] args[4]=service
11:47:47,709 DEBUG [Twiddle] args[5]=-
oServiceID[name=Example_Service,vendor=domain,version=0.1]
11:47:47,722 DEBUG [Twiddle] Command name: service
11:47:47,726 DEBUG [Twiddle] Command arguments: -
oServiceID[name=Example_Service,vendor=domain,version=0.1]
11:47:47,727 DEBUG [Twiddle] command proto type properties:
file:////var/smile/install/scripts/JSLEE/twiddle/lib/slee-twiddle.properties
11:47:47,767 DEBUG [Twiddle] Adding command 'usage.resource'; proto:
org.mobicents.tools.twiddle.jslee.ResourceUsageCommand@1f4689e
11:47:47,771 DEBUG [Twiddle] Adding command 'alarm'; proto:
org.mobicents.tools.twiddle.jslee.AlarmCommand@bb7465
11:47:47,774 DEBUG [Twiddle] Adding command 'timer'; proto:
org.mobicents.tools.twiddle.jsleex.TimerCommand@1db4f6f
11:47:47,779 DEBUG [Twiddle] Adding command 'activity'; proto:
org.mobicents.tools.twiddle.jsleex.ActivityCommand@e2dae9
11:47:47,785 DEBUG [Twiddle] Adding command 'profile'; proto:
org.mobicents.tools.twiddle.jslee.ProfileCommand@11f2ee1
11:47:47,786 DEBUG [Twiddle] Adding command 'usage.profile'; proto:
org.mobicents.tools.twiddle.jslee.ProfileUsageCommand@1c99159
11:47:47,794 DEBUG [Twiddle] Adding command 'resource'; proto:
org.mobicents.tools.twiddle.jslee.ResourceCommand@9173ef
11:47:47,800 DEBUG [Twiddle] Adding command 'deploy'; proto:
org.mobicents.tools.twiddle.jslee.DeployCommand@1edc073
11:47:47,808 DEBUG [Twiddle] Adding command 'router.stats'; proto:
org.mobicents.tools.twiddle.jsleex.RouterStatsCommand@872380
11:47:47,811 DEBUG [Twiddle] Adding command 'router.cfg'; proto:
org.mobicents.tools.twiddle.jsleex.RouterCfgCommand@16fa474
11:47:47,816 DEBUG [Twiddle] Adding command 'trace'; proto:
org.mobicents.tools.twiddle.jslee.TraceCommand@20be79
11:47:47,822 DEBUG [Twiddle] Adding command 'profile.edit'; proto:
org.mobicents.tools.twiddle.jslee.ProfileEditCommand@99681b
11:47:47,827 DEBUG [Twiddle] Adding command 'service'; proto:
org.mobicents.tools.twiddle.jslee.ServiceCommand@1bc887b
11:47:47,829 DEBUG [Twiddle] Adding command 'usage.service'; proto:
org.mobicents.tools.twiddle.jslee.ServiceUsageCommand@166a22b
11:47:47,835 DEBUG [Twiddle] Adding command 'deployer'; proto:
org.mobicents.tools.twiddle.jsleex.DeployerCommand@105738
11:47:47,839 DEBUG [Twiddle] Adding command 'slee'; proto:
org.mobicents.tools.twiddle.jslee.SleeCommand@a3d4cf
11:47:47,881 DEBUG [TimedSocketFactory] createSocket, hostAddr: /
10.0.0.30, port: 1099, localAddr: null, localPort: 0, timeout: 0
11:47:48,473 DEBUG [ServiceCommand] Converted result: Stopping

---------- Forwarded message ----------
From: Richard Good <richard.g...@smilecoms.com>
Date: Oct 27 2011, 10:50 am
Subject: Twiddle errors
To: mobicents-public

Hi

One further comment:

If I deactivate the service and confirm that nothing is referencing
the
service with:
./twiddle.sh -s $IP_ADDRESS deploy
-rServiceID[name=Test_Service,vendor=test.vendor,version=0.1]
I still get the error on un-install unless I wait 15 seconds.

A quick and dirty fix is to sleep for 15 seconds between de-activate
and
un-install, though this is obivously not ideal.

Regards
Richard.

On 27 October 2011 10:24, Richard Good <richard.g...@smilecoms.com>
wrote:> Hi,

> Thanks for the quick responses.

> Is the service doing any active service logic when deactivated?
> I have this problem across all of my SBB's (all of which worked fine in
> 2.4.0) so I don't think it is a problem with the SBB still doing service
> logic.

> (Mobicents sets a timeout and kills any sbb entity still existent). There
> were many changes since 2.4.0, but should not affect greatly the time to
> deactivate!
> the un-install command works fine if I try it after about 10 seconds - is
> there a chance that the Mobicents timeout you mentioned is longer in 2.5.0?

> Is there a twiddle command (I can not find one..) that can somehow wait for
> a service to become de-activated? This is my use case:

>    - *DEACTIVE SERVICE*

> ./twiddle.sh -s $IP_ADDRESS service
> -dServiceID[name=Test_Service,vendor=test.vendor,version=0.1]

>    - *SOMEHOW WAIT FOR SERVICE TO MOVE FROM STOPPED STATE TO INACTIVE
>    STATE*

> ??

>    - *UNINSTALL DU*

> ./twiddle.sh -s $IP_ADDRESS deploy
> --un-install=DeployableUnitID[url=file:/opt/JSLEE/

> jboss-5.1.0.GA/server/default/deploy/TestDU.jar/<http://jboss-5.1.0.ga/server/default/deploy/TestDU.jar/>
> ]

>    - *REINSTALL DU*

> cp TestDU.jar /opt/JSLEE/jboss-5.1.0.GA/server/default/deploy

> *
> *Regards
> Richard.

>> > I get:
>> > 'success'

>> org.mobicents.tools.twiddle.op.AbstractOperation.invoke(AbstractOperation.java:250)
>> >         at

>> org.jboss.mx.interceptor.ReflectedDispatcher.handleInvocationExceptions(ReflectedDispatcher.java:184)
>> >         at

>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >         at

>> org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
>> >         at

>> org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:855)
>> >         at

>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>> >         at

>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>> >         at

>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >         at

>> org.mobicents.slee.container.management.ServiceManagementImpl.uninstallService(ServiceManagementImpl.java:732)
>> >         at

>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >         at

>> > Any ideas?

>> > Regards
>> > Richard.
This email is subject to the disclaimer of Smile Communications (PTY) Ltd. at http://www.smilecoms.com/disclaimer

[Eduardo Martins]

Check if this is not the cause:

 http://groups.google.com/group/mobicents-public/t/1b9a31fcfa958ee0

Otherwise please open issue.

-- Eduardo
..............................................
http://emmartins.blogspot.com

[Richard Good]

Hi

Thanks that fixed the problem!

I changed the entitiesRemovalDelay property to 1 in /opt/JSLEE/jboss-5.1.0.GA/server/default/deploy/mobicents-slee/META-INF/jboss-beans.xml

Now I do:

• Deactivate SBB service with twiddle
• Copy new SBB jar over old SBB jar in deploy folder (I no longer both using twiddle to uninstall the SBB, doesn't seem necessary?)

And in about 1 minute the new SBB is started.

I have 2 questions:

• With the default JSLEE configuration how can I deactivate and remove a service using Twiddle?
  
• Is it possible to make the entitiesRemovalDelay property less than 1 minute?  i.e. When a service is de-activated, remove it immediately.

Regards

Richard.

--

Richard Good

Manager: Applications & Services

Smile Communications (Pty) Ltd

mobile: +27 (0) 72 3898365

email:  richar...@smilecoms.com

Physical address: 12 Culross Road, Bryanston, 2191

Postal address: Postnet suite 605, Private Bag x5, Fourways North, 2086

[Alexandre Mendonça]

Hi Richard,

Please see inline.

--

Alexandre Mendonça // JBoss R&D
http://ammendonca.blogspot.com/

On Tue, Feb 21, 2012 at 09:49, Richard Good <richar...@smilecoms.com> wrote:

Hi

Thanks that fixed the problem!

I changed the entitiesRemovalDelay property to 1 in /opt/JSLEE/jboss-5.1.0.GA/server/default/deploy/mobicents-slee/META-INF/jboss-beans.xml

Now I do:

• Deactivate SBB service with twiddle
• Copy new SBB jar over old SBB jar in deploy folder (I no longer both using twiddle to uninstall the SBB, doesn't seem necessary?)

When using the SLEE JBoss Deployer (by copying to the deploy folder), you don't have to worry about any of those install/activate/deactivate/uninstall operations, those are automatically performed by the deployer, so you should not need to use twiddle at all. You should be able to just deploy a new DU jar, and it will take care of everything.

 

And in about 1 minute the new SBB is started.

I have 2 questions:

• With the default JSLEE configuration how can I deactivate and remove a service using Twiddle? 

You should be able to do it anyway, using Twiddle. What's happening is that some entities are still active (eg, there's a charging session going on and/or a SIP call, etc) and so they are not forced to end and be removed by default, letting the application release them gracefully. If when you try to deactivate the service it stays in STOPPING and there's no session going on, usually it means there are leaks of SBB Entities and/or ACs that should be fixed. You can use the SLEE Management Console for checking that.

 

• Is it possible to make the entitiesRemovalDelay property less than 1 minute?  i.e. When a service is de-activated, remove it immediately.

No, the unit is in minutes and 0 means no forced removal. Also, I think that's not something you'd want.. it may be helpful in development, to save those "one minutes", but as stated above, the best is really to fix what's hanging the service deactivation.