org.omg.CORBA.NO_PERMISSION: vmcid: 0x49424000 minor code: 306 completed: No

[muthuku...@hotmail.com]

Hi All,

How are you?
Do you have any idea on what the following exception means?

This happens in our application when we are doing a performance test with 15 concurrent sessions and 500,000 transaction requests. For one and half hours everything runs fine. After that we get this exception message for one or two minutes and all the transactions in this time frame fails. Then again everything works fine after that for some time, before we see this again.

Thanks for taking time to help me out with this. Your help is much appreciated.

java.rmi.AccessException: CORBA NO_PERMISSION 0x49424306 No; nested exception is:

org.omg.CORBA.NO_PERMISSION:

Trace from server: 1198777258 at host elsoxfappt001 >>

org.omg.CORBA.NO_PERMISSION: vmcid: 0x49424000 minor code: 306 completed: No

at com.ibm.ISecurityLocalObjectBaseL13Impl.CSIServerRI.receive_request(CSIServerRI.java:991)

at com.ibm.rmi.pi.InterceptorManager.iterateReceiveRequest(InterceptorManager.java:762)

at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:599)

at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:463)

at com.ibm.rmi.iiop.ORB.process(ORB.java:439)

at com.ibm.CORBA.iiop.ORB.process(ORB.java:1737)

at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2300)

at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:65)

at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:95)

at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1471)

<< END server: 1198777258 at host elsoxfappt001

vmcid: 0x49424000 minor code: 306 completed: No

at com.ibm.CORBA.iiop.UtilDelegateImpl.mapSystemException(UtilDelegateImpl.java:204)

at javax.rmi.CORBA.Util.mapSystemException(Util.java:84)

[Paul Ilechko]

muthuku...@hotmail.com wrote:
> Hi All,
>
> How are you?
> Do you have any idea on what the following exception means?
>
> This happens in our application when we are doing a performance test with 15 concurrent sessions and 500,000 transaction requests. For one and half hours everything runs fine. After that we get this exception message for one or two minutes and all the transactions in this time frame fails. Then again everything works fine after that for some time, before we see this again.
>
> Thanks for taking time to help me out with this. Your help is much appreciated.
>
>
> java.rmi.AccessException: CORBA NO_PERMISSION 0x49424306 No; nested exception is:
>
> org.omg.CORBA.NO_PERMISSION:
>
>

Are you testing RMI clients, or web components making calls to EJBs
internally?

Do you have WAS security enabled? What authentication method - LTPA?

[muthuku...@hotmail.com]

Hi Paul,

Yes we are doing EJB calls from a reomte client, which is running on a Tomcat Server.

Was security is enabled and the authentication mechanism is LTPA as well.

let me know, if you need any thing else.
Thanks for your reply.

Thanks,
Muthu

[Paul Ilechko]

Then I have a feeling that you maybe hitting the LTPA timeout, and
perhaps not handling it well in your client code. You need to
reauthenticate when the token expires, and then resubmit the request
that failed.

[muthuku...@hotmail.com]

Paul,
We do authenticate seperately for each and every request. We pass the username and password every time we send a transaction. So, effectively all the 500,000 do send authentication requests. So, I don't think it is Time Out.
Correct me if I am wrong in my understanding.

For me it looks like IBM has some problem with the authentication and security when we are doing load test.
Thanks,
Charan

[Paul Ilechko]

muthuku...@hotmail.com wrote:
> Paul,
> We do authenticate seperately for each and every request. We pass the username and password every time we send a transaction. So, effectively all the 500,000 do send authentication requests. So, I don't think it is Time Out.
> Correct me if I am wrong in my understanding.

That's not how RMI authentication works - you don't pass a username and
pwd on the request, you do a JAAS login, which this is then retained on
the CSIv2 session until it expires.

>
> For me it looks like IBM has some problem with the authentication and security when we are doing load test.

It's very unlikely that you would be the first to find this problem, so
I tend to doubt your suggestion without some realistic proof. You are,
of course, welcome to open a PMR with IBM support if you think it is a bug.

[muthuku...@hotmail.com]

Paul,
I am going to increase the LTPA Time out as you suggested. I will let you know how it goes.
Thanks,
Charan

[Paul Ilechko]

That's not really what I'm suggesting - i think you need to see how your
code handles timeout situations, and do some more detailed testing to
see if that really is the problem. Perhaps you need to get some trace.

[muthuku...@hotmail.com]

Hi Paul,
You are absolutely spot on. Looks like LTPA Time out is the issue. I have changed the LTPA Timeout to 11 minutes, I started seeing the error in 8th and 9th minute I started my tests. Where as it used to take 1:45 minutes before to get this exception, where we had 120 min timeout setting.

There are few things that I don't understand. If an LTPA Timeout happens, Why should Websphere invalidate the existing session, which I believe is the reason for this.
The error message we received doesn't indicate any where that it is an LTPA Timeout. What kind of exception are we supposed to capture in the client code for LTPA Timeout to re-authenticate?
Are there any side effects if we increase the LTPA Timeout to 24 hours?

Thanks paul for your response, I would appreciate if you can adivse on these questions as well.
Thanks,
Charan

[Paul Ilechko]

muthuku...@hotmail.com wrote:
> Hi Paul, You are absolutely spot on. Looks like LTPA Time out is the
> issue. I have changed the LTPA Timeout to 11 minutes, I started
> seeing the error in 8th and 9th minute I started my tests. Where as
> it used to take 1:45 minutes before to get this exception, where we
> had 120 min timeout setting.
>
> There are few things that I don't understand. If an LTPA Timeout
> happens, Why should Websphere invalidate the existing session, which
> I believe is the reason for this.

You are doing RMI here, correct? In that case, the CSIv2 session is
established on the initial request, and the JAAS subject that you create
is associated with that Session. You don't have to reauthenticate until
the Session expires, which I would assume is based on the lifecyle of
the LTPA token. Remember, LTPA expirations are absolute, they are not
relative to usage like HTTP Sessions are.

The error message we received
> doesn't indicate any where that it is an LTPA Timeout. What kind of
> exception are we supposed to capture in the client code for LTPA
> Timeout to re-authenticate? Are there any side effects if we increase
> the LTPA Timeout to 24 hours?

You're going to have to experiment with this, I don't know exactly what
error you will get.

Increasing the LTPA token timeout decreases security, as it provides a
longer time interval for someone who has stolen a token to access your
systems. The default is two hours. For web based access it needs to be
long enough that you don't annoy users who have to re-login. For RMI
access that should be transparent to users, so the time can be shorter.

BTW, I don't think you mentioned what WAS version you were on. There was
an old error with invalid "No Permissions" exceptions in WAS 5.1.1.x

http://www-1.ibm.com/support/docview.wss?rs=180&context=SSEQTP&uid=swg1PQ94240

[muthuku...@hotmail.com]

Paul,

Yes we are using RMI.

we are using Websphere 6.0.2.15.

Regards,
Muthu

[muthuku...@hotmail.com]

Paul,

But how many transactions will fail because of this. I had founs so far around 620 errors after the time out spanning for next five minutes.
How much time does it take to invalidate the LTPA token? Do you think these are also because of the same token invalidation?
If it is absolute time out, Does it mean that I will face errors every 120 minutes? I see errors intermittenly after first 120 minutes. i am confused.

Do you know of some link where I can get detailed information on how LTPA timeout works? I don't want to bug you with too many questions.
Thanks again
regards,
muthu

[muthuku...@hotmail.com]

Paul,

They say this has been found in 6.1 during stress testing. May be the same issue is present in 6.0?? Any thoughts?

http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg1PK28579

[Paul Ilechko]

You don't need to know anything about how LTPA works. You're going to
get an exception when you call the EJB method - some kind of permission
exception. When you get that - and you'll have to do some testing to
find out exactly what the exception is - your client will need to
re-login to WAS, i.e. do the same JAAS login that it did initially to
get the token in the first place.

[muthuku...@hotmail.com]

Paul,

I keep getting the failures after the first LTPA Timeout intermittently. In total, I had more than 1000 requests failed. Does this mean there were so many time outs in 3 hours? Even if it is LTPA time out I would assume to have 1 time out in this time period and to a maximum of 20 requests which were in server at that time, not intermittently after the initial time out.

Did you check this URL?
http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg1PK28579

Thanks,
Muthu

[Paul Ilechko]

open a PMR. That's the only way to get a definitive answer.

[muthuku...@hotmail.com]

I am going to raise a PMR. Thanks for your help.

[ting...@us.ibm.com]

Hi Muthu,
Do you have used default websphere jaas or you have your own jaas or other security module.
I have exact same error in my code, but I have my own jaas module. I would like to narrow if the problem is caused by the customer sercurity module.
Also, please keep post your progress here, I will update with my information also.
I posted a similar issue here http://www-128.ibm.com/developerworks/forums/dw_thread.jsp?forum=266&thread=178991&cat=9

[ting...@us.ibm.com]

After enable the trace and compared with other test code, I think the problem is on this IOException.
[10/19/07 23:25:27:875 EDT] 00000087 ORBRas 3 com.ibm.rmi.iiop.Connection doReaderWorkOnce:2716 RT=2:P=455859:O=0:WSTCPTransportConnection[addr=9.22.113.76,port=2193,local=9100] IOException [java.net.SocketException: Connection reset] for com.ibm.rmi.iiop.Connection@4a6b7df1 timeStamp=30 isServer=true state=2 transportConnection=com.ibm.ws.orbimpl.transport.WSTCPTransportConnection@4891bdf1 socket=Socket[addr=green.bocaraton.ibm.com/9.22.113.76,port=2193,localport=9100] key=[com.ibm.ws.orbimpl.transport.ConnectionKey@16098eb8, host=green.bocaraton.ibm.com, port=2193]

This error blocked the reauthentication after the ltpa token timeout.

Any idea how to debug this error?
Thanks!

[ting...@us.ibm.com]

In my case, the code works in WAS6.0.2.11, but failed on 6.0.2.19. So I compared the trace file from both systems. Looks like the client code stoped (no more trace) on the 6.0.2.19, it results server IOException. But 6.0.2.11 system reauthenticate automatically after the NO_PERMISSION exception.

Could anybody helps on this? What else can I do to get more debug informations?

Thanks,

Here are the trace information compared with the two systems
Client trace on the broken system, which is 6.0.2.19
[10/22/07 20:04:56:296 EDT] 0000000a SASRas 3 [SecurityContextImpl.csi_continue_security_context], [ServerID: 888454077]
Message type is ASSOC_CONTEXT_ERROR - client_context_id: 1, major_status: 0, minor_status: 1229079296, error_token length: 0

[10/22/07 20:04:56:296 EDT] 0000000a SASRas 3 [CSIClientRI.receive_exception], [ServerID: 888454077]
*** MESSAGE COMPLETED ***

[10/22/07 20:04:56:296 EDT] 0000000a SASRas < [CSIClientRI.receive_exception], [ServerID: 888454077]
Exit

Server trace on the broken system, which is 6.0.2.19
[10/22/07 20:04:56:281 EDT] 00000084 ORBRas < com.ibm.rmi.iiop.ServerResponseImpl writeTo:348 ORB.thread.pool : 0 Exit
[10/22/07 20:04:56:281 EDT] 00000084 ORBRas < com.ibm.rmi.iiop.Connection doWork:2571 ORB.thread.pool : 0 Exit
[10/22/07 20:04:56:422 EDT] 0000008a ORBRas 3 com.ibm.rmi.iiop.Connection doReaderWorkOnce:2716 RT=3:P=611000:O=0:WSTCPTransportConnection[addr=9.22.113.76,port=2630,local=2635] IOException [java.net.SocketException: Connection reset] for com.ibm.rmi.iiop.Connection@561f9f2d timeStamp=58 isServer=false state=2 transportConnection=com.ibm.ws.orbimpl.transport.WSTCPTransportConnection@6e3bdf2d socket=Socket[addr=green.bocaraton.ibm.com/9.22.113.76,port=2630,localport=2635] key=[com.ibm.ws.orbimpl.transport.ConnectionKey@233d1ad9, host=9.22.113.76, port=2630]

Client trace on the working system, which is 6.0.2.11
[10/22/07 17:11:00:984 EDT] 0000000a SASRas 3 [SecurityContextImpl.csi_continue_security_context], [ServerID: -1]
Message type is ASSOC_CONTEXT_ERROR - client_context_id: 1, major_status: 0, minor_status: 1229079296, error_token length: 0

[10/22/07 17:11:00:984 EDT] 0000000a SASRas 3 [CSIClientRI.receive_exception], [ServerID: -1]
*** MESSAGE COMPLETED ***

[10/22/07 17:11:00:984 EDT] 0000000a SASRas < [CSIClientRI.receive_exception], [ServerID: -1]
Exit
[10/22/07 17:11:00:984 EDT] 0000000a SASRas > [SecurityConnectionInterceptor.getConnectionKey], [ServerID: -1]
Entry
[10/22/07 17:11:00:984 EDT] 0000000a SASRas 3 [SecurityConnectionInterceptor.getConnectionKey], [ServerID: -1]
profile.getHost returns: true

Server trace on the working system, which is 6.0.2.11
[10/22/07 17:11:00:969 EDT] 00000099 ORBRas < com.ibm.rmi.iiop.ServerResponseImpl writeTo:348 ORB.thread.pool : 0 Exit
[10/22/07 17:11:00:969 EDT] 00000099 ORBRas < com.ibm.rmi.iiop.Connection doWork:2571 ORB.thread.pool : 0 Exit
[10/22/07 17:11:01:000 EDT] 0000009b ORBRas > com.ibm.rmi.iiop.Connection createStreamObject:1172 RT=1:P=141094:O=0:WSSSLTransportConnection[addr=9.22.113.76,port=2570,local=9403] Entry
[10/22/07 17:11:01:000 EDT] 0000009b ORBRas 3 com.ibm.CORBA.iiop.ObjectKey <init>:320 RT=1:P=141094:O=0:WSSSLTransportConnection[addr=9.22.113.76,port=2570,local=9403] 0x4a4d4249000000124773e3aa37643062633737336533616166633334000000240000005949454a5002000b8e953b077365727665723103454a420000003facac000200010134000000525449436174616c6f675f7376632e65617223525449436174616c6f675f7376632e6a61722353657276696365436174616c6f67

[Paul Ilechko]

ting...@us.ibm.com wrote:
> In my case, the code works in WAS6.0.2.11, but failed on 6.0.2.19. So
> I compared the trace file from both systems. Looks like the client
> code stoped (no more trace) on the 6.0.2.19, it results server
> IOException. But 6.0.2.11 system reauthenticate automatically after
> the NO_PERMISSION exception.
>
> Could anybody helps on this? What else can I do to get more debug
> informations?
>
> Thanks,
>
>

Open a PMR, this sounds like a bug.

[ting...@us.ibm.com]

Hi Muthu,
Have you solved the problem yet?
Which version of WebSphere did you use? Which version of Java did your client use?
I found out that I got NO_PERMISSION error when the EJB client was using JRE 1.5 and talking to WebSphere 6.0.2.19 which is JRE 1.4.2. And the EJB client works find if it uses JRE1.4.2.
Do you see the same thing in your test?
Thanks,
Ting

[S@ren]

I had the same problem.

Apparently the IBM orb implementation changed, so the client side no longer automatically retries when it receives the NO_PERMISSION exception from the server.

I looked at two versions of the application client orb WAS 5.1.1 (java 1.4.2) and 6.1.0.21 (java 1.5.0)

In the old version a client request that returned the NO_PERMISSION exception was automatically retried according to the com.ibm.CORBA.authenticationRetryCount property. This is no longer the case in the newer orb. Here you must implement your own retry/reauthentication mechanism. In the new orb only exceptions of the following types are retried: COMM_FAILURE, TRANSIENT and NO_RESOURCES.

[Knowledge master]

I did open up a PMR 67199,122,000 and an APAR with the CSIv2 SessionManager retry logic using the wrong/ expired token was logged. More info can be found here.

Http://wasissues.blogspot.com