Re: Error code 104

[radomir]

I think it's not related to a particular User as we see really a lot of HardDeadlineExceededError errors (followed by the error code 104 message at the end of log) for the last few days, which is very disturbing. 

From what I've seen, it happens when request is routed to a new instance that fails to start in 60 seconds (although our application usually start in less than 20). In our case HardDeadlineExceededError was thrown mostly on jar file read (which is beyond our control):

Failed startup of context com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@1d6e3d2{/,/base/data/home/apps/s~bugdigger-hr/241r2.361896469816654578}
com.google.apphosting.api.DeadlineExceededException: This request (878f5068315d018e) started at 2012/09/24 21:26:31.318 UTC and was still executing at 2012/09/24 21:27:30.767 UTC.
	at java.io.FileInputStream.readBytes(Native Method)
	at java.io.FileInputStream.read(FileInputStream.java:231)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
	at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
	at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
	at java.util.zip.ZipInputStream.read(ZipInputStream.java:163)
	at java.util.jar.JarInputStream.read(JarInputStream.java:194)
	at java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:109)
	at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:87)
	at java.util.jar.JarInputStream.getNextEntry(JarInputStream.java:135)
	at java.util.jar.JarInputStream.getNextJarEntry(JarInputStream.java:166)
	at net.sourceforge.stripes.vfs.DefaultVFS.listResources(DefaultVFS.java:169)
	at net.sourceforge.stripes.vfs.DefaultVFS.list(DefaultVFS.java:62)
	at net.sourceforge.stripes.vfs.VFS.list(VFS.java:200)
	at net.sourceforge.stripes.util.ResolverUtil.find(ResolverUtil.java:202)
	at net.sourceforge.stripes.util.ResolverUtil.findImplementations(ResolverUtil.java:164)
	at net.sourceforge.stripes.config.BootstrapPropertyResolver.getClassPropertyList(BootstrapPropertyResolver.java:222)
	at net.sourceforge.stripes.config.RuntimeConfiguration.init(RuntimeConfiguration.java:298)
	at net.sourceforge.stripes.controller.StripesFilter.init(StripesFilter.java:125)
	at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:662)
	at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1250)
	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:467)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.createHandler(AppVersionHandlerMap.java:219)
	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.getHandler(AppVersionHandlerMap.java:194)
	at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:134)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447)
	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:452)
	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459)
	at com.google.tracing.TraceContext.runInContext(TraceContext.java:701)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328)
	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456)
	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
	at java.lang.Thread.run(Thread.java:679)

Doesn't look good.

On Monday, September 24, 2012 11:39:12 PM UTC+2, Rodrigo Kossmann wrote:

Hi guys.
Happens the following error when entering my application with a specific User:

201.86.195.206 - - [24/Sep/2012:14:24:48 -0700] "GET / HTTP/1.1" 500 0 - "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1" "tci.chainit.com.br" ms=60020 cpu_ms=22858 exit_code=104 instance=00c61b117c6e51b4aec6d2e4089516b479c2a0
E2012-09-24 18:24:47.974
org.datanucleus.transaction.Transaction rollback: Operation rollback failed on resource: org.datanucleus.store.appengine.DatastoreXAResource@2d659, error code UNKNOWN and transaction: [DataNucleus Transaction, ID=Xid=
W2012-09-24 18:24:48.375
Error for /
com.google.apphosting.runtime.HardDeadlineExceededError: This request (b08d8999601781ee) started at 2012/09/24 21:23:48.426 UTC and was still executing at 2012/09/24 21:24:48.373 UTC.
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:263)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)
	at com.google.appengine.tools.development.TimedFuture.get(TimedFuture.java:41)
	at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
	at com.google.appengine.api.datastore.FutureHelper$CumulativeAggregateFuture.get(FutureHelper.java:145)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
	at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:72)
	at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:33)
	at com.google.appengine.api.datastore.DatastoreServiceImpl$2.runInternal(DatastoreServiceImpl.java:113)
	at com.google.appengine.api.datastore.DatastoreServiceImpl$2.runInternal(DatastoreServiceImpl.java:110)
	at com.google.appengine.api.datastore.TransactionRunner.runInTransaction(TransactionRunner.java:31)
	at com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:110)
	at com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:94)
	at com.google.apphosting.runtime.jetty.DatastoreSessionStore.saveSession(DatastoreSessionStore.java:94)
	at com.google.apphosting.runtime.jetty.SessionManager$AppEngineSession.save(SessionManager.java:167)
	at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:41)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:266)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
	at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447)
	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:452)
	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459)
	at com.google.tracing.TraceContext.runInContext(TraceContext.java:701)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328)
	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456)
	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
	at java.lang.Thread.run(Thread.java:679)
C2012-09-24 18:24:48.376
Uncaught exception from servlet
com.google.apphosting.runtime.HardDeadlineExceededError: This request (b08d8999601781ee) started at 2012/09/24 21:23:48.426 UTC and was still executing at 2012/09/24 21:24:48.373 UTC.
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:263)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)
	at com.google.appengine.tools.development.TimedFuture.get(TimedFuture.java:41)
	at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
	at com.google.appengine.api.datastore.FutureHelper$CumulativeAggregateFuture.get(FutureHelper.java:145)
	at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:90)
	at com.google.appengine.api.datastore.FutureHelper.getInternal(FutureHelper.java:72)
	at com.google.appengine.api.datastore.FutureHelper.quietGet(FutureHelper.java:33)
	at com.google.appengine.api.datastore.DatastoreServiceImpl$2.runInternal(DatastoreServiceImpl.java:113)
	at com.google.appengine.api.datastore.DatastoreServiceImpl$2.runInternal(DatastoreServiceImpl.java:110)
	at com.google.appengine.api.datastore.TransactionRunner.runInTransaction(TransactionRunner.java:31)
	at com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:110)
	at com.google.appengine.api.datastore.DatastoreServiceImpl.put(DatastoreServiceImpl.java:94)
	at com.google.apphosting.runtime.jetty.DatastoreSessionStore.saveSession(DatastoreSessionStore.java:94)
	at com.google.apphosting.runtime.jetty.SessionManager$AppEngineSession.save(SessionManager.java:167)
	at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:41)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
	at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:266)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
	at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
	at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:447)
	at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:452)
	at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:459)
	at com.google.tracing.TraceContext.runInContext(TraceContext.java:701)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:336)
	at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:328)
	at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:456)
	at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:251)
	at java.lang.Thread.run(Thread.java:679)
W2012-09-24 18:24:48.427
A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may be throwing exceptions during the initialization of your application. (Error code 104)

We believe that this User object is in some state which impedes saving.
Someone has gone through this error before?
Thanks.

[Takashi Matsuo]

Hi Ricardo,

Are you saying a particular entity always throws an error when changing?

Can you tell me your app-id and the key in question?

On Thu, Sep 27, 2012 at 4:42 AM, Ricardo <ric...@proelosoftware.com> wrote:

It seems, to me, the rollback operation is not finishing in the 60´s second wich each request has to ends and the roolback operation is canceled. In this way the object ends in this a state that we cant write on it.

 

This problems seems to me very serious because the user object end in this "unchangeble" state and that could not be fixed by its own, netheir by the JDO our Datanucleous.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/mf2-i9hQfGkJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Takashi Matsuo | Developers Advocate | tma...@google.com

[Ricardo]

Hi Takashi,

 

Yes, that´s right. Always this Rollback critical error. I neither can catch the error, what gives me a awful message to my user.

 

Anoter strange behavior that I could experience is that the restore tool inside the datastore admin console, never ends when this entity is inside one of the packages.

 

app-id = proeloc

key = aglzfnByb2Vsb2NyDwsSB0NvbXBhbnkYi6ECDA

 

 

Thanks a lot. This matter is urgent.

[Takashi Matsuo]

Hi Ricardo,

Thanks for the info.

What kind of change are you trying to make on this entity?

Can you change this entity via the datastore viewer of the admin console? Does it also fail?

-- Takashi

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/BqgvBnceV2UJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Ricardo]

I´m changing a property of a child thata has a relation owened one-to-many, and in the end saving the parent´s entity.

Yes, I can change the entities (parent and child) in the admin console without problems.

[Takashi Matsuo]

On Sat, Sep 29, 2012 at 12:17 PM, Ricardo <ric...@proelosoftware.com> wrote:

I´m changing a property of a child thata has a relation owened one-to-many, and in the end saving the parent´s entity.

In one transaction? How big is the change? Are those child entities necessarily to have the owned one-to-many relationship?

I'm asking like this, but I also think you'd better post this question to the stack overflow with the actual code and the stack trace. Can you do that?

-- Takashi

 

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/scrUaMzIG8oJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Rodrigo Kossmann]

Hi.

"In one transaction?" - Yes.

"How big is the change?" - It happens when I try to save the parent entity, independent of changes. In this case, the changes are small.

"Are those child entities necessarily to have the owned one-to-many relationship?" -  Not necessarily. But I believe that the error does not happen because the relationship. It happens when we update data in own entity.

the code:

CompanyUser.validate("email", email);

CompanyUser.validate("password", password, password);

Validator.checkValidation();

CompanyUser user;

try {

user = company.getUserByEmail(email);

//(user.getPassword() == "") se o usuário administrador resetar a senha, alterar o e-mail ou cadastrar um novo usuário

//(user.isWaitingConfirmation()) nos mesmos casos acima e mais quando o usuário alterar seu próprio e-mail ou o usuário administrador se cadastrar (cadastro na home do site)

if(Tools.isEmptyString(user.getPassword()) || user.isWaitingConfirmation()) {

if(user.getLastAccessDate() == null) //Se for usuário novo

throw new AuthenticationException("Seu usuário encontra-se bloqueado.<br />Para ativar seu acesso ao portal, acesse seu e-mail e siga as instruções.");

else

throw new AuthenticationException("Seu usuário encontra-se bloqueado.<br />Para reativar seu acesso ao portal, acesse seu e-mail e siga as instruções.");

}

byte[] bDigest = user.getPasswordBytes();

byte[] bSalt = user.getPasswordSaltBytes();

byte[] proposedDigest = user.getHash(password, bSalt);

if (Arrays.equals(proposedDigest, bDigest)) {

try{

user.setLastAccessDate(Tools.getCurrentTime());

company.saveDetached();

} catch (DeadlineExceededException e) {

Tools.sendMail("ERRO Severidade média: Não esta salvando ultimo acesso de Usuario", "ERRO: Não esta salvando ultimo acesso do Usuario " + user.getEmail() + " da empresa " + user.getCompany().getName()+ "" +

"Objeto esta com transação aberta.", Tools.getChaintSupportEmail());

}

return user;

}

} catch (ObjectNotFoundException e) {

user = null;

}

throw new AuthenticationException("Usuário/Senha não conferem");

 

 the stack:

Unfortunately, we can't post the full stack because the error does not happen locally.

Thanks for the help, 

[Ricardo]

Hi Takashi,

Follow the stack overflow link:
http://stackoverflow.com/questions/12674796/org-datanucleus-transaction-transaction-rollback-on-google-app-engine

Just to know what expect from Google´s support. Which are the further steps to solve this problem? Like I said, this is a huge problem, and so on, seems to be a serious bug. This application is a key product from our company.

Kind regards,

Ricardo