/_ah/background stops executing with Firebase

[Olof Gunnarsson]

I'm using Google Cloud Endpoints and Google App Engine with Firebase (java). Firebase calls are done in a background thread "/_ah/background", which works fine. However, after a couple of days the background thread stops executing. Calls to GAE still work but calls to Firebase are not executed. A restart of the instance makes everything work again.

I get these in the log which contains log messages from my calls to Firebase, and then suddenly I don't get anymore of these logs and the calls to Firebase stops executing.

Any ideas?

17:02:46.795GET0 B56,999.2 sUnknown/_ah/background

17:02:46.795GET0 B34,555.2 sUnknown/_ah/background

17:02:46.795GET0 B17,351 sUnknown/_ah/background

17:02:46.795GET0 B17,350.9 sUnknown/_ah/background

17:02:46.795GET0 B8,925.6 sUnknown/_ah/background

17:02:46.795GET0 B8,923.4 sUnknown/_ah/background

17:02:46.795GET0 B8,866 sUnknown/_ah/background

17:02:46.795GET0 B8,131.1 sUnknown/_ah/background

17:02:46.795GET0 B8,103.3 sUnknown/_ah/background

17:02:46.795GET0 B8,101.8 sUnknown/_ah/background

17:02:46.795GET0 B8,079.2 sUnknown/_ah/background

[Nick (Cloud Platform Support)]

Hey Olof,

This is interesting. Could you clarify whether the logs are associated with recent calls, or whether they are perhaps older background threads which are closing due to their age? It appears the logs output got a bit mangled as well during copy-paste, is there any chance a more faithful reproduction of the complete logs info, or a screenshot with all fields expanded, could be provided?

Cheers,

Nick
Cloud Platform Community Support

[Olof Gunnarsson]

I don't fully understand the logs, but it looks like it's older background threads. The timestamps in the screenshot are 5 hours apart.

The log messages "onAuthenticated" and "onDataChange" are Firebase callbacks. These log messages are not written after a couple of days, and they are also not executed (I delete stuff from Firebase in onDataChange, but the deletions stop after a couple of days)

Attached is a better view of the log.

[Nick (Cloud Platform Support)]

Hey Olof, 

Thanks for the screenshot. I believe something like this has been observed before, but I'll need some time to find it, maybe it's unrelated. My intuition right now is that it's related to how the Firebase library opens/holds/fails to close threads. Could you provide as much detail as possible about the system you've set up, versions of any libraries, details about the runtime environment, etc.?

Thanks for your patience in the meantime, I hope to have more to report tomorrow.

Regards,

Nick
Cloud Platform Community Support

[Nick (Cloud Platform Support)]

Hey Olof,

Apologies for the slight delay. I'm still looking into any possible cause. Could you share any of the information mentioned in my last message to help speed this along? I'm looking at https://github.com/GoogleCloudPlatform/firebase-appengine-backend right now. Is this the library you're using?

Sincerely,

Nick
Cloud Platform Community Support

[Nick (Cloud Platform Support)]

Just a quick update, it seems based on the above-linked repository, that there could be one explanation:

if MessageProcessorServlet creates an instance of MessagePurger (extends Thread) and starts it, this thread will run forever unless interrupted, since it has a "while (true)" loop in its run() method. MessageProcessorServlet should call interrupt() on the MessagePurger during MessageProcessorServlet.destroy(). If the servlet isn't destroyed, though, or is destroyed without a call to destroy(), the MessagePurger thread might run long enough (as here) to be killed by the runtime. Or, given that in this case there doesn't appear to be any error text with the log line, perhaps these are simply long-running MessagePurger threads which were properly destroyed, albeit after a long run.

I hope this is helpful, let me know your thoughts and any feedback on whether this is the library you're using.

Cheers,

Nick
Cloud Platform Support

[Olof Gunnarsson]

Thanks for your update.

No, that's not the library I'm using.

I'm using firebase-client-jvm as in https://www.firebase.com/docs/android/guide/setup.html

This is Firebase V2, there is a newer Firebase V3 - https://firebase.google.com/docs/server/setup

but I cannot upgrade yet, since it requires a rewrite.

This is my build.grade in Android Studio:

def appengineVersion = "1.9.38"

dependencies {
    appengineSdk "com.google.appengine:appengine-java-sdk:${appengineVersion}"
    compile "com.google.appengine:appengine-endpoints:${appengineVersion}"
    compile "com.google.appengine:appengine-endpoints-deps:${appengineVersion}"
    compile 'javax.servlet:servlet-api:2.5'
    compile 'com.firebase:firebase-client-jvm:2.5.2'
    compile 'org.json:json:20160212'
    compile 'com.firebase:firebase-token-generator:2.0.0'
    compile 'org.apache.velocity:velocity:1.7'
    compile "com.google.appengine:appengine-api-1.0-sdk:${appengineVersion}"
    compile 'com.google.appengine.tools:appengine-gcs-client:0.6'
}

I'm using Google App Engine Standard Environment with Manual scaling. I access GAE with Cloud Endpoints from and iOS and Android client.

Here's a code snippet that works a couple of days, but then stops working:

@ApiMethod(name = "sendNotificationsBean")
public NotificationsBean sendNotificationsBean(@Named("uid") final String uid) {

  logger.info("Calling sendNotificationsBean method with uid:" + uid);   // this is executed every time, even after a couple of days
  final Firebase firebase = new Firebase(DataService.getInstance().NEW_BASE_URL);
  firebase.authWithCustomToken(DataService.getInstance().getNewFirebaseToken(), new Firebase.AuthResultHandler() {
      @Override
      public void onAuthenticationError(FirebaseError error) {
          logger.info("Login Failed! " + error.getMessage());    // this is *not* executed after a couple of days
          ApiProxy.flushLogs();
      }

      @Override
      public void onAuthenticated(AuthData authData) {
          logger.info("Login Succeeded!");    // this is *not* executed after a couple of days
          firebase.addListenerForSingleValueEvent(new ValueEventListener() {
              @Override
              public void onDataChange(DataSnapshot dataSnapshot) {
                  for (DataSnapshot notificationItem : dataSnapshot.getChildren()) {   // this is *not* executed after a couple of days

See above code with comments where the problem is.

[Nick (Cloud Platform Support)]

Hey Olof,

The source code for that library is unfortunately closed, and so it's not really possible to reason about the potential for thread creation/destruction. I'd suggest posting to the Firebase Mailing List, or as firebase.google.com/support provides as options, you could attempt to file a bug report or request one-on-one troubleshooting - it says you get 5 free one on one support questions every 365 days. That last option might be the best, as you'll probably get closest to engineering most immediately that way.

My intuition from before remains in force, though, to get back to the issue itself. It seems a thread being opened and never closed, or closed after a great duration is the cause. It's strange that this is impairing the functionality of the library, though, so again I suggest going to troubleshooting for this. If you have any further questions, feel free to ask them here, but I think that's the best path forward from here.

Cheers!

Nick
Cloud Platform Community Support

[Olof Gunnarsson]

Thanks, I'll try their support. I've been in contact with them earlier with another issue but they seem swamped.

I figured since Firebase is Google and Google App Engine is Google that this forum would be a good place.

Thanks again.

Olof

[Aran Arunakiri]

Hi Olaf, 

I am facing the sam exact problem as you....did you find any solution or work around for this ? It will be of great help. 

Thanks

[Olof Gunnarsson]

No, not yet. I'm still restarting the instance every other day as a workaround.

[Aran Arunakiri]

Me too :(

[Olof Gunnarsson]

I've upgraded to Firebase 3.0 today, will let you know if it works better.