FirebaseDB component truncates text with a colon (:)

[Bob Glicksman]

I have a Firebase database, as pictured in the enclosure.  When I go to read the value of the tag "published_at", the tag in Firebase has the value "2018-04-16T04:08:40.228Z"; however, AI2 only returns the text string "2018-04-16T04".  It appears that the FirebaseDB component truncates a value at a colon (:) character, even though the colon is inside of a quoted string.  Is this a bug?  A feature"?  Is there a workaround to allow me to obtain the entire string that is in the Firebase database at this tag?  It is a well formatted timestamp, which is important in my app.

[SteveJG]

This appears to be a Firebase feature (  https://www.google.com/search?q=Firebase+use+of+colon&oq=Firebase+use+of+colon&aqs=chrome..69i57.5535j0j7&sourceid=chrome&ie=UTF-8 )  There might be something in the linked threads that will help.  Firebase is a Google feature,  App Inventor does not control what they do.

There is a good reason you may not want to continue to use Firebase and should switch to CloudDB Bob.   A developer has got to do what he/she has got to do.  Google is changing the way apps (all apps, not just App Inventor apps) communicate with Firebase.  App Inventor's FirebaseDB control may, at any moment, not be operable with Firebase as Google continues to make changes.  MIT recently provided this heads up advice to Forum moderators.

If Google changes the wire protocol for Firebase, the existing Firebase Component will break *for all users* even those who have their own Firebase account. 

MIT has no idea when (or even if)  the Firebase Component will become incompatible with the Google service. When it does, MIT will decide to make the necessary changes and commit valuable resources to implementing changes to at least support Firebase on Google devices, or just tell people to use CloudDB as it replicates most capabilities of Firebase for App Inventor's larges user base, educational use. After it breaks, even if MIT decides to discover a work around method of communication, users should expect to have a period of time when the control will be inoperable.

If you are developing for long term or critical use activities, perhaps CloudDB is a better option. MIT is not telling Firebase users at the moment, to move. This is advice informing that what works today may not work in the future.

Does CloudDB have the colon issue?   I do not know.   Read about Redis (the basis for CloudDB)   Here is what can be done with a Redis database (CloudDB)  https://redislabs.com/redis-enterprise/cloud/   .  Please let us know if the colon is also an issue with the CloudDB

Can Redis do everything that can be done using Firebase with an App Inventor FirebaseDB control?    Of course not.  The CloudDB can do things that AI2 users cannot do with FirebaseDB (including storing images...not just the link).  A CloudDB can be under total control of the developer (he/she can use his own server).  With Firebase you can store on Google using your own Firebase account.    Apples and oranges.  The link does explain what can be done with a Redis/CloudDB and it might have the functionality you require.

For educational purposes, and using either the FirebaseDB and CloudDB, they work pretty much the same (and CloudDB can store images , even using MIT's server.    :)   ).

Regards,

Steve

[Abraham Getzler]

Take SteveJG's advice to heart about CloudDB versus Firebase.

Regarding the colons, will your app be the only program to read that date/time data?

If so, consider storing it as a pure numeric Milliseconds Since 1970 value,

and formatting and reformatting the value for display in your app

using AI2 Clock blocks.

That would avoid the immediate problem, but not help with the longer term problem mentioned above.

ABG

[Bob Glicksman]

@SteveJG and @Abraham Getzler:  Thank you both for your rely and for the information that you have provided.  FYI - my application is to obtain event notification on my mobile device when an IoT device sends an alert message.  The IoT device is a Particle Photon (or Electron) and particle allows me to create a webhook that stores data to Firebase.  The data is minimal, but is enough for my app (running in background) to react to the .DataChanged event and use the Pura Vida Simple Notification Extension to place a notification on my phone.  Since you have information that FirebaseDB may be broken by Google but CloudDB is the future, I'll look into this option.

BTW: I am surprised that you believe that the string truncation is a Firebase issue and not an App Inventor component issue.  The full string is in the Firebase DB just fine and shows up just fine (in full)  with web access to the database via my browser.

Re. Changing the timestamp format:  Yes, this is my fallback option.  The timestamp that I am using is auto-generated by the Particle webhook and I have no control over it.  But I can always publish my own timestamp in whatever format I wish.  However, it would be nice not to have to do this and to use the already auto-generated timestamp.

[Abraham Getzler]

In looking at your original data screen shot, I notice that you are missing trailing " marks

on some (but not all) of your data values.

This is indicative of something broken.

The usual intent of a " mark is to separate data values

that might contain colon characters from the colon characters that

are part of JSON string structure, as documented at the JSON web site JSON.org

Do you have control over that data format?

Is it broken going into FireBase, or is the display broken?

Can you bypass the error by requesting tag "Publication/published_at" ?

ABG

[SteveJG]

This article seems to indicate that Firebase  (not the AI2 FirebaseDB control)  use a colon for specific purposes  https://firebase.googleblog.com/2017/12/read-and-write-your-realtime-database.html   .   Could this cause issues?   I do not know.  It is for you to discover Bob.

[Bob Glicksman]

No, the data is all there, final double quote and all. The screenshot shows only a partial tag value -- it is the way that the Firebase Console displays this.  If you mouse into the value text, you can see that the whole text string is there, terminating " and all.

I also performed a test:  I manually (via the Firebase console) added another tag "test" : "10:20"  and I can see the entire tag and value on the Firebase console.  But when I perform a .GetTag from App Inventor on this tag, the .GotTag value displays as "10".  This clearly shows the truncation at a colon.  

In any event, I gather from you knowledgeable folks that there probably isn't a future in the Firebase Real-Time database and I need to look into the Firebase Cloud DB going forward.  That's a real shame, since my Synchronized Shopping List app (https://github.com/TeamPracticalProjects/Synchronized_Shopping_List) is working great with the Real-Time database.  But alas, if Google is in the process of deprecating or re-purposing this Firebase tool, I guess that it is time to move on to where Google is going :)

[Bob Glicksman]

The colon (:) is the standard JSON delimited between tag and value, but only when it is outside of a set of double quotes".  Inside a pair of double quotes, everything is just a JSON string variable type.  Since Firebase returns JSON objects, I suspect that  there is a bug in the App Inventor .GotValue that fails to see that the : is part of a valid string variable and not a JSON object delimiter.  But, of course, it could be in how Firebase formats up the value for its response.  I can't prove where the bug is, but I can prove that it is a bug (see above).

[Abraham Getzler]

@Bob,  You might want to consider this Power Users board comment from MIT's Jeff Schiller on the future of Firebase before investing further into it ...

The purpose of this note is to clarify some facts about our support of Firebase.

We first integrated Firebase into MIT App Inventor when Firebase was a stand-alone company prior to its purchase by Google.

At that time Firebase offered a client side SDK for interacting with Firebase databases. This SDK was in the form of a collection of “Jar” files which worked on any Android platform. In particular it worked both on devices running Google’s flavor of Android and Amazon’s flavor, used on the Fire Tablets. Fire Tablets are quite popular in some environments and our ability to support Firebase on the Fire Tablets is an important feature.

Firebase offers several ways to authenticate the end-device to Firebase. In particular we used their “custom” authentication. This permitted us to have one Firebase account, run and paid for by MIT, that all could use without having the either pay money or even deal with an administrative setup process. You just put the component into your project and you had shared variables.

Of course we were concerned from the beginning that this might lead to a significant cost to MIT if usage was very high. However we took the risk, figuring we could also implement some mechanism to control our costs. This didn’t happen. Or more to the point, the cost of offering Firebase was not very significant compared to the cost of running the public service itself (Google used to help us pay for it, but isn’t currently. So we pay the public price for using App Engine). [Donations accepted :-) ].

Google purchased Firebase and has been integrating it into Google’s infrastructure. This has had two consequences.

1. They no longer offer the client side SDK that we use in the Component. Instead they have made the Firebase API part of Google Play Services. However Google Play Services is not available on Amazon Fire devices.
   
2. They appear to be deprecating the “custom” authentication we rely upon. The remaining authentication mechanisms require each developer to create (and pay) for their own account.

Firebase continues to work for us today because we are still using the now deprecated client side SDK that we downloaded from Firebase when they were an independent company. And, and this is a very important *AND*, the “wire” protocol used by Firebase has not changed, so the SDK continues to work. In other words, the way the bits are sent to Firebase (now Google) servers has not changed so the SDK we use still works.

However we may be living on borrowed time. At any time now Google may choose to “break” the wire protocol we rely on. When that happens the Firebase Component will quite suddenly break for everybody. The fact that the SDK is deprecated and no longer available as a stand alone package, means we are effectively on notice.

Updating the Firebase Component to use Google's supported API through Google Play Services will immediately mean that we will no longer support the Fire Tablet, not something we want to do. And although I believe as of today we may be able to kludge the use of "custom" authentication with the Google Play Services version, it is pretty clear that it isn't the direction Google is going in.

All of this is one of the risks one takes on when one chooses to use a proprietary infrastructure. There are plenty of cases where proprietary infrastructure, especially those provided by startup companies, have been shutdown. Sometimes on little notice.

This was one of our motivations for the CloudDB component. It is also the motivation of use choose an Open Source backend for CloudDB. The CloudDB server is based on the open source Redis project. MIT is providing the backend, but you can use your own Redis server if you wish. Most importantly, if you wish to develop an application that you really want to work over a long period of time, you can deploy your own Redis server and you are protected. This is not to say that MIT is planning on shutting down the server we provide. The point is you can protect yourself, even from us, by being in control of your own infrastructure. Similarly we are in complete control of the infrastructure we provide so we can be assured that we will not have to face the deprecation situation that we may face with Firebase.

I hope this helps explain things.

-Jeff

 

[Abraham Getzler]

(JIS Power Users post added to FireBase section of FAQ)

https://groups.google.com/forum/#!msg/mitappinventortest/wqJin0qO8M0/8xUafdROAgAJ

ABG

[Bob Glicksman]

@Abraham:  I do appreciate your re-posting this information.  It is very important that all developers, myself included, understand this.  There is, however, another set of issues at this transitory point in time.  Google Coud DB is in Beta, and Google warns developers that it may change at any time.  So until Cloud DB is out of beta, we are all faced with a choice between an implementation that may break because it is on the way out, or one that may break because it is on the way in :).  

I have looked at the App Inventor Cloud DB component and it works the same way as the FirebaseDB component did; in particular, for authentication.  So, as the Power User's statements says, this is only guaranteed to work long term if you deploy your own Redis server.  I am not a commercial developer.  I am a hobbyist (albeit a retired developer) who develops interesting projects and gives back by publishing my (well documented) projects on GitHub under an open source, non-commercial license (see:  https://github.com/teampracticalprojects  if you are interested).  So I am depending upon services like Google's to host web services that are needed for my projects.

One last point:  I do agree that when Google finally takes CloudDB out of beta, they are likely to deprecate Firebase Real-Time DB.   It makes no sense for them to maintain two cloud databases within their Firebase universe.  However, when they do this, they will have to provide their large installed based with a simple, if not totally transparent, migration path.  Otherwise, they will anger REAL developers (ones who make a living out of their apps) by suddenly breaking all of these legacy apps.  My own feeling about this (an opinion, not a known fact) is that Google will support all legacy Real-time DB authentication and queries in Cloud DB, giving their developer community an easy migration path to Cloud DB.  

I don't know if you are a commercial developer or not, but if you are (or have been, in the past), you will recognize that people who make a living out of their apps generally cannot afford to take time out to rewrite their curent products just because a cloud service changes.  Google knows this well and has a long-standing history about handling migrations so that their users don't have to put new projects aside to fix breakages in their current products.  This is the reason for my opinion on the subject; hence my general lack of concern about sticking with Firebase Real-Time database for the time being.  Of course, I could well be wrong.  If I could predict the future with high accuracy, my retirement might be spent sailing my 300 ft yacht around the Carribean rather than doing little microcontroller iOT projects :-).

[Abraham Getzler]

Google Coud DB is in Beta, and Google warns developers that it may change at any time

Where did you find a connection between CloudDB, the AI2 CloudDB component, and Google?

I don't see one.

ABG

 

[SteveJG]

To keep things straight Bob,  the CloudDB interfaces to a Redis server database.   It has nothing to do with Google.

http://ai2.appinventor.mit.edu/reference/components/experimental.html#CloudDB

and 

https://redis.io/  

and as stated earlier  Here is what can be done with a Redis database (CloudDB)  https://redislabs.com/redis-enterprise/cloud/  

What you refer (and Jeff referred to)  as a Google Cloud DB) is probably this:  

https://cloud.google.com/sql/   

What App Inventor's CloudDB links to is MIT's Redis web server  and which can be linked to a developers Redis server.

[Bob Glicksman]

I just assumed that the AI2 CloudDB component is what you use to connect to the Google Firebase Cloud DB ("Cloud Firestore").  Not correct?  

I found the statement from Google about Cloud DB being in  beta on the Firebase.Google.com.  Google's comparison between the two is here:  https://firebase.google.com/docs/database/rtdb-vs-firestore?authuser=0.  

Am I confused about which Google product we are talking about?  It's entirely possible that I am :) 

[Bob Glicksman]

Ah yes, I did confuse this with Google's Cloud Firestore.  Sorry.