Firebase collections --> bigquery

[michael griffith]

I've been reading the documentation on big query and looking at the documentation and I'm confused about how to make my data in Firebase/Firecloud stream to bigquery,  At this point I am not really interested in streaming the google analytics to BQ, I am interested in the actual data the app writes to my firestore collections.  We already have permission from users to use this data for analysis and modeling, but I cannot figure out how to move the data.  I thought there would be a way to just stream the data into a series of normalized tables for building dashboards with tools like Tableau and Superset. 

What am I missing here?  Is there an example somewhere?

Thanks in advance  

[Sam Stern]

Hi Michael,

The easiest way to stream Firestore to BQ is to use this Extension, take a look:

https://firebase.google.com/products/extensions/firestore-bigquery-export

- Sam

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/300aa002-96fb-4ba5-898a-0796f9603b05n%40googlegroups.com.

[michael griffith]

Sam, 

Thanks, I was able to export the data from firesstore to BQ.  A couple of follow up questions.  Once the extension is in place, I assume any changes will be propagated to the BQ data set?  Also, is there any way to flatten the JSON stored in firecloud into a normalized table?  In the exported data, the JSON just appears as a column that represents the document.  That's not really what I expected to happen, unless I am doing something wrong.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAHafJBqLVTk5TxyHyk0S0GMQPwHGsNizGMY3hHEaLzyhZz5LRg%40mail.gmail.com.

[Sam Stern]

Hi Michael,

Yes if you've deployed the extension then it will stream all new data from Firestore to BQ as it comes in.  So not only will you have the latest data but you'll also have a full change log.

If you go back to the link I sent you and read the section on "Generate schema views" you'll see how you can tell the Extension what your data looks like and it will break it out into a more legible format for you.

- Sam

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAFhAot4qm1PNKC6W800wDbTQVq87SaPsXAAvGNqQHzVveJnDrg%40mail.gmail.com.

[michael griffith]

Thanks Sam, 

I missed the step about creating a custom schema.  After reading the docs a little closer and tinkering a bit I was able to get a schema for a collection. Do I need to install the extension for each collection I want to export to big query?  

Thanks in advance

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAHafJBonR%3DGNzyMGSzaeboRC6E3A5hO7i4Nd8wX_XHqYGKh7Ww%40mail.gmail.com.

[Sam Stern]

Yep you can install multiple "instances" of the same Extension. In this case one per collection you want to mirror.  Also Extensions are open source so you can always go and create your own Cloud Functions based on the Extensions code to mirror all of your collections at once if you find managing N different extension instances too tedious!

- Sam

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAFhAot4_kw_T%3DYFd94%3D%2BBWx%2BhxpD9q7OwpyPYr4LEASPQBy8EQ%40mail.gmail.com.

[michael griffith]

Right on. thx!

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAHafJBpPUChHsR6QBOy1hqEsLsfRvzQM0kjWxhv48iPm0hpQLw%40mail.gmail.com.

[michael griffith]

So I have a collection with a map in firecloud, and I am trying to export that to big query.  The schema looks like this, but the map is never converted to the schema query in BQ.  What am I missing?

{
  "fields": [
    {
      "name": "participantId",
      "type": "string"
    },
    {
      "name": "surveyId",
      "type": "string"
    },
    {
      "name": "responses",
      "type": "map",
      "fields": [
        {
          "name": "question",
          "type": "string"
        },
        {
          "name": "response",
          "type": "string"
        }]
    }
  ]
}

[Sam Stern]

Honestly I'm not sure if that's working as intended or not! Might be better to ask the Extensions team directly on GitHub:

https://github.com/firebase/extensions

- Sam

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAFhAot76FCUW6jpnyhD3YVL1vpdDxM%3DzqKB5eGyhjPjPu2MDvw%40mail.gmail.com.