Firestore backup/export and import

[MMLLEVVY]

Hi all,

trying to understand whether I can migrate my app to Firestore and it seems, that it should be fairly easy. One thing I noticed is the fact, that there is no autobackup/backup/export feature - do you plan to add it? Additionally, do you plan to allow importing data from json/xml or any other data format?

Ps. Surely I can do backup on my own, connecting e.g. from node.js and fetching data for all collections, but this will be very costly (lots of reads, right?).

[Cory Wolff]

That's what I did, but I used Python. I exported my RTDB in json, then looped through the file and added to Firestore db. ~7,000 records only took a few minutes.

[Michael Lehenbauer]

Hi Marek,

We do intend to have options for importing / exporting your Cloud Firestore data in the future, but details are still TBD.  For now your best bet is rolling your own solution like Cory suggests.  Sorry we don't have a better answer yet!

-Michael

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/4889b9b7-bba3-4218-b753-206016c005b8%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[geoffrey poremba]

Ugh, this just about sucked all of my enthusiasm away for this product. Seems like such a trivial thing to integrate into the console. After all, the RTDB already has one. Maybe tomorrow I'll get some enthusiasm back to take some time to throw this together. Frankly, though, I would have thought this would be one of the first things you'd put together. I mean, how do you test large datasets to see if the stuff works? You surely don't create the records by hand. Obviously you have tools to bulk load up data into the datastore. A json importer should have been done before the current mechanism. Probably would have taken less time as well. I think your Product Manager dropped the ball on this one.

On Thursday, October 5, 2017 at 9:33:17 AM UTC-7, Michael Lehenbauer wrote:

Hi Marek,

We do intend to have options for importing / exporting your Cloud Firestore data in the future, but details are still TBD.  For now your best bet is rolling your own solution like Cory suggests.  Sorry we don't have a better answer yet!

-Michael

On Wed, Oct 4, 2017 at 1:12 PM, Cory Wolff <cwol...@gmail.com> wrote:

That's what I did, but I used Python. I exported my RTDB in json, then looped through the file and added to Firestore db. ~7,000 records only took a few minutes.

On Wednesday, October 4, 2017 at 12:43:19 PM UTC-4, MMLLEVVY wrote:

Hi all,

trying to understand whether I can migrate my app to Firestore and it seems, that it should be fairly easy. One thing I noticed is the fact, that there is no autobackup/backup/export feature - do you plan to add it? Additionally, do you plan to allow importing data from json/xml or any other data format?

Ps. Surely I can do backup on my own, connecting e.g. from node.js and fetching data for all collections, but this will be very costly (lots of reads, right?).

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

[Samuel Stern]

Hi Geoffrey,

I understand your frustration, and you're not the only one waiting for this!  However it's not really "trivial" as you say.  Any JSON data is valid Realtime Database data, so import/export is much simpler for that product.  For Cloud Firestore there are more questions that need answering.  What does the document/collection structure look like when exported?  Is JSON the right format?  How do we reliably import/export data sets that are much too large to be reasonably handled in a web interface?  We have some ideas, but we are still fine-tuning our plan.

We did not rush to get this done for Cloud Firestore's first Beta launch because we want to make sure we get it right.  We will definitely have this feature before we exit Beta and we hope you'll stick around and use it then.  For now if you have any questions about writing an import/export script we're happy to answer.  It's not too hard using our Python/Java/Go/Node client libraries.

Oh and if you're concerned about how the product will perform on large datasets, you will be happy to know that Cloud Firestore is built on top of the technology that powers Cloud Datastore and Cloud Spanner.  So it can handle absolutely massive datasets (Snapchat and Super Mario Run are some prominent Datastore customers).  And most importantly, because Cloud Firestore does not allow queries that don't match an index we can guarantee that query performance scales with the size of the result set.  So querying for 10 results out of 100 documents is the same as querying for 10 results in 100 million documents.

- Sam

P.S. We try to keep everything friendly on this mailing list.  Product managers are people too!  We're working hard to bring you a great product, please try and give us a chance.  Especially during Beta.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/b0a724a8-3a7b-4876-b471-23551a89433f%40googlegroups.com.

[Ian Grossberg]

Hey y'all,

Running into this issue ourselves and then finding this thread inspired us to throw together a simple general purpose backup tool you can find here: https://www.npmjs.com/package/firestore-backup

If you do end up using it feedback is more than welcome. Hope it helps!

On Wednesday, October 4, 2017 at 12:43:19 PM UTC-4, MMLLEVVY wrote:

[geoffrey poremba]

Hey Ian,

Thanks for sharing your stuff. In looking at your usage instructions, I am a little confused at what you're tool does and how it helps.

For example, If I were to create a tool, it would take a firebase rtdb json export and it would push this structure into cloud firestore.

My big problem is that I have data in firebase rtdb and I want it in cloud firestore. Firebase has a handy json import/export and I'd just like to see the exact same functionality for cloud firestore. It looks like your project provides part of the functionality, which is doing an export, but how do you do the import? Is there something in Cloud Firestore that I missed that allows you to import json like Firebase does?

Best Regards,

Geoff

[Ian Grossberg]

I understand, it doesn't sound like it's suited for your needs then. Rather this tool currently only does exporting of Firestore databases with plans to add importing from the same format.

Brainstorming, how would you imagine describing the relationship between RTDB's overall structure and individual documents/collections in Firestore? Would you build a schema of nested, relative XPaths that would encapsulate a document or collection of documents? I'd imagine also having to say whether an XPath should be a subcollection or a root level collection perhaps?

[geoffrey poremba]

Well I guess I would simply do what the current tool in Firebase does. Firebase will Export json, which you can then process according to the following rules as you import into Cloud Firestore:

1. If it is a json list, create a collection in cloud firestore

2. If it is a json object, create a document

You can iterate through the Firebase json structure, as cloud firestore supports nested documents and collections. Maybe this week I will hack something together and share it with you all. 

[King John]

Hi Michael,

  I have the same frustration as Geoffrey. 

  I have a data set with over 300K entries, and I created a simple nodejs script to upload it to the firestore, but failed badly, I believe it was due to the limit of writes/reads with my free tier account, because I can upload my small sample dataset.  

  If there is no bulk upload mechanism available, how can people upload big dataset to the firestore for testing?

  Do you have any workaround/suggestion for me to upload my dataset? 

thanks a lot. 

-John  

On Thursday, October 5, 2017 at 12:33:17 PM UTC-4, Michael Lehenbauer wrote:

Hi Marek,

We do intend to have options for importing / exporting your Cloud Firestore data in the future, but details are still TBD.  For now your best bet is rolling your own solution like Cory suggests.  Sorry we don't have a better answer yet!

-Michael

On Wed, Oct 4, 2017 at 1:12 PM, Cory Wolff <cwol...@gmail.com> wrote:

That's what I did, but I used Python. I exported my RTDB in json, then looped through the file and added to Firestore db. ~7,000 records only took a few minutes.

On Wednesday, October 4, 2017 at 12:43:19 PM UTC-4, MMLLEVVY wrote:

Hi all,

trying to understand whether I can migrate my app to Firestore and it seems, that it should be fairly easy. One thing I noticed is the fact, that there is no autobackup/backup/export feature - do you plan to add it? Additionally, do you plan to allow importing data from json/xml or any other data format?

Ps. Surely I can do backup on my own, connecting e.g. from node.js and fetching data for all collections, but this will be very costly (lots of reads, right?).

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

[geoffrey poremba]

Ok, I decided to devote a few hours to bang out an import tool for Cloud Firestore. I'll be posting it to a Github repo for you all to use if you want. It works like a champ with my datamodel, no guarantees that it will work with yours. However, it should be simplistic enough for you to modify it to suit your own purposes if it doesn't.

G.

[David Mrozinski]

That will be a real life saver.   Look forward to the github posting.

Dave

[geoffrey poremba]

This is very quick and dirty, but feel free to try it out and see if it works for you:

https://github.com/xgeoff/CloudFirestoreDataLoader

G.

[Dale Nguyen]

Hi, I just write a backup and restore for Firestore. You can have try at the guide from my GitHub.

https://github.com/dalenguyen/firestore-import-export

Thanks,

[Max Grand]

Hi Dale,

not sure what you mean by 'Change the databaseURL when initializeApp with your own' and '<your-collection-name>' after 'node export.js' Could you further elaborate?

[maxim...@gmail.com]

Cory,

is there any chance you can share the code you used for the upload part?

Thx,

Max

[Cory Wolff]

Sure. You can view it here: https://github.com/cwolff411/workstuffs/tree/master/firetore%20import

Just keep in mind that when I wrote this script it was about a week after they announced Firestore. Some things may have changed i.e. they might have added support for geocode types.

I exported my RTDB to json, loaded it using Python and simply looped through the records and added them one by one.

Good luck!

[Maximilian Grand]

Thanks Cory for sharing this, I'm also working on a solution in Node.js

Soon to come for the entire group here

--
You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/k1CQK6YH2Zs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/8fadcef7-e335-4fbc-8f8b-4ef9d12f15ae%40googlegroups.com.

[maxim...@gmail.com]

For those interested:

We have developed a comprehensive solution to import large .xlsx files directly to either Firestore or RTDB by using Node.js. Hope this can be useful to those seeking a straightforward way to solve this problem. See here:

https://github.com/alexandrtovmach/xlsx2firebase

Also here:

https://stackoverflow.com/questions/46640981/how-to-import-csv-or-json-to-firebase-cloud-firestore

Cheers

[Dale Nguyen]

Hi Max, 

You will see it when you open export.js file :)

Thanks,

[Jared Loosli]

Hi all...looks like we're all trying to do the same things. For what it's worth, I put together a node package for importing and exporting Firestore data that works well for us. I was looking to get the export down to a single json file while keeping collections separate....I'd love to get some feedback on it and join forces with others who are working on the same thing. Package link is below.

https://www.npmjs.com/package/node-firestore-import-export