Re: [Firebase] Importing large data fails

[Michael "Kato" Wulf]

Can you break up the dump? If you enter individual paths in Firebase, you can export them in part and then import them back to the same path. Might work around the issue for the short term.

On Thu, May 9, 2013 at 7:26 AM, racetrack <amir...@gmail.com> wrote:

Hi,

I'm trying to import a dump of one Firebase into another Firebase, but I'm getting errors:

curl "https://exportfrom.firebaseio.com/.json?auth=FIREBASE_SECRET&format=export"  > /tmp/export.json

curl -X PUT -d @/tmp/export.json "https://importinto.firebaseio.com/.json?auth=FIREBASE_SECRET"

curl: (55) TCP connection reset by peer

It's worth mentioning that the dump (/tmp/export.json) is 52MB, and I have no issues importing small files into Firebase.

thanks

 

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Michael Wulf
www.wordspot.org

[Michael Lehenbauer]

It looks like we have an internal limit of around 30MB for a single incoming REST request.  Breaking up the data is the easiest short-term solution.  Else, feel free to ping me off list and we can figure out a way to get your data imported.

Sorry for the inconvenience!  We'll look into removing the limit or at least providing a useful error message.

Thanks,

-Michael

[Joshua Haas]

I'd be interested to know what you guys end up doing... My backup solution right now is a daily export of my firebase into s3, so if that operation isn't easily reversible that's something I want to keep tabs on!

[Michael Lehenbauer]

Worst case, we can certainly do it for you manually.  It should also be pretty easy to do the piecemeal approach from a node.js script or similar.  And it looks like we already have an internal bug open to remove the limit entirely.  I'll look into bumping its priority.  We know backing up / restoring your entire Firebase is a necessary requirement. :-)

Thanks!

-Michael

[Michael Lehenbauer]

To follow up on this thread, we just released a node module to make importing large chunks of JSON easy and painless.

https://github.com/firebase/firebase-import

(We're still intending to remove the limit from the REST api, but that'll be a little further down the pipeline.)

-Michael

[rr]

Has this issue been resolved ? I understand this limit is imposed on REST requests, how about the Import button on the FB GUI ? Is there a limit there ? Do you have an idea of the largest file ever exported/imported ? Thanks !

[Jacob Wenger]

Hey there,

All of the limits on your data are listed in our documentation. You should not be see any limits to the export functionality from your Firebase dashboard. However, if you are trying to transfer GB of data, this request could take some time to complete. This is simply a matter of how fast you can transfer data, not of Firebase itself.

If you are having issues importing or exporting your data via the Firebase dashboard or the REST API, please contact sup...@firebase.com and we can help you more easily there.

Jacob

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[rr]

Great ! Thanks, I am not having issues, just wanted to make sure upfront there are no surprises down the road.

Cheers !
-odie

--
You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/-GGAXdhe6kA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-tal...@googlegroups.com.