Firestore getDocument takes too long to return

3,113 views
Skip to first unread message

Rodrigo Sol

unread,
Nov 6, 2017, 12:26:01 PM11/6/17
to fireba...@googlegroups.com
Hi there,

Is firestore too slow, or what I'am doing wrong?


I have designed my schema like this: 

- RealProperty
--- Photos
--- Tags
--- Legal
--- Commercial
--- Owner
--- Search

The search sub document is created by a cloud function. It consolidates all data I need to retrieve when a user do a search. 

For each keyEntered event fired in geofire I call the following code to get my subdocument:

db.document(key).collection("search").document("unique").getDocument(...)

Geofire returns keys almost immediately, but the first firestore document takes more than 5s to be fetched. The subsequents getDocuments calls runs very quick too.

It search subdocument itself id just 8 primitive key pairs.


Is there a way to speed up this? Is there another approach more suitable for the scenario above?



Thanks

--

Michael Lehenbauer

unread,
Nov 7, 2017, 12:17:03 PM11/7/17
to Firebase Google Group
Hi Rodrigo,

Do you mind enabling logging with Firestore.enableLogging(true) and send me the output? This may just be expected latency during beta, but I can take a look at the logs and see if anything looks amiss.

Thanks!
-Michael

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CADbvOHdgmO9vm0Ks%3DRHVmCMoPb0smAd-mu6DuXLjRqvyLgm4rw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Rodrigo Sol

unread,
Nov 15, 2017, 10:31:00 AM11/15/17
to fireba...@googlegroups.com
Hi Michael,

To try to explain to you what is happening I have created a small project available on github (https://github.com/rarolabs/firestore_benchmark). 

When you click into start button, it creates N arbitrary records with M arbitrary keys. These N records are sent to RT Database and Firestore. Finally it shows some statistics.

I am using an iPhone 7s plus, connected to 20mb dedicated link from Brazil, it takes 2.38s (RT) and 43.91s (Firestore)  to write the same 1000 records with 10 key/value.

I am also sending the logs for a 100 records run as you requested. I really hope this is just a beta issue, or even better if I am doing something wrong. 

Can you take a look on my code?

Thanks.


To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAJCyStcq5p6CP-tnmxvVyX-Qdwz1%3D-6YYRuFazK1OvQFWvdakQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.
log.txt

Michael Lehenbauer

unread,
Nov 15, 2017, 3:23:22 PM11/15/17
to Firebase Google Group
Hey Rodrigo,

Thanks for the added info! The detailed measurements are very helpful. Looking into this a bit, the latency you're seeing for writes is probably expected at the moment, and it's actually not really a backend latency issue. It's a limitation in the client to only have 10 writes outstanding at a time at present (defined here if you're curious). This was essentially added as a means of cheap flow control, but we're intending to improve it and with your feedback, I'll see if we can get those improvements somewhat expedited. :-)

Anyway, as-is, your 1000 writes will essentially happen as 100 batches of 10, and 440ms per batch from Brazil seems within reason...  So I think that adds up.

Your original email was about getDocuments latency though, which would /not/ be related to this 10 max-pending-writes limit.  So if you're seeing issues there still, we should dig in.  For what it's worth, there have been a few read latency improvements going out this week, so there's a chance your issue has been resolved.  

Thanks!
-Michael

Michael Lehenbauer

unread,
Nov 15, 2017, 4:20:05 PM11/15/17
to Firebase Google Group
Hey Rodrigo,

Following up again with two notes:
  1. FYI- I may have been hasty to have blamed this on the 10 outstanding writes limit.  We'll dig into this more and see if that's really the limitation and how we can improve things going forward.

  2. The workaround here is to use batch writes (see here). You can do up to 500 writes in a single batch, and this should be much faster.
Thanks,
-Michael

Rodrigo Sol

unread,
Nov 17, 2017, 1:15:43 AM11/17/17
to fireba...@googlegroups.com
Hi Michael,

Can you take a look at these screenshots in attachment. I have added a new feature to mensure getDocuments time. 

The current behavior is close to my real situation. I need to fetch keys after a geofire query. 

The first getDocument call takes 3.9s to returns in firestore and just 190.5ms in Realtime Database. 

It sounds strange. What do you think?

Thanks.

On Thu, Nov 16, 2017 at 9:46 AM, Rodrigo Sol <rod...@rarolabs.com.br> wrote:
Thanks Michael,

You are right. My actual problem is about reads. It is not about writes. I do no have needs to write 1000 in sequence. I used batch write to store a documents with subdocuments and it worked great.

I will add a new feature to this benchmark app to mensure reads and I will let you know my results.

Thanks.


For more options, visit https://groups.google.com/d/optout.



--
IMG_3320.PNG
IMG_3319.PNG

Rodrigo Sol

unread,
Nov 21, 2017, 10:16:42 AM11/21/17
to fireba...@googlegroups.com
Hi Michael,

Can you take a look at these screenshots in attachment. I have added a new feature to mensure getDocuments time. 

The current behavior is close to my real situation. I need to fetch keys after a geofire query. 

The first getDocument call takes 3.9s to returns in firestore and it takes just 190.5ms in Realtime Database. 

It sounds strange. What do you think?

Thanks.
On Thu, Nov 16, 2017 at 9:46 AM, Rodrigo Sol <rod...@rarolabs.com.br> wrote:
Thanks Michael,

You are right. My actual problem is about reads. It is not about writes. I do no have needs to write 1000 in sequence. I used batch write to store a documents with subdocuments and it worked great.

I will add a new feature to this benchmark app to mensure reads and I will let you know my results.

Thanks.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
IMG_3320.PNG
IMG_3319.PNG

Michael Lehenbauer

unread,
Nov 27, 2017, 1:14:52 PM11/27/17
to Firebase Google Group
Hi Rodrigo,

Sorry for the slow reply. My guess is in the short-term this is just the performance you can expect at this point in Beta, but longer-term we'd definitely like to do better. I can log a bug to make sure this gets investigated. Can you clarify exactly what your app is doing?  Is it essentially just 1000 calls to db.document('collection/<random-doc>').get() ?

If you can provide a small code snippet, that would be even better.

Thanks,
-Michael

On Tue, Nov 21, 2017 at 1:19 AM, Rodrigo Sol <rod...@rarolabs.com.br> wrote:

Hi Michael,

Can you take a look at these screenshots in attachment. I have added a new feature to mensure getDocuments time. 

The current behavior is close to my real situation. I need to fetch keys after a geofire query. 

The first getDocument call takes 3.9s to returns in firestore and it takes just 190.5ms in Realtime Database. 

It sounds strange. What do you think?

Thanks.
On Thu, Nov 16, 2017 at 9:46 AM, Rodrigo Sol <rod...@rarolabs.com.br> wrote:
Thanks Michael,

You are right. My actual problem is about reads. It is not about writes. I do no have needs to write 1000 in sequence. I used batch write to store a documents with subdocuments and it worked great.

I will add a new feature to this benchmark app to mensure reads and I will let you know my results.

Thanks.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CADbvOHfqnGhxBEAviEfbGiWD2%3Db7DhATZW2X1tgozAL3qGjd1A%40mail.gmail.com.

James Spivey

unread,
Nov 28, 2017, 4:06:23 PM11/28/17
to Firebase Google Group
Following along as I am seeing some of the same latency issues. I have 3000 or so records, very basic, and see on average a 4 second latency between call and return. In beta so understood but clearly not something I can use as a POC to a client in that state. We started a migrate back to RTDB because of it but would really love to use Firestore long term for the needs.

Kato Richardson

unread,
Nov 28, 2017, 4:17:00 PM11/28/17
to Firebase Google Group
Hi James,

Unless you have some new dynamics to add to this, everything Michael said already would still apply.

4s sounds pretty long, but it'd be hard to make any guesses without a minimal repro and some sample data (which you should submit in a new thread instead of appending on to this one).

☼, Kato

On Tue, Nov 28, 2017 at 2:06 PM, James Spivey <spi...@zentostudios.com> wrote:
Following along as I am seeing some of the same latency issues. I have 3000 or so records, very basic, and see on average a 4 second latency between call and return. In beta so understood but clearly not something I can use as a POC to a client in that state. We started a migrate back to RTDB because of it but would really love to use Firestore long term for the needs.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/4c07e08c-095f-4c19-bf76-0c9114de9cdd%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Kato Richardson | Developer Programs Eng | kato...@google.com | 775-235-8398

Michael Lehenbauer

unread,
Nov 28, 2017, 8:28:48 PM11/28/17
to Firebase Google Group
On Kato's note, the more details you can provide the better.  Are you doing 3000 get() requests?  Or a single query that returns 3000 documents?  Actual data (the exact # of requests / results and the timing from request to response) would be best, of course.

In any case, thanks for the feedback. Performance is top-of-mind for us.

-Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CADypTEboRt8JwXcfHD-PD%3DcBM%3DmLR6z-d%3DHXOH16LR5wvPiCVA%40mail.gmail.com.

James Spivey

unread,
Nov 28, 2017, 8:36:36 PM11/28/17
to Firebase Google Group
Absolutely. Was slammed today but was going to build a little standalone to show usage and timing for yall to see/work off. It is a single request for a collection so not a document request a ton of times. I currently have slimmed the documents down to be as skinny as possible and have a separate collection with the full size documents and a reference to those fullsize appended to the minimized one to help save on bandwidth and improve speed. Will try to emulate what that looks like in the demo. I do not imagine the reference would add any delays in a collection call. Was not trying to drop a "it's having issues" on yall without something to look at, just been behind the 8 ball on tasks as we have a big release next month but wanted to comment on this to keep track of it in case there was an evolution in it. As an aside, I did that 3000 record add through individual calls (there is no other way I am aware of to have it auto generate the ID and do it in bulk) and it seemed to be pretty snappy, although it ran through a Functions call and i forgot to timestamp it so do not have exact numbers.


On Tuesday, November 28, 2017 at 6:28:48 PM UTC-7, Michael Lehenbauer wrote:
On Kato's note, the more details you can provide the better.  Are you doing 3000 get() requests?  Or a single query that returns 3000 documents?  Actual data (the exact # of requests / results and the timing from request to response) would be best, of course.

In any case, thanks for the feedback. Performance is top-of-mind for us.

-Michael
On Tue, Nov 28, 2017 at 1:16 PM, 'Kato Richardson' via Firebase Google Group <fireba...@googlegroups.com> wrote:
Hi James,

Unless you have some new dynamics to add to this, everything Michael said already would still apply.

4s sounds pretty long, but it'd be hard to make any guesses without a minimal repro and some sample data (which you should submit in a new thread instead of appending on to this one).

☼, Kato
On Tue, Nov 28, 2017 at 2:06 PM, James Spivey <spi...@zentostudios.com> wrote:
Following along as I am seeing some of the same latency issues. I have 3000 or so records, very basic, and see on average a 4 second latency between call and return. In beta so understood but clearly not something I can use as a POC to a client in that state. We started a migrate back to RTDB because of it but would really love to use Firestore long term for the needs.

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
--

Kato Richardson | Developer Programs Eng | kato...@google.com | 775-235-8398

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.

Kato Richardson

unread,
Nov 29, 2017, 11:12:32 AM11/29/17
to Firebase Google Group

Hey James, no worries. We understand busy : ) Looking forward to seeing the repro so we can look for some optimizations. Thanks for contributing back.

For generating auto ids for mass writes, you can just use something like docRef = db.collection("cities").doc(); id = docRef.id; to obtain the auto-generated id. That's buried in our docs here.

☼, Kato

​

To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.

To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/ee2bc4b9-13af-4852-bab6-5b1d4e2094f0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Rodrigo Sol

unread,
Nov 30, 2017, 10:17:00 AM11/30/17
to fireba...@googlegroups.com
Thanks Michael,

Yes. It is just 1000 calls to db.document('collection/<random-doc>').get() .

You can take a look at the source code here: https://github.com/rarolabs/firestore_benchmark



On Mon, Nov 27, 2017 at 4:14 PM, 'Michael Lehenbauer' via Firebase Google Group <fireba...@googlegroups.com> wrote:
Hi Rodrigo,

Sorry for the slow reply. My guess is in the short-term this is just the performance you can expect at this point in Beta, but longer-term we'd definitely like to do better. I can log a bug to make sure this gets investigated. Can you clarify exactly what your app is doing?  Is it essentially just 1000 calls to db.document('collection/<random-doc>').get() ?

If you can provide a small code snippet, that would be even better.

Thanks,
-Michael
On Tue, Nov 21, 2017 at 1:19 AM, Rodrigo Sol <rod...@rarolabs.com.br> wrote:
Hi Michael,

Can you take a look at these screenshots in attachment. I have added a new feature to mensure getDocuments time. 

The current behavior is close to my real situation. I need to fetch keys after a geofire query. 

The first getDocument call takes 3.9s to returns in firestore and it takes just 190.5ms in Realtime Database. 

It sounds strange. What do you think?

Thanks.
On Thu, Nov 16, 2017 at 9:46 AM, Rodrigo Sol <rod...@rarolabs.com.br> wrote:
Thanks Michael,

You are right. My actual problem is about reads. It is not about writes. I do no have needs to write 1000 in sequence. I used batch write to store a documents with subdocuments and it worked great.

I will add a new feature to this benchmark app to mensure reads and I will let you know my results.

Thanks.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAJCySteiQN-Z_RAc3bGpxKbcXdBremCQW8eKzg8qw0%2BnSmzXcg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages