Firestore batch writes are slow - am I doing it correctly?

[opy...@gmail.com]

Hello,

Our Firebase functions are in Belgium, and Firestore is in London.

The ping between Brussels and London seems to be around 10ms.

My function has this timing code:

async function handle (path, uid, companyName) {
  console.time('handle')
  console.time('search')
  const result = await searchByCompanyName(companyName, 5)
  console.timeEnd('search')
  console.time('write')
  return firestoreDb.writeBatch(path, result)
    .then(() => {
      console.timeEnd('write')
      return console.timeEnd('handle')
    })
}

The write timing seems to hover around 150-200ms.

function writeBatch (path, arr, idRef = 'id') {
  const batch = db().batch()
  const ref = db().collection(path)

  console.log(`batching ${arr.length} results`)
  return Promise.all(
    [
      arr.map(async item => {
        let itemRef = ref.doc(item[idRef])
        return batch.set(itemRef, item)
      })
    ]
  ).then(() => {
    return batch.commit()
  })
}

I also tried this:

function writeBatch (path, arr, idRef = 'id') {
  const batch = db().batch()
  const ref = db().collection(path)

  console.log(`batching ${arr.length} results`)

  arr.forEach(item => {
    let itemRef = ref.doc(item[idRef])
    batch.set(itemRef, item)
  })

  return batch.commit()
}

Are there any other improvements I can make? 150-200ms seems like a lot considering the ping itself is only 10ms.

The array is also capped to 5 results at all times.

Thanks,

Juan

[Hiranya Jayathilaka]

What does db() do?

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/20ab3b44-ab3f-4adb-8431-2cce4311a662%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Hiranya Jayathilaka | Software Engineer | h...@google.com | 650-203-0128

[opy...@gmail.com]

Gets a db handle.

let _db = null

/**
 * Initialises the DB handle if it hasn't yet.
 *
 * PS remember this for date handling:
 *
 * // Old:
 * const date = snapshot.get('created_at');
 * // New:
 * const timestamp = snapshot.get('created_at');
 * const date = timestamp.toDate();
 *
 * @returns {*}
 */
const db = () => {
  if (_db === null) {
    const settings = {timestampsInSnapshots: true}
    _db = admin.firestore()
    // gets rid of
    // "The behavior for Date objects stored in Firestore is going to change AND YOUR APP MAY BREAK."
    _db.settings(settings)
  }
  return _db
}

[Elie Zedeck RANDRIAMIANDRIRAY]

Hey Juan,

I run batch operations a lot as well, and indeed, it is slow, just like yours! Some of my ops (the max 500 doc updates) goes as high as 2500ms ... almost 3 seconds. Fortunately, it's not a use case that needs speed, so I'm ok with that.

@Firebase: But I'm also curious is it really meant to be so slow? Thinking in scaled use cases, I don't remember exactly how many is the maximum writes on Firestore, but I think during beta it was 10000 and this limit was supposed to be removed when GA (I'm too lazy to check the numbers now). I mean, 500 updates taking 2.5s, that's not even close to the Beta's 10000, interpolate and imagine how long would it take to complete 10000 operations at that pace. It is true that I did not run my ops in parallel, but still!

Best regards,
Elie Zedeck

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CAE6e%2BPhytzC87PhXhaXU4O0ggQK0zYAhFjYQ%3DrNSUVDxmA4S0Q%40mail.gmail.com.

[opy...@gmail.com]

I ran an experiment just using straight writes (not batch) and it still takes ~100ms. A read seems to take the same.

I now wonder if there is some middleware which consistently adds a few tens of milliseconds to any CRUD going from my function to Firestore.

const handle = async (uid, data) => {
  const companyNumber = data.number
  console.time('read')
  const teamProfile = await firestoreDb.read(`team/${companyNumber}`)
  console.timeEnd('read')


  const data2 = _.assign({}, data, {
    random: Math.random().toString(36).substring(7)
  })

  console.time('write')
  return firestoreDb.write('junk/' + uid, data2).then(() => {
    return console.timeEnd('write')
  })
}

[opy...@gmail.com]

To any Googlers reading this... in your opinion, is the ~100ms overhead slow, or not?

[Samuel Stern]

Hi there,

I just tracked down internal targets and while I can't share much, we consider 100ms for a write to be in the acceptable range.

- Sam

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/8d5c7cdb-63c8-4ffd-8633-06f59aaa288f%40googlegroups.com.

[opy...@gmail.com]

Thanks for the reply.

I say ~100ms, but looking at the numbers (see original post) it fluctuates between 100 and 200ms. Is that still an acceptable range?

[Samuel Stern]

Yes for many workloads, 100-200ms is still reasonable and does not point to anything going wrong.  You can probably speed up writes by turning off some single-field indexes and making your documents smaller but I wouldn't expect anything too dramatic.

- Sam

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/f1053fba-68ce-4b5e-9d9a-105c7719fb33%40googlegroups.com.