Paging large data sets and how to make Apps Script understand generators

[Bruce Mcpherson]

We’re all used to dealing with iterables such as Arrays, Maps and Sets. Creating a collection of input items, transforming then outputting is what we do all the time. However, sometimes that’s not possible – for example, where we have a potentially large input data set of unknown size that would be too big to fit in memory, or perhaps an output method with rate limits. Dealing with items 1 by 1, despite being by far the simplest to code, is often the least efficient to execute – so we use a paging method (chunker) to get an input chunk at a time, and a bulk method (bulker) to flush out the transformed results.

All of that can get a bit tricky, especially if you add asynchronicity into the mix.The use of iterators and generators can make coding as simple as 1 in -> 1 out, while handling chunking and bulking automagically. In this article, we’ll look at how to implement these.

You can think of using a chunker and a bulker together together as a processor with 2 different sized funnels being filled and emptied at potentially different rates.

I’ll also cover generators, and how to make them work in Google Apps Script even though it doesn’t understand the syntax.

Article here - https://ramblings.mcpher.com/paging-large-data-sets-and-how-to-make-apps-script-understand-generators/