How to process a list of every file on My Drive - running out of quota

[Michael Ellis]

Hi All!

I'm trying to use the various scripts that have been suggested to locate and list all files which have multiple folders as parents in My Drive.

My trouble is, the scripts all run out of quota, just trying to build the list (it never gets to writing anything to the spreadsheet).   There are 10's of 1000's of files in my Drive because we converted all our old on-premise servers years ago to Drive.  

I've been studying the code samples of CBL and Bruce McPherson's batch processing examples.   I believe I understand them generally and could reproduce them, but I'm struggling in my use case.

Is there a way to get a paginated list from the Query function which gives me the files?    How can I break it down into smaller chunks which I can process, somehow mark as completed so that the next batch of files can be processed?   I don't know if I can depend on the Query function to return the list in the same sequence every time.   What key can I use to tell the Query function to start at "Previous Key" and return to me the next 1000 entries (or something like that).   I can wait for the list to generate if it has to do it in chunks, even if it takes me several days to accumulate the list running every 5 minutes or something like that.   I just don't know how to manage the Drive files list in groups.

Any tips would be really appreciated.

Here's the reference libraries I've been studying:  Take a look at the CBL library for batch processing and Bruce's site for parallel processing.

Here's a similar post which gave me some ideas for the code, but not the solution to the part I'm struggling with:  https://groups.google.com/forum/#!msg/google-apps-script-community/CfcuiH_t9P8/ZGODS4cxAAAJ

Thanks!

Mike

[Andrew Roberts]

Hi Michael,

I came across the similar issue when I was trying to list all my files - timeouts rather than quota! 

You might be interested in collaborating on this project - DeleteOldFiles. It's main purpose is, as it says, delete old files but it does that as a two stage process: 1. list files, 2. delete files. The script times out listing lots of files (before reviewing and deleting old ones). If we could get batch processing working on this it would be easy enough to also flag files that have multiple parents. 

At the moment the "list files" does cache the list so you can at least retrieve that if the script times out.

I'm thinking that there must be some kind of batch call you can do with the Drive API but I've not explored it yet.

--
You received this message because you are subscribed to the Google Groups "Google Apps Script Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-script-c...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-script-community/1aa29f97-2731-4e7c-88aa-9dc078799093%40googlegroups.com.

[Kim Nilsson]

@Andrew,

that URL seems wrong, as I end up on a 404.

[Michael Ellis]

Andrew,

I would be very happy to collaborate on that together.   I know you have more programming skills than I do (I've already learned a lot from your examples over the last few years), but I'm game to help however it makes sense to get this accomplished.

I also cannot use that link - it shows a 404 error for me.   I did a search on Github for All Users for the term DeleteOldFiles and found a number of entries, but none are relevant to AppsScript (only other languages seem to be present).

How do you think we proceed?

I'm open to a Google Meet to discuss, if you want.  Give me your timezone and the best time to connect.

Mike

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-script-community+unsub...@googlegroups.com.

[Andrew Roberts]

My bad - I'd not made the repo public yet. This is the library that is accessed through the minimal bound script on this test GSheet you can take a copy of.

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-script-c...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-script-community/1aa29f97-2731-4e7c-88aa-9dc078799093%40googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "Google Apps Script Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-script-c...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-script-community/e0d0e348-6b48-46c8-a5a0-ea0cbe27778d%40googlegroups.com.