How to limit memory use?

[Richard Cooke]

When I ran this command on a user:

gam config auto_batch_min 1 redirect csv ./filelistperms.csv multiprocess user someb o...@acme.com print filelist fields id,title,permissions,owners.emailaddress query "(visibility='domainCanFind' or visibility='domainWithLink')"

I watched it run for hours and hours.  When the file count got over 1.5 MILLION! (going up 100 at a time), I realized my system memory was getting low.

And as I watched, the swap memory was starting to get used.

This was my second attempt at this script, my first try was running unattended, and I came back to find my laptop crashed.  I see now it likely ran itself into the ground - out of memory.

So what/how do I avoid this?  Can I limit the number of files it processes?  Or better manage memory.  I assume its the results file that is filling memory because the file is not there or in the gam cache folder while its running, or after I Ctrl-C it.

Thanks in advance.

[Richard Cooke]

I never did figure out how to limit either: 

 - The amount of RAM GAM uses for a file buffer, or

 - The number of items it processes at once.

Either would have made it easily workable on any machine. 

I wound up switching to a Ubuntu 20 box with 32GB of RAM and very fast CPUs.  On this machine, the first part that generates the raw file data took around 5 hours, the file was 1.7GB.  This was the step that was crashing my 16GB laptop.

The second command reduced that to a 243MB file with 1.65M lines!  

The third command took 8 DAYS to process all the files!

Watching the resource monitor, the only real impact was to RAM.  The CPUs were hardly doing anything.  Network traffic was pretty light for most of the time as well (surprisingly).

I'm posting this for the next person to run into this issue, and needs some idea of what to do:  Get more RAM.

[Ross Scroggs]

Richard,

Do: gam config verify

What is the value of: num_threads

This is how many parallel processes gam uses.

You mention three commands, I only see one. What are the others?

Ross

--
You received this message because you are subscribed to the Google Groups "GAM for Google Workspace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/a32b4ccb-6924-41fa-8076-513049fa030cn%40googlegroups.com.

--

Ross Scroggs

ross.s...@gmail.com

[Richard Cooke]

Hi Ross, how the heck are you?

num-threads = 5

I am adapting/using the commands from the script "GetSharedWithDomainDriveACLs.py".  The 3 commands, done one user at at time, are:

1. gam config auto_batch_min 1 redirect csv ./filelistperms.csv multiprocess user road_...@acme.com print filelist fields id,title,permissions,owners.emailaddress query "(visibility='domainCanFind' or visibility='domainWithLink')"
2. python3 GetSharedWithDomainDriveACLs.py filelistperms.csv deleteperms_road_runner.csv
3. gam csv ./deleteperms_road_runner.csv gam user "~Owner" delete drivefileacl "~driveFileId" "~permissionId"

I did it this way to get records for each user on what files were changed, just in case....  Also handy to get the totals per user for my report.

[Ross Scroggs]

Richard,

Step 1 can be simplified as you are referencing a single user.

1. gam redirect csv ./filelistperms.csv user road_...@acme.com print filelist fields id,title,permissions,owners.emailaddress query "(visibility='domainCanFind' or visibility='domainWithLink')"

Ross

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/4f224662-8488-42e2-92b6-2db861e9d611n%40googlegroups.com.

--

Ross Scroggs

ross.s...@gmail.com

[Richard Cooke]

Thanks, but can you enlighten me on how that affects resources?

Before, I would watch the resource meter for free RAM slowly go down until it ran out.  Then the OS would try and use swap. At some point it runs out of any free RAM and just locks up.  So I thought the need was to limit how much it processed at once.  either by # of results, or RAM use.

Was I close?

[Ross Scroggs]

Richard,

"config auto_batch_min 1 redirect csv ./filelistperms.csv multiprocess" causes GAM to setup a child process to collect/merge the output

from all of the subprocesses; in your case, as there is a single user, there is no need for this. It probably isn't a big savings but run

a timed test of the two forms to see how much difference it makes.

Ross

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/995ce737-532e-4855-a6cf-a2f40e8215f5n%40googlegroups.com.

--

Ross Scroggs

ross.s...@gmail.com