GAM print and PowerShell Objects

[William Yox]

This is less of a GAM question and more of a PowerShell one and is likely out of scope, but I wanted to see if anyone was doing GAM-related tasks in PowerShell I know there are a couple of us around here so hopefully you'll see it!

I'm creating a PowerShell script that does several things related to deprovisioning. One of which is applying labels to a target users files (I should be using DLP rules for this probably). To do this, I need a list of files to Label. 

I'm trying to avoid the built-in CSV functions in GAM and use PSObjects for iteration instead. Furthermore, I'm using ForEach-Object's -Parallel parameter to speed up the process. 

This example gets a list of users into $TargetUsers. Since 'print' creates CSV output we can easily create a new object.

 $TargetUsers = gam print users query $orgUnitPath fields primaryEmail,Suspended | ConvertFrom-Csv

Now, I want to get a list of files for each $TargetUser.primaryEmail, but process them in parallel, AND add the results to a single PSObject.

$TargetUsers.primaryEmail | ForEach-Object -Parallel {
            $data += gam user $_ print filelist showownedby me fields id,name,owners.emailaddress,mimetype,permissions,quotaBytesUsed,size,webContentLink,webViewLink | ConvertFrom-Csv
        }

<Log> $data

This works...kinda. Since we're running this for each user, a new row with the headers is inserted into the resulting object. 

Example:

$data 

Owner     id       name      mimeType                           owners
-----  --       ----      --------                           ------
user1@ ItemID 1 FileName1 application/vnd.google-apps.folder 1
user1@ ItemID 2 FileName2 image/png                          1
user1@ ItemID 3 FileName3 image/png                          1
Owner  id       name      mimeType                           owners
user2@ ITemID 4 FileName4 application/vnd.google-apps.folder 1
user2@ ITemID 5 FileName5 image/png                          1
user2@ ITemID 6 FileName6 image/png                          1

Currently I'm just using gam redirect csv for this...and if this proves fruitless, I'll just keep doing that.

[Chris River]

You should be aware that the number of permissions returned from each file can be variable, which Powershell doesn't handle very well when combining. You'll end up with the number of columns resulting from the object that is first output from GAM into the $data object. So, if you need those permissions, you should isolate them into their own objects to ensure that you capture all of them from their own files.

Another thing to be aware of is that you need to use a thread safe container to store the results from, otherwise you'll end up losing some data or potentially even some garbled data. This article provides more information, but essentially you'll want to use something like:

$data = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()

I couldn't get your piece of script to work for me, but usually you also need to use $using:data to be able to use the external variable in the foreach -parallel script.

So, you might try something like (you can replace $tempdata to $data below if you prefer, if that is less confusing or easier to adapt to your current script):
$data = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()

$TargetUsers | ForEach-Object -ThrottleLimit 10 -Parallel {

  $tempdata = $using:data
  $tempdata.TryAdd($_.primaryemail, (gam user $_.primaryemail print filelist showownedby me fields id,name,owners.emailaddress,mimetype,permissions,quotaBytesUsed,size,webContentLink,webViewLink | ConvertFrom-Csv))
}

After that is finished, you'll be able to access the dictionary $data with the data accessible by email address, such as by $data['test...@example.com']. You can easily (and safely) export these to separate CSV files. Combining the data into a single object/CSV is tricky. I think one way is to find the object with the most columns, then combine them with that one at the top. I think it is safe enough for something like this, but if any objects contain any properties that that first object has, then they will still be lost.

[Maj Marshall Giguere]

If you want only one permission per row, tack on the "oneitemperrow" option.

---

The contents of this email are intended only for the recipient(s) listed above. If you are not the intended recipient, you are directed not to read, disclose, distribute, or otherwise use this transmission. If you have received this email in error, please notify the sender immediately and delete the transmission. Delivery of this message is not intended to waive any applicable privileges.

--
You received this message because you are subscribed to the Google Groups "GAM for Google Workspace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/35175858-d7b1-4e23-958d-b314d481185en%40googlegroups.com.

[William Yox]

You should be aware that the number of permissions returned from each file can be variable, which Powershell doesn't handle very well when combining...Another thing to be aware of is that you need to use a thread safe container to store the results from, otherwise you'll end up losing some data or potentially even some garbled data. 

I was not aware that that was a problem for PowerShell and very good to know. Luckily in this case, the very first column, 'permissions' is the one I'm looking for - Any file with more than one permission (or shared). Hopefully that won't be an issue, but I'm going to test that with a few users and review the results. I read over the article when reviewing the documentation for -Paralell and will make some adjustments to the script accordingly. 

For some context (if you're curious), here's an overview

The script has several functions, 

Enable users

• this function uses gam to unsuspend users, give them a license, and add them to a security group which can use drive labels.

Labelling function

• from a $users object, use gam print to get all files that are shared into an object
• apply drive file labels to ids in the $sharefiles object  files as the user

Disables users

• Does the reverse enable users

Logging

• Each function's step writes the output of the gam command to a json file

It seems like the easiest (and possibly safest) method is to continue to use CSVs in the labelling function to ensure the accuracy of the data. 

You've given me some things to think about. As always, I appreciate it Chris. 

[Chris River]

The oneitemperrow was a good suggestion, Maj, thanks!. In your case, William, it sounds like you might find it helpful to filter the output so the extra permissions columns aren't included. You can do this via config csv_output_header_drop_filter "permissions\..*". You can also add a row filter so that only the rows with more than 1 permission are output, via csv_output_row_filter "permissions:count>1", unless you want to keep those as well for reference or whatever.

So, the modified script would be:

$data = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()

$TargetUsers | ForEach-Object -Parallel {
  $tempdata = $using:data
  $tempdata.TryAdd($_.primaryemail, (gam config csv_output_header_drop_filter "permissions\..*" csv_output_row_filter "permissions:count>1" user $_.primaryemail print filelist showownedby me fields id,name,owners.emailaddress,mimetype,permissions,quotaBytesUsed,size,webContentLink,webViewLink | ConvertFrom-Csv))
}
$data

Since the extra permissions. properties will be unreliable, I think it would be good to drop those to prevent confusion, make it easier to join the data together as needed, and to limit the unnecessary data size (e.g. if the first file happens to have 100 permissions or something).