Inconsistent user counts between gam report user and gam print users?

[Ian Crew]

Hello:

I’m getting some extraordinarily strange results from the following commands (bolded to indicate the major differences):

$ gam config csv_output_header_filter email,accounts:creation_time redirect csv ~/gam_output/google_users_2024-06-25-2.csv report users fulldatarequired all allverifyuser REDA...@example.edu fields accounts:creation_time

Getting Reports for all users

Got 380259 Reports for all users on 2024-06-22...

$ gam config csv_output_header_filter email,accounts:creation_time redirect csv ~/gam_output/google_users_2024-06-25-3.csv report users fields accounts:creation_time

Getting Reports for all users

Got 379776 Reports for all users on 2024-06-23...

$ gam redirect csv ~/gam_output/google_users_2024-06-25-4.csv print users

Getting all Users, may take some time on a large Google Workspace Account...

Got 382019 Users: REDA...@example.edu - REDA...@example.edu                 

When I do an export from the Google Admin Console, I also get 382019 users.

What I’m confused by is why I’m getting different numbers of users with each of the three commands. I do know that the issue is real: I can find a user in the “print users” results that I don’t see in either of the “report users" results, for example this one:

gam config csv_output_header_filter email,accounts:creation_time report users user abcxyz...@example.edu fields accounts:creation_time

Getting Reports for abcxyz...@example.edu

Got 1 Report for abcxyz...@example.edu on 2024-06-20...

email,accounts:creation_time

abcxyz...@example.edu,2022-05-24T01:14:05-07:00

(As you can see, that user does indeed have an accounts:creation time, so I’m really not sure why they wouldn’t show up in either of the first two commands.)

Ideas most appreciated!

Thanks,

Ian

[Ian Crew]

More details:

This seems to affect both GAMADV-XTD3 6.76.07 and 6.70.07.

This affects any “report users” command: the fields don’t matter:

 ~/bin/gamadv-xtd3/gam report users fulldatarequired all allverifyuser REDA...@example.edu fields accounts:used_quota_in_mb,accounts:drive_used_quota_in_mb,accounts:gmail_used_quota_in_mb,accounts:g

plus_photos_used_quota_in_mb,accounts:is_archived,accounts:is_disabled,accounts:is_suspended,accounts:last_login_time,accounts:last_sso_time todrive

is also missing ~1800 users from the report.

This is so strange….

Cheers,

Ian

[Ian Crew]

So I figured out a sort of (very slow) workaround for this, as follows:

# use gam report users to get the data we can

~/bin/gamadv-xtd3/gam redirect csv ./gam_report_output.csv report users fulldatarequired all allverifyuser REDA...@example.edu fields accounts:used_quota_in_mb,accounts:drive_used_quota_in_mb,accounts:gmail_used_quota_in_mb,accounts:gplus_photos_used_quota_in_mb,accounts:is_archived,accounts:is_disabled,accounts:is_suspended,accounts:last_login_time,accounts:last_sso_time

# get the list of emails from that report, and sort them.

awk -F "," '{print $1}' ./gam_report_output.csv | tail -n +2 | sort > gam_report_output_emails_sorted.csv

#get and sort the list of all users from gam print users

~/bin/gamadv-xtd3/gam redirect csv ./gam_print_users_output.csv print users

tail -n +2 ./gam_print_users_output.csv | sort > gam_print_users_output_sorted.csv

#use the linux comm command to get the list of just the users that were skipped in gam report users.

echo "primaryEmail" > gam_not_in_report.csv

comm -2 -3 gam_print_users_output_sorted.csv gam_report_output_emails_sorted.csv >> gam_not_in_report.csv

#use gam’s CSV input function to iterate through that list of “missed" users one by one.

~/bin/gamadv-xtd3/gam config process_wait_limit 360 auto_batch_min 1 redirect csv ./gam_not_in_report_users_storage.csv multiprocess csv gam_not_in_report.csv gam report users user ~primaryEmail fulldatarequired all allverifyuser REDA...@example.edu fields accounts:used_quota_in_mb,accounts:drive_used_quota_in_mb,accounts:gmail_used_quota_in_mb,accounts:gplus_photos_used_quota_in_mb,accounts:is_archived,accounts:is_disabled,accounts:is_suspended,accounts:last_login_time,accounts:last_sso_time

It turns out that on our sandbox instance (which has only 58 users) there was one user that was skipped. On our prod instance, 2,298 users are being skipped.

The results of doing this were interesting. For a bunch of the 2,298 users, I got "WARNING: Start date can not be earlier than 2023-04-03”; but for a lot of them it was able to report on usage. The strange bit is that for the “gam report users” output, the date was 2024-06-23 for all of the records. For the 2,298 missing ones, the date ranged from 2024-06-19 through 2024-06-23. 

I’m not quite sure why I’m getting the WARNING: Start date can not be earlier than 2023-04-03 errors, though. It seems odd to me that accounts wouldn’t have the data I’m requesting in the report?

This is all still quite confusing (and concerning)….

Thanks,

Ian

[Ross Scroggs]

Ian,

When reporting on individual users, don't use all allverifyuser REDA...@example.edu.

Send me a Meet/Zoom invitation.

Ross

----

Ross Scroggs

Ross.S...@gmail.com

--
You received this message because you are subscribed to the Google Groups "GAM for Google Workspace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/A8B80FB0-AC18-4CC7-B6EF-79C81A07CC3B%40berkeley.edu.

[Ian Crew]

Hi all:

Reporting back, Ross and I spent a while looking at this.

It does seem like there’s a real issue with the Reports API: Users Usage Report (/admin/reports/v1/usage, https://developers.google.com/admin-sdk/reports/v1/guides/manage-usage-users) from Google. We’re seeing 3 specific issues:

1) /admin/reports/v1/usage/users/all is not returning all users in the workspace instance. As I mentioned below, it skipped 1/58 users on our sandbox instance and 2,298 of ~382,000 on our production instance.

2) For some of the “skipped” users, it’s possible to get the info for them using the one-by-one technique I laid out in my message from about an hour ago, which calls admin/reports/v1/usage/users/jo...@example.com/ for each user. But what’s interesting is that the report date for those users is often several days earlier than for the users returned by  /admin/reports/v1/usage/users/all.

3) Some of the “skipped” users (even those created and first logged in just 10-12 days ago) aren’t showing up at all when looked up via the admin/reports/v1/usage/users/jo...@example.com/ api.

I’m going to try to raise this with Google, hopefully I can get them past the “we don’t support GAM” bit, and Ross is working on a way to record at least a sort of “we didn’t find any data for that user” in the output for case #3.

Cheers,

Ian

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/2D82EF38-01EA-4508-B9BE-C7D87EE95435%40gmail.com.

[✉ Kevin Melillo]

Good afternoon,

Not sure if this will help or hinder your efforts.  Have you run the report multiple times to see if the users skipped are the same each time?  Or do they change?

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/0217B23F-18BD-4E33-BF9B-27F02867A70F%40berkeley.edu.

[Ian Crew]

They do seem to be at least reasonably consistent, yes. But we haven’t been able to figure out any particular commonality either for why they’re skipped or for why they’re not available via the reports api at all. gam info user works properly for all of them, as far as I can tell.

Thanks!

Ian

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CAKM%3DboaZJFXxu8H5_uEjt7riiN8EA6Q39CvnwwSn4_p5AkA-9g%40mail.gmail.com.

[Kim Nilsson]

Very interesting.

I checked the numbers in a few domains with the following inconsistent but still similar results.