Google Endpoint - Query LastSyncTime

[scre80]

Hi, 

I am trying to run an extract from GAM for endpoint devices (Console does not give us all the required fields). GAM fails to do a direct extract as we have too many devices, so I was trying to conduct a query based extract based on last sync time. 

Looking at the API info below. 

https://cloud.google.com/identity/docs/reference/rest/v1/devices

https://github.com/taers232c/GAMADV-XTD3/wiki/Cloud-Identity-Devices

2 questions are

1 - Can you run a query for the lastsynctime (Other fields have worked for me such as OS) - Have been trying gam info device query "lastSyncTime:2022-06-28T06:05Z"

2 - Can you run a query based on a time frame, ie from and to dates. Unsure how to do this specifically. 

Thanks All. 

[Jay Lee]

What error do you get when you try to extract all devices? I'd really like to explore that issue before figuring out workarounds...

Jay

--
You received this message because you are subscribed to the Google Groups "GAM for Google Workspace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-apps-man...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/caa00741-4536-41db-aad2-6e58eae0a0efn%40googlegroups.com.

[scre80]

Get a Error: 400 as below, in both GAM and GAM Advanced, both fail with same error around the 160,000 devices. We have over 1.3mil devices in endpoint.

I did figure out the sync query - gam print device query "sync:2022-06-28" > file.csv,  just seeing if I can do a date range now on that. 

Thanks

Scott

[Jay Lee]

Got it. And how many devices do you have in your domain?

Jay

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/78b3574e-5985-4c28-a5fb-eaeab8da4005n%40googlegroups.com.

[Ross Scroggs]

Scott,

See: https://github.com/taers232c/GAMADV-XTD3/wiki/ChromeOS-Devices#cros-query-searchable-fields

Scrool down to Last Sync and Format for date searches.

### Format for date searches

* `YYYY-MM-DD` - A single date

* `YYYY-MM-DD..YYYY-MM-DD` - A date range

* `..YYYY-MM-DD` - All dates on or before a date

* `YYYY-MM-DD..` - All dates on or after a date

In all the Google APIs that list items, the items come in pages; each page gives a reference to the next page;

you proceed until you get a null nextPageToken

pageTokten=None

while True:

  result = callGAPI(cd.chromedevices(), 'list', pageToken=pageToken, ...)

  pageToken = result['nextPageToken']

  if not pageToken:

   break

The problem is that Google returns an invalid nextPageToken value and you get the 400 error.

Do you have an approximation as to how long the command runs until it errors out?

Ross

--

ross.s...@gmail.com

On Jun 28, 2022, at 5:22 AM, scre80 <src...@gmail.com> wrote:

Get a Error: 400 as below, in both GAM and GAM Advanced, both fail with same error around the 160,000 devices. We have over 1.3mil devices in endpoint.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/78b3574e-5985-4c28-a5fb-eaeab8da4005n%40googlegroups.com.
<Screenshot 2022-06-14 162107.jpg>

[scre80]

Hi Ross, 

Thanks for the info, I had just come across that and managed to get it working with the below command. 

gam print device query "sync:2022-06-13..2022-06-28" > output.csv

As for the 400 error, I have not timed it exactly, believe it would be around the 40 to 45min mark. If you need exact timings, can run it again tomorrow and see. 

Thanks

Scott

[Jay Lee]

Pretty sure this 400 error occurs whenever the access token is refreshed, the pageToken is associated to the access token and so when the access token changes it invalidates the current page. You can show this with something like:

gam oauth refresh    # this will get you a brand new refresh token rather than one that might be up to an hour old

gam print devices nodeviceusers

Ross, even without hundreds of thousands of devices it should be reproducible if you set page size for the list API call to 1 and then sleep a few minutes in between pages.

I'll see if we can get any movement on a fix to this issue.

Jay Lee

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/bcd8a53c-312e-49ed-9fa3-237f3d0b1017n%40googlegroups.com.

[Ross Scroggs]

Jay,

pageSize=1, time.sleep(60) after each page

ERROR:  JSON: {'error': {'code': 400, 'message': 'Request contains an invalid argument.', 'status': 'INVALID_ARGUMENT'}}

on 61st device

Ross

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CA%2BVVBp_KXAw1MagMSA631M%2Bm-GJ1mYW4yvaMZGntmNSbabSLOA%40mail.gmail.com.

--

Ross Scroggs

ross.s...@gmail.com

[scre80]

Thanks Ross and Jay, 

Even with a restricted range, it seems to fail, first had a 503 error so tried again, then failed with Request contains an invalid argument. Not sure how long this look, was over an hour. On a shorter range, ie 2 days and 50,000 devices, worked fine. 

Thanks

Scott

[Jay Lee]

FYI, I have a workaround that seems to get around the issue. See:

https://github.com/GAM-team/GAM/issues/1534

the initial workaround is committed though there's still some bugs in the formatting output to be cleaned up.

Jay

Jay Lee

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/0bf4379d-6cff-4d74-ad81-71e644e2ec99n%40googlegroups.com.

[Scott Crew]

Thanks Jay, 

Testing it now. 

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CA%2BVVBp_%2BDRsPjqzUNKLa8JFxTS0ZdEcZa8TUNpJh4vkmSudp%2BA%40mail.gmail.com.

[Jay Lee]

Great, the big question is if it can get past 1 hour and keep pulling devices. GAM will print out some information when it encounters the issue but should continue. Note that until my latest commit today you'll likely see crashes or bad output but you can at least confirm you can get past an hour.

Also, it's going to take a long time and a lot of system memory to download 1.3 million devices and their user information. After GAM pulls all devices it needs to go back and download the device users. The device user list API call is limited to 20 results per API call and page so it will likely take significantly longer than pulling the devices.

Jay

Jay Lee

To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-manager/CAKy4HPWEjf3DhWqxHUXds23LS7qfNJzo5dVbNEzn9Mgur6ySTA%40mail.gmail.com.