Test data for Google fit

[Ralph van Etten]

Hi,

I am testing the shimmer with Google fit and I was wondering if there is a way to easily and automatically get some test data in my Google fit account?

I would like to have a couple of months of data without waiting a few months or manually adding it :)

Thanks,

Ralph.

[Emerson Farrugia]

Hi Ralph,

You might be in luck with Google Fit. Many APIs don't have write endpoints, but Fit does. You should be able to create a script to do what you need as, as follows.

1. Pick a data type that's supported by both the Google Fit shim and the sample data generator (Appendix A).
2. Use the data generator to create a few months of data, in Open mHealth format.
3. Map that generic data into Google Fit's format. To do that, find the mapper that maps from Google Fit to OmH, and invert the mapping.
4. Call the Google Fit API to add the mapped data points to a data source.

At first glance, I don't see why that wouldn't work. The inverse mapping is probably the most work, but our code should be clean and clear enough to help you pull that off. If you have any questions about it, feel free to ask.

Let us know how it goes!

Emerson

This e-mail message is intended exclusively for the addressee(s). Please inform us immediately if you are not the addressee. 
MedVision BV

Aagje Dekenkade 71
2251 ZV, Voorschoten

www.medvision360.com

--
You received this message because you are subscribed to the Google Groups "omh-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to omh-developer...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ralph van Etten]

Hi,

You might be in luck with Google Fit. Many APIs don't have write endpoints, but Fit does. You should be able to create a script to do what you need as, as follows.

1. Pick a data type that's supported by both the Google Fit shim and the sample data generator (Appendix A).
2. Use the data generator to create a few months of data, in Open mHealth format.
3. Map that generic data into Google Fit's format. To do that, find the mapper that maps from Google Fit to OmH, and invert the mapping.
4. Call the Google Fit API to add the mapped data points to a data source.

At first glance, I don't see why that wouldn't work. The inverse mapping is probably the most work, but our code should be clean and clear enough to help you pull that off. If you have any questions about it, feel free to ask.

Thanks, I'll try it when I have time available.

Ralph.

[Ralph van Etten]

Hi,

Today I had some time to spend on creating something for importing data into Google Fit and I found an issue with the way the Google Fit shim retrieves data.

I managed to import heart rate data into Google Fit using the REST API, this seems to work fine, however the Google Fit shim is unable to retrieve this data because the shim only retrieves data using the data stream ID of the Android service.

According to the documentation, the data stream ID for REST applications is :

type:dataType.name:developer project number:device.manufacturer:device.model:device.uid:dataStreamName 

while for Android applications it is :

type:dataType.name:application.packageName:device.manufacturer:device.model:device.uid:dataStreamName 

(some fields are optional)

For heart rate, the shimmer always uses the derived:com.google.heart_rate.bpm:com.google.android.gms:merge_heart_rate_bpm stream ID (see here) which implies the shimmer will only retrieve Google Fit data created by the com.google.android.gms application. 

I think the shimmer should use the list API first to retrieve a list of data sources (stream IDs) of a certain data type and then fetch the data from all the data sources it found.

But that might lead to other problems if an user has many datasources as the API is limited by default to 5 requests/second/user.

Is it intentional that the Google Fit shimmer is only capable of returning data stored by Android?

Thanks,

Ralph.

[Chris Schaefbauer]

Hi Ralph,

This is an interesting issue you raise. The merged data sources (there is a merged data source for most measures) contain data, for a single type, that is processed and merged "from all data sources, including from 3rd parties" [Satinder Singh, Moderator for the Google Fit Developers community, 2014-10-30]. 

Although the naming implies the data sources are android-based (with the com.google.android.gms part), our experience suggests that is not the case. Furthermore, the Google Fit team has previously suggested that these are the appropriate endpoints to use for getting all data of a certain type across data sources [https://plus.google.com/+DavidSchmidt/posts/TJjRTefojC6]. 

Google has been a bit hand wavy about how data points in the merged endpoints are created, but they suggest they use a set of heuristics and machine learning in merging in order to reduce duplicated data points from different data sources and to ensure the integrity of data points. When we asked them about the merged data sources in September 2015, they told us they are "presently (re)evaluating how to handle these" [personnal correspondence] and that they do not have any documentation for them, unfortunately. We have not heard anything new since then.

I just tested this again to make sure nothing has changed and I am able to create a new data source using the REST API (which had a dataSourceId = raw:com.google.heart_rate.bpm:947638375722), push data to that data source using the REST API, and then retrieve it using Shimmer pointed to the "derived:com.google.heart_rate.bpm:com.google.android.gms:merge_heart_rate_bpm" data source. 

Could you tell us a little bit more about the data source you are using to write the data? Maybe I am missing something on my end or making an incorrect assumption?

The idea of first using the list API is interesting. My concern is that, in my experience, many data sources contain duplicate data, which might be why the merge data sources exist. We have not dug into this before, so it might very well be easy to de-duplicate. But that is another challenge that comes to mind.

Hopefully we can figure out why you are unable to retrieve the data you are creating through the REST API, because this may be a systemic issue that may be leading us to miss data from Google Fit. 

Thanks so much for bringing this to our attention.

Thanks,

Chris Schaefbauer

[Ralph van Etten]

Hi Chris,

On 01/06/2016 11:37 PM, Chris Schaefbauer wrote:

> This is an interesting issue you raise. The merged data sources (there
> is a merged data source for most measures) contain data, for a single
> type, that is processed and merged "from all data sources, including
> from 3rd parties" [Satinder Singh, Moderator for the Google Fit
> Developers community, 2014-10-30].

Ah, great. I wasn't aware such a data source exists. I could not find
any information about them.

> I just tested this again to make sure nothing has changed and I am able
> to create a new data source using the REST API (which had a dataSourceId
> = raw:com.google.heart_rate.bpm:947638375722), push data to that data
> source using the REST API, and then retrieve it using Shimmer pointed to
> the
> "derived:com.google.heart_rate.bpm:com.google.android.gms:merge_heart_rate_bpm"
> data source.

Good. That means I'm doing something wrong when uploading the samples :)

I tried to follow the sample described on
https://developers.google.com/fit/rest/v1/workout
but using the Java library for fitness.

It seems to run without errors, but it just doesn't return any data.
You can find the code here: https://github.com/MEDvision/googlefit

I checked the URLs and data send by the Google fit library and they all
seem to be correct.

What are you using to add data using the REST API? Perhaps I can use it
to figure out what I'm doing wrong.

> My concern is that,
> in my experience, many data sources contain duplicate data, which might
> be why the merge data sources exist.

Yes, I agree the merge data sources are the best solution for this.

Thanks,

Ralph van Etten

--
*MedVision BV*

Aagje Dekenkade 71
2251 ZV, Voorschoten
www.medvision360.com

[Chris Schaefbauer]

Hi Ralph,

I use an http networking library in Python to submit HTTP requests. Specifically I am using requests_oauthlib, which has guidance on handling Google OAuth flows. Once you get an access token (which can be done using the google OAuth playground or using any OAuth library) you can submit the requests using your favorite HTTP library by using the token as a Bearer token in the "Authorization" header. Here are examples of the requests I am making:

To create a new data source:

POST https://www.googleapis.com/fitness/v1/users/me/dataSources/

The request also needs to contain a data payload with the JSON for the new data to be created. Here is an example of what I submitted:

https://gist.github.com/schaefba/b541fc63cf462a499b70

In that example, the 947638375611 value at the end of the "raw:com.google.heart_rate.bpm:947638375611" part would be your application ID which can be found in the Google API console. 

To create new data to the REST api once a datasource has been created:

PATCH https://www.googleapis.com/fitness/v1/users/me/dataSources/raw:com.google.heart_rate.bpm:947638375611/datasets/1429113600000000000-1439113600000000012

You need to make sure you use the PATCH method in this case, instead of a POST.

The request also needs to contain a data payload with the JSON for the new data to be created. Here is an example of what I submitted:

https://gist.github.com/schaefba/e233af2f9a27885ece22

You also need to make sure that you include a {"Content-type": "application/json"} header in your HTTP request. This was one problem I had early on, which prevented me from creating data.

Hopefully that is helpful for you. Please let me know if there is anything that I can clarify or if there is anything else I can do to help. 

Thanks,

Chris Schaefbauer