Open API for R, Matlab/Octave etc

[Mark Liversedge]

Hi,

As you know for 3.3 we've extended datafilters into formulas for user defined metrics and ride data series. No doubt we will continue to extend them functionally for simple use-cases; where data is manipulated using simple constructs.

But the approach is pretty limiting; the available functions need to anticipate the kind of analysis the user wants to perform. And whilst it is wholly possible to extend the functions to include lots of advance maths and statistical functions and algorithms, you would essentially end up trying to implement R or matlab inside a fitness application !

So whilst we will continue to develop the formula approach, we have also now developed an open API that can be called from R, matlab and similar.

REST-like web services

If you start goldencheetah v3.3 with the --server command line option it will run as an http server offering a number of services, by default on port 12021 (this can be changed in the httpserver.ini file that is created in the athlete directory). It is stateless, will only read data from the athlete directory and can be running whilst you have GC running as normal.

These web-services provide functions for listing athletes, listing activities with their metrics, fetching activity data in a variety of formats. You can also fetch zone config history and mean maximal data for a ride or date range.

Above I have started the GoldenCheetah API server and connected from R studio, retrieved a ride and plotted it.

By way of example, below are the meanmax power bests and IF/TSS metrics for all rides for this athlete, using similar approaches:

You will see that from R (for example) you can connect to the server and retrieve data using the RCurl package in a couple of lines e.g;

library(RCurl)
bests <- read.csv(text=getURL("http://localhost:12021/athlete/bests?series=watts&since=2014/01/01&before=2014/12/31"))
plot(secs, watts)

Endpoint Descriptions

The endpoints and parameters are as follows;

1. List Athletes

http://localhost:12021/

Will list all the athletes and their basic info (weight, height, date of birth, sex et al)

2. List Activities and Metrics

http://localhost:12021/athletename

Will list all rides and all metrics for the named athlete. The following parameters can be used to limit the output;

since=yyyy/mm/dd 

before=yyyy/mm/dd

metrics=NP,IF,TSS,AveragePower ... etc

3. Fetch an Activity

http://localhost:12021/athletename/activity/filename

Will retreive the activity (by default in json format) the format can be specified in a parameter;

format=xxx where xxx is one of csv, tcx, json, pwx

4. Fetch Mean Maximal Data

http://localhost:12021/athlete/meanmax/activity 

Will fetch the meanmax for a specific activity, by default it will retreive power, but can specify using a parameter;

series=xxx where xxx is one of watts,hr,cad,speed,nm,vam.xPower,NP

5. Fetch Mean Maximal Bests

http://localhost:12021/athlete/meanmax/bests

Will fetch the very best MM data for a date range, by default it will fetch for all dates, but you can specify using a parameter;

since=yyyy/mm/dd

before=yyyy/mm/dd

series=xxx where xxx is one of watts,hr,cad,speed,nm,vam.xPower,NP

6. Fetch Configured Zone parameters

http://localhost:12021/athlete/zones

Will fetch the zones config, by default for power (CP,W', Pmax over time) but you can specific using a parameter;

for=xxx where xxx is one of power, hr, pace, swimpace

We will no doubt add more endpoints and some samples/tutorials on working with them from matlab and R.

Most interestingly, researchers are increasingly publishing their algorithms and code using R and Matlab ... making it easy for us to use it ahead of any special development. A good example would be the core temperature algorithm that was implemented in GC v3.2.

Mark

[Ruud Goorden]

Looks great. Can't wait to see some of the tutorials

[Mark Liversedge]

I just discovered a program called 'Orange' here: http://orange.biolab.si/

Its an amazing tool for data analytics, mining etc. Since it uses python as a scripting language its quite easy to integrate with the GC API.

It has a number of advanced stats modules for classification, clustering, modelling etc. Has a graphical UI and is basically a fabulous, free and mature tool for analytics. Its a great playground for working with your data and some of the things I've been able to do with it are likely to arrive into GC to make it easy for others to do the same thing.

Here are some screenshots, recommend playing with it when v3.3 is released and will definitely make a tutorial for the API and include using Orange alongside GC (as well as R).

Cheers,

Mark

[mike veloclinic]

Mark,

I can get R to read the list of athletes from GC. For athletes with only a couple activities it will also read the list of activities and will read an individual activity. However, if I have an athlete with more than just a few activities I get:

> activities <- read.csv(text=getURL("http://localhost:12021/mike p"))

> activities

[1] X413.Entity.too.large

<0 rows> (or 0-length row.names)

Also, 

I can't seem to get bests to work even on the athletes with just a few activities where everything else works. I've tried a few different variations but get something like this each time:

> library(RCurl)

> bests <- read.csv(text=getURL("http://localhost:12021/doug/bests?series=watts&since=2014/01/01&before=2015/12/31"))

> attach(bests)

The following object is masked from bests (position 3):

    malformed.url

The following object is masked from bests (position 4):

    malformed.url

The following object is masked from bests (position 5):

    malformed.url

The following object is masked from bests (position 6):

    malformed.url

> plot(secs, watts)

Error in plot(secs, watts) : object 'secs' not found

> bests

[1] malformed.url

<0 rows> (or 0-length row.names)

Any ideas ?

m

[Karl Billeter]

On Sun, Nov 15, 2015 at 05:18:38PM -0800, mike veloclinic wrote:
> Mark,
>
> I can get R to read the list of athletes from GC. For athletes with only a
> couple activities it will also read the list of activities and will read an
> individual activity. However, if I have an athlete with more than just a
> few activities I get:
>
> > activities <- read.csv(text=getURL("http://localhost:12021/mike p"))
> > activities
> [1] X413.Entity.too.large
> <0 rows> (or 0-length row.names)

Try http://localhost:12021/mike%20p


Karl

[Karl Billeter]

On Sun, Nov 15, 2015 at 05:18:38PM -0800, mike veloclinic wrote:

> Also,
> I can't seem to get bests to work even on the athletes with just a few
> activities where everything else works. I've tried a few different
> variations but get something like this each time:
>
> > library(RCurl)
> > bests <-
> read.csv(text=getURL("http://localhost:12021/doug/bests?series=watts&since=2014/01/01&before=2015/12/31"))
> > attach(bests)

I think this needs to be .../doug/meanmax/bests?series....


Karl

[Mark Liversedge]

Yes. The URL should be encoded there are functions to do this. Perhaps this: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/URLencode.html

I haven't tried it myself. 

[Mark Liversedge]

The example in the opening para is wrong, the syntax is correct in point 5.

Mark

 

[mike veloclinic]

Thanks all, I have it working now !

[Nathan Townsend]

Hi Guys,

I've got the API working as it creates an object of bests in the environment window, but having difficulty getting it to plot meanmax.  Any suggestions?

> library(RCurl)

> bests <- read.csv(text=getURL("http://localhost:12021/DAGO/meanmax/bests/"))

Error in read.table(file = file, header = header, sep = sep, quote = quote,  : 

  no lines available in input

library(RCurl)

> bests <- read.csv(text=getURL("http://localhost:12021/DAGO/meanmax/bests?series=watts/"))

> attach(bests)

The following object is masked from bests (pos = 3):

    unknown.series.requested.

[Mark Liversedge]

On Thursday, 19 November 2015 08:45:21 UTC, Nathan Townsend wrote:

I've got the API working as it creates an object of bests in the environment window, but having difficulty getting it to plot meanmax.  Any suggestions?

Check your URLs via your browser first, then use with R.

It will display the raw data returned so you can see if you are getting what you expected, or if GC is running etc.

 

[Nathan Townsend]

No problems when I run list athletes (see attached), but if I enter any of those athlete names then I get nothing for meanmax/bests in the browser

[Mark Liversedge]

On Thursday, 19 November 2015 10:26:36 UTC, Nathan Townsend wrote:

No problems when I run list athletes (see attached), but if I enter any of those athlete names then I get nothing for meanmax/bests in the browser

 

What happens when you look at meanmax bests in GC for those athletes?

[Nathan Townsend]

See attached. This is what I'm expecting I guess!?!

On another note, here we can see a perfect example of why taking many data points to model P(t) is totally bogus

 

[Mark Liversedge]

And the URL and browser response?

[Nathan Townsend]

Opens a tab called "Redirecting", but there is nothing there. It's just a blank white page.

[Nathan Townsend]

On Thursday, 19 November 2015 17:14:05 UTC+3, Nathan Townsend wrote:

Opens a tab called "Redirecting", but there is nothing there. It's just a blank white page.

 

and in R, opening the"bests" object shows no data either. It says "unknown.series.requested" then "No data available in table"

[Nathan Townsend]

On Thursday, 19 November 2015 17:14:05 UTC+3, Nathan Townsend wrote:

Opens a tab called "Redirecting", but there is nothing there. It's just a blank white page.

 

and in R, opening the"bests" object shows no data either. It says "unknown.series.requested" then "No data available in table"

I have the " list athletes" working fine.  See attached.  Let me try some of the other commands.

[Mark Liversedge]

On Thursday, 19 November 2015 14:16:34 UTC, Nathan Townsend wrote:

On Thursday, 19 November 2015 17:14:05 UTC+3, Nathan Townsend wrote:

Opens a tab called "Redirecting", but there is nothing there. It's just a blank white page.

Sounds like something interesting is happening on your computer ;)

you need to open:

http://localhost:12021/DAGO/meanmax/bests

If that doesn't work then your PC config / browser / proxy settings are getting in the way or GC is not running. Will leave with you for now to try and diagnose

Mark 

[Ale Martinez]

I don't know if this was already posted, the last version of R package "cycleRtools: Tools for Cycling Data Analysis" featuring GoldenCheetah v3.3 integration is available on CRAN: https://cran.r-project.org/web/packages/cycleRtools/index.html

[Mike Patton]

Hi Mark. 

Any chance that the various CP model parameters could be made available in the same way that Bests are available? It would be really helpful to be able to specify the time ranges and model used in the same way they can be specified in the dialog box within GC. Am I better to raise a feature request than post here?

Mike

[Mark Liversedge]

On Wednesday, 13 January 2016 02:12:46 UTC, Mike Patton wrote:

Any chance that the various CP model parameters could be made available in the same way that Bests are available? It would be really helpful to be able to specify the time ranges and model used in the same way they can be specified in the dialog box within GC. Am I better to raise a feature request than post here?

The model estimates are generated and not stored, the CP/Pmax/W'/zone config are available.

What are you looking to achieve?

We /could/ and /want/ to update the modelling side to store estimates, that way we don't have a startup delay.

We /could/ and /should/ enable you to use the models via the API (regardless of athlete). 

Mark

[Mike Patton]

The model estimates are generated and not stored, the CP/Pmax/W'/zone config are available.

What are you looking to achieve?

I wrote an R script that allows me to import MMP data in order to quickly produce individualized training zones/levels/etc. for a pretty large number of athletes. The default % options are great in general, but being able to anchor the upper zones to actual or modeled MMP values is useful in my opinion. I currently export the chart data from the CP chart and import that, but if the modeled data were available via the API, then I could cut out a step. 

 

We /could/ and /want/ to update the modelling side to store estimates, that way we don't have a startup delay.

Based on the feedback from the coaches I work with, that would be a nice usability enhancement, especially with athletes having many years of data stored in GC.

 

We /could/ and /should/ enable you to use the models via the API (regardless of athlete). 

As I said, that data is available with a simple export process, so it's nothing urgent, especially since I'm doing this sort of thing infrequently. With that said, I can't access the un-rounded (if that's a word) value of W' as far as I can tell, in order to compute the curve myself since it's currently rounded to the nearest 100J/0.1kJ. Probably doesn't matter that much, but having the full modeled values via the API would still be useful.

Cheers.

Mike

[Mark Liversedge]

Thanks Mike,

I will look at persisting the model estimates and integrating them into the RideCache.

There are some performance gains to be too, so it is something to add for v4.

Additionally to your use case, I want to perform some independent model validation once we have data, and this will support that.

Cheers

Mark 

[Ruud Goorden]

Been trying to use R, the but for some reason I can't connect to the localhost. The API is enabled in GC. Also typing in the URL on internet explorer won't give a result. Have sought quite extensively what reason could be but no luck untill now. Does anybody have some good tips (maybe somekind of small checklist / of most common basic setting issues (browser or proxy or...)?) so I can go through these? Would be really great

[Mark Liversedge]

On Tuesday, 23 February 2016 13:42:04 UTC, Ruud Goorden wrote:

Been trying to use R, the but for some reason I can't connect to the localhost. The API is enabled in GC. Also typing in the URL on internet explorer won't give a result. Have sought quite extensively what reason could be but no luck untill now. Does anybody have some good tips (maybe somekind of small checklist / of most common basic setting issues (browser or proxy or...)?) so I can go through these? Would be really great

Really simple R script:

#install.package(RCurl)

library(RCurl)

# takes a while as processes entire rideDB backing store, obvs use an appropriate URL !

metrics <- read.csv(text=getURL("http://localhost:12021/Mark%20Liversedge/?metrics=IF,TSS"))

# attach to process

attach(metrics)

#simple scatter plot

plot(TSS,IF)

r <- lm(IF~TSS)

abline(r, col="red")

Hope this helps.

Mark

 

[Ruud Goorden]

TNX Mark. I ll try. I already see I made an error with the URL ;)

By the way this one is nice as well:

https://cran.r-project.org/web/packages/trackeR/vignettes/trackeR.pdf

[Mark Liversedge]

Yes, Jordan added support to GoldenCheetah to it !

Mark 

[Ruud Goorden]

O. Thats cycleRtools. That is another package than this one I am referring to. I thought this one would be an nice addition as well

[Mark Liversedge]

On Tuesday, 23 February 2016 17:50:33 UTC, Ruud Goorden wrote:

O. Thats cycleRtools. That is another package than this one I am referring to. I thought this one would be an nice addition as well

OOPS. My bad. Muddled them up.

I promised to update the GC file readers for them, but haven't really made a start.

Anyone fancy helping?

Its here: https://github.com/GoldenCheetah/FileIO

Mark 

[Ruud Goorden]

Unfortunately keep getting the error:

Error in function (type, msg, asError = TRUE)  : 
  Failed to connect to localhost port 12021: Connection refused

Also turned off firewall which did not help. Any other options? I mean also when I enter http://localhost:12021 in the browser it won't find it. Very Strange and frustrating :(

Op dinsdag 23 februari 2016 16:39:40 UTC+1 schreef Mark Liversedge:

[Mark Liversedge]

On Wednesday, 24 February 2016 09:06:06 UTC, Ruud Goorden wrote:

Unfortunately keep getting the error:

Error in function (type, msg, asError = TRUE)  : 
  Failed to connect to localhost port 12021: Connection refused

Also turned off firewall which did not help. Any other options? I mean also when I enter http://localhost:12021 in the browser it won't find it. Very Strange and frustrating :(

It will either be because you have not enabled them in preferences:

Or you disabled network access for GoldenCheetah (usually some form of popup comes up moaning about network access). 

Mark

[Ruud Goorden]

Unfortunately this also did not solve it. I didn't disable network access either. Arggh.

Op woensdag 24 februari 2016 11:28:11 UTC+1 schreef Mark Liversedge: