[Golden-Cheetah-Users] Removing Power Spikes

[watts so small]

Hello All,
I have searched and cannot seem to find a simple answer to deleting or
changing spikes in power over the course of a given ride. Yesterday,
on a 7 hour ride. i had several 2k power readings that were positively
bogus. I want to edit them out so that they don't affect my power
profile. Is there a way?
Thanks for any help

--
_______________________________________________
Golden-Cheetah-Users mailing list
golden-che...@googlegroups.com
http://groups.google.com/group/golden-cheetah-users?hl=en

[Jamie Kimberley]

I've been having a lot of these too. I generally save as a .gc
format (yay human readable) and then just search and replace. Then
you can delete the associated .cpi file so that it will recalculate
your cpi for the ride on relaunch. Note that it may take a few
iterations to get all of the large spikes as it is tough to find all
th epower spikes in this way. you could also export to .csv and
sort by watts to find the spikes and then edit the .gc file. I
suppose you could use the find best intervals feature too with the
duration equal to your recording interval.

I've noticed that my spikes are generally when I start fro ma
standstill and I get very high cadence reading, perhaps I'm flicking
back and forth over a reed switch when I'm trackstanding at a light.
I'm guessing that we could attempt to filter data points based on
some combination of cadence and power. Note that i'm using a
Quarq. I don't think the hub based PM's are as prone to these
errors.

Ideally this would be a user interactive menu item akin to "find
best intervals" so the user could determine in the spikes were bogus
or not.

If we can come up with a framework, lets put this up as a feature
request in the tracker.

Jamie

"You don't use science to show that you're right, you use science to
become right."--xkcd
__________________
Jamie Kimberley
Postdoctoral Fellow
Department of Mechanical Engineering
The Johns Hopkins University
Office: 410.516.5162
Mobile: 217.621.8272
Fax: 410.516.4316
E-Mail:jamie.k...@jhu.edu

[Robert Carlsen]

i believe the only way would be to export as CSV and manually edit the file.
if you leave the CSV file in the rider library folder, it should get picked up on the next launch of GC.

anyone else?

[Mark Liversedge]

Spooky!!!

I have coded up a feature to add user-configurable post processing
routines to ride data. So far I have written two modules; firstly. to
remove drops in recording and secondly to create CPI files. I've
literally just started a third to 'remove data spikes'.

But I'm struggling to define a data spike, it's a form of outlier
detection, but I think it needs to span all ridefiles, not just the
current one. I think for a first pass I should keep it simple and just
allow the user to define maximum allowed values and smooth/interpolate
when they are above a user defined limit.

I'd prefer to do it algorithmically tho.

Will post a patch soon.
Mark

[Robert Chung]

On May 24, 8:24 am, Mark Liversedge <liverse...@gmail.com> wrote:

> But I'm struggling to define a data spike, it's a form of outlier
> detection, but I think it needs to span all ridefiles, not just the
> current one. I think for a first pass I should keep it simple and just
> allow the user to define maximum allowed values and smooth/interpolate
> when they are above a user defined limit.
>
> I'd prefer to do it algorithmically tho.

There are a couple of different approaches. One is univariate, and
typically one would find a suitable data transform, identify the
outlier(s) in that space, and then transform back. That's simple but
it doesn't always work well. The second way is to do it bivariately
(or multivariately). For example, are the spikes only in power, or are
they also in speed? If only in power then you're going to see a
mismatch in the speed-power relationship: for example, you might see a
sudden jump in VE. Of course, there are other measures than VE -- I
was just pointing out that one since some of us are already familiar
with it.

[Mark Liversedge]

My problem at the mo is finding a suitable data transform!

Any suggestions?

[Robert Chung]

On May 24, 11:41 am, Mark Liversedge <liverse...@gmail.com> wrote:
> My problem at the mo is finding a suitable data transform!
>
> Any suggestions?

You mean for the univariate case?

One possible transform is the percentage change relative to a short
span smooth. Most power spikes I've seen only last one or two
observations so rather than set an absolute power threshold, set a
threshold compared to a running smooth of maybe 5 seconds that
suddenly jumps up and then immediately returns to the level of the
previous smooth. Basically, rather than look for oddball values by
their level, you're looking at values that spike and return.

[Mark Liversedge]

I have found with my own files that power spikes tend to be
immediately preceeded by zero values and possibly caused by the reed
switch in my SRMs being 'double' triggered after freewheeling or from
standing starts. HR samples tend to jump at any point (usually caused
by overhead power cables interfering).

I think the smoothing algorithm would work well for HR, but less so
for the Power example?

I'll code it up anyway and see what we end up with.

[Robert Chung]

On May 24, 1:31 pm, Mark Liversedge <liverse...@gmail.com> wrote:
> I have found with my own files that power spikes tend to be
> immediately preceeded by zero values

How high do they spike and how long do they last?

[Mark Liversedge]

On May 24, 9:49 pm, Robert Chung <rech...@gmail.com> wrote:

> How high do they spike and how long do they last?

1500w+ for 1 or 2 seconds. I haven't totally discounted that they
could actually be real :-)

[Jamie Kimberley]

what are your cadence values like? All of my spikes (including a
3599 W peak) seem to have very high cadences right after a stop
(like you said about tripping a reed switch is likely). So you
might be able to filter on what a the biggest change in cadence
is over a set period.

I can look at my spikes in a little more detail if you like. Using
your 2-D plot might give some insight :).

Jamie

"You don't use science to show that you're right, you use science to
become right."--xkcd
__________________
Jamie Kimberley
Postdoctoral Fellow
Department of Mechanical Engineering
The Johns Hopkins University
Office: 410.516.5162
Mobile: 217.621.8272
Fax: 410.516.4316
E-Mail:jamie.k...@jhu.edu

[Zach Rogers]

I don't recall my spikes having a high cadence, speed, or otherwise associated with them, and best I can recall from memory, they never occur at any particular event. FWIW, this is all with a wired powertap & I quickly "fix" them in WKO+ by scrolling to the point (usually just one, maybe two samples) and adjust it to something more typical based upon the neighboring values. It's all quite easy, but it's easy to imagine something more automated. The process I have is that I quickly notice on the plot that there is a spike (or two), roughly locate where in time it occurred, switch to the numerical/spreadsheet type view & scroll to the time period of interest then correct the spike.

Next time I have an occurrence, I'll be sure to make the raw file available.

-Zach

[Jamie Kimberley]

Yes, with my wired PT I would get spurious spikes.

With my Quarq (and other people with srm) It seems to be related to
cadence errors tied to starting with hard accelerations froma stand
still. i.e. large torque values coupled with an artificially
high cadence.

Ideally we'd be able to handle both types of spikes in the data.

Jamie

"You don't use science to show that you're right, you use science to
become right."--xkcd
__________________
Jamie Kimberley
Postdoctoral Fellow
Department of Mechanical Engineering
The Johns Hopkins University
Office: 410.516.5162
Mobile: 217.621.8272
Fax: 410.516.4316
E-Mail:jamie.k...@jhu.edu

[Fredrik Lingvall]

On Tue, May 25, 2010 at 2:52 AM, Jamie Kimberley <jamie.k...@jhu.edu> wrote:

Yes, with my wired PT I would get spurious spikes.

With my Quarq (and other people with srm) It seems to be related to cadence errors tied to starting with hard accelerations froma stand still.  i.e. large torque values coupled with an artificially high cadence.

Ideally we'd be able to handle both types of spikes in the data.

Jamie

What does these spikes look like? Can they clearly be distinguished as outliers or can one have "intermediate spikes" as well? And, I guess that there can be false drops in power data as well.

/Fredrik

[Jamie Kimberley]

All, I did a little digging through my data. For my quarq paired to
a garmin 500 I'm generally seeing spikes in power corresponding to
spikes in cadence (makes some sense as power is proportional to
cadence).

While I had though that most of these were generated when starting
from a standstill when trackstanding I actually found quite a few
spikes in the middle of the two crits i did this past weekend. I've
attached a text file with the data fields and some surrounding
points in GC format.

Also of note is that there are some samples that have floating point
representation of cadence and HR which are normally integers. Not
sure what is going on there, but might be worth looking at.

the general trend is that there is a fairly large jump in cadence
say from 90-150 RPM when the spikes occur. they are certainly
during hard accelerations, and might possibly be tied to shifting
when sprinting out of a corner (have the GPS data so I can probably
check). Regardless of the cause We now have a little bit of an idea
of what the spikes look like (at least for my data), how to
determine what values are reasonable data as opposed to false
spikes" is a bit of a mystery to me.

Thanks,
Jamie

[Justin Knotzke]

   Using Garmin's software, do you see the same thing ?

   J

[Andy Coggan]

On May 24, 9:56 am, Jamie Kimberley <jamie.kimber...@jhu.edu> wrote:

> I've noticed that my spikes are generally when I start fro ma
> standstill and I get very high cadence reading, perhaps I'm flicking
> back and forth over a reed switch when I'm trackstanding at a light.

> Note that i'm using a Quarq.

OT, but I've noticed this as well, i.e., the Quarq seems prone to
artifact when you re-start pedaling after a pause.

Andy Coggan

[Justin Knotzke]

On Tue, May 25, 2010 at 10:39, Andy Coggan <aco...@earthlink.net> wrote:

OT, but I've noticed this as well, i.e., the Quarq seems prone to
artifact when you re-start pedaling after a pause.

   Having sampled over six months of pure ANT+ data using Golden Embed, I'm very sure the power/cadence spikes have nothing to do with Quarq Cinqo. 

   GE parses every single ANT+ message. Not once have I seen a Cinqo (using Pod v1 and Pod v2)  spit out "artifacts". 

   Unless you have access to the actual ANT+ messages in it's raw format, you can never say where the error actually is.

    J

[Andy Coggan]

On May 25, 9:48 am, Justin Knotzke <jknot...@shampoo.ca> wrote:

> On Tue, May 25, 2010 at 10:39, Andy Coggan <acog...@earthlink.net> wrote:
>
> > OT, but I've noticed this as well, i.e., the Quarq seems prone to
> > artifact when you re-start pedaling after a pause.
>
>    Having sampled over six months of pure ANT+ data using Golden Embed, I'm
> very sure the power/cadence spikes have nothing to do with Quarq Cinqo.
>
>    GE parses every single ANT+ message. Not once have I seen a Cinqo (using
> Pod v1 and Pod v2)  spit out "artifacts".
>
>    Unless you have access to the actual ANT+ messages in it's raw format,

> you can never say where the error actually is.\

Down-sampling of the ANT+ data stream to 1 s recording might explain
seemingly "spontaneous" spikes, but these consistently occur when
cadence is very low, i.e., where even 1 s sampling exceeds the Nyquist
frequency.

Another reason to question the raw data is that even the *recorded*
samples are physically impossible for me to generate. IOW, I'm not
just comparing each spike to what came immediately before and after
it.

Andy Coggan

[Justin Knotzke]

On Tue, May 25, 2010 at 10:59, Andy Coggan <aco...@earthlink.net> wrote:

Down-sampling of the ANT+ data stream to 1 s recording might explain
seemingly "spontaneous" spikes, but these consistently occur when
cadence is very low, i.e., where even 1 s sampling exceeds the Nyquist
frequency.

   And what is computing this 1 second data ? 

   Again, unless you have the raw ANT+ data, the best you can do, is speculate. 

Another reason to question the raw data is that even the *recorded*
samples are physically impossible for me to generate. IOW, I'm not
just comparing each spike to what came immediately before and after
it.

   Recorded eh ? You mean calculated and recorded. 

   Both Cinqo and PT do not actually send watts or cadence data. They send data by which the slave must calculate in order to record or display watts/cadence. 

   Show me the real raw ANT+ data and I will tell you exactly what the device sent out and what the real watts/cadence is. 

   J

 

[Robert Chung]

On May 25, 7:10 am, Jamie Kimberley <jamie.kimber...@jhu.edu> wrote:
> All, I did a little digging through my data.

For these examples it looks like the cadence residual from a 5-point
running median could be a place to start.

[Justin Knotzke]

   Jamie just jogged my memory in a private email.

   Paul, who is the mastermind of most of the core GE stuff wrote an application called nant:

   git clone git://e2c3.com/nant.git for public read-only

   It will run under Linux and Mac. With an ANT+ USB stick, you should be able to see the ANT+ messages parsed and calculated as you pedal. Then, compare this with what you are seeing from your Garmin and you might be able to determine where the problem lies: with your PM or your head unit. 

   The real proof is there. 

   J

[Fredrik Lingvall]

By looking at the power data in Jamie's file (spikes.txt) I do not think it's meaningful to do any interpolation on power data (or cadence data)  - it varies too much from one second to another even for normal data (for example, from 66 W to  823 W in just one second). Interpolation (or prediction) is only meaningful if one knows that the parameter varies slowly.

What one can do is to detect the outliers (which is not trivial here I think) and then just ignore them in the calculation.

/Fredrik

On Tue, May 25, 2010 at 4:10 PM, Jamie Kimberley <jamie.k...@jhu.edu> wrote:

All, I did a little digging through my data.  For my quarq paired to a garmin 500 I'm generally seeing spikes in power corresponding to spikes in cadence (makes some sense as power is proportional to cadence).

While I had though that most of these were generated when starting from a standstill when trackstanding I actually found quite a few spikes in the middle of the two crits i did this past weekend.  I've attached a text file with the data fields and some surrounding points in GC format.

Also of note is that there are some samples that have floating point representation of cadence and HR which are normally integers.  Not sure what is going on there, but might be worth looking at.

the general trend is that there is a fairly large jump in cadence say from 90-150 RPM when the spikes occur.  they are certainly during hard accelerations, and might possibly be tied to shifting when sprinting out of a corner (have the GPS data so I can probably check).  Regardless of the cause We now have a little bit of an idea of what the spikes look like (at least for my data), how to determine what values are reasonable data as opposed to false spikes" is a bit of a mystery to me.

Thanks,
Jamie

[Robert Chung]

On May 25, 11:40 am, Fredrik Lingvall <f...@ifi.uio.no> wrote:

> What one can do is to detect the outliers (which is not trivial here I
> think)

Hmmm. It may not be *that* difficult. Jamie's data appear to include
altitude. The VE profile should show a big discontinuity at the power
spike(s).

[Mark Liversedge]

I've coded up a univariate Outlier algorithm that compares a point value
to a moving average and provides a ranked list of outlier points. I
think its good for identifying 'candidate' time-series anomalies but not
good for saying with any certainty that the candidate IS an outlier.

I think the next step is to work out multivariate algorithms to confirm
that a point IS an outlier. This might be by comparing other data-series
like cadence or VE but also we might want to think about setting some
absolutes related to how variant data can be i.e. when we say power data
is stochastic -- exactly how stochastic is it? I mean, if you look at
30s windows of data it isn't THAT variant.

Anyway, by way of an example, I have added this rather noddy outlier
algorithm to the LTM chart to highlight rides that (for the metric in
question) are outliers when compared to the rest of the rides for the
period selected. Attached is a plot of my average power last season with
the top 25 outliers highlighted with a big circle.

This made my jaw drop -- I knew I got lazy last year and was impacted by
work commitments in the later half of the season, but this chart really
brings it home!!!!

[Fredrik Lingvall]

On Fri, May 28, 2010 at 1:19 PM, Mark Liversedge <liver...@gmail.com> wrote:

I've coded up a univariate Outlier algorithm that compares a point value to a moving average and provides a ranked list of outlier points. I think its good for identifying 'candidate' time-series anomalies but not good for saying with any certainty that the candidate IS an outlier.

I don't think a moving average tells you much about a sample being an outlier or not. For example, it takes less than a second to stop pedaling, that is, the samples  just before that stop will give you no hint (no information) that you were about to stop pedaling. The same goes for a sudden increase in power (up to a certain level). Of course, if the sample is above your max-power then you now it's an outlier but intermidiate power spikes are tricky to detect.
 

I think the next step is to work out multivariate algorithms to confirm that a point IS an outlier. This might be by comparing other data-series like cadence or VE but also we might want to think about setting some absolutes related to how variant data can be i.e. when we say power data is stochastic -- exactly how stochastic is it? I mean, if you look at 30s windows of data it isn't THAT variant.

I think both cadence and power can change can change very fast (i.e., faster than we sample) but one can at least set some resonable tresholds to catch the most obvious cases.

BTW: Jaynes has a nice discussion about outliers in his book (see chapter 21), E.T. Jaynes, "Probabilty Theory - The Logic of Science"

/Fredrik

[Mark Liversedge]

On 28/05/10 13:23, Fredrik Lingvall wrote:

On Fri, May 28, 2010 at 1:19 PM, Mark Liversedge <liver...@gmail.com> wrote:

I've coded up a univariate Outlier algorithm that compares a point value to a moving average and provides a ranked list of outlier points. I think its good for identifying 'candidate' time-series anomalies but not good for saying with any certainty that the candidate IS an outlier.

I don't think a moving average tells you much about a sample being an outlier or not. For example, it takes less than a second to stop pedaling, that is, the samples  just before that stop will give you no hint (no information) that you were about to stop pedaling. The same goes for a sudden increase in power (up to a certain level). Of course, if the sample is above your max-power then you now it's an outlier but intermidiate power spikes are tricky to detect.

I agree. Whilst working on the rideFile spike issue I realised that I needed to ignore negative power deltas and just look at increases (since like you say, stopping pedalling gives a huge delta).

So, looking across that season I have the average deviation to the 30s moving average and the maximum deviation from the 30s average and the following plot, which is against data that has all spikes 'cleaned'.

I think it gives me a hint at my own characteristics: Generally, over a 30s period I will, on average deviate from the moving average by 60-80w (i.e. my average power increases over 30s), however, the maximum I have managed is 876w over the same period, usually when turning on the gas.

This tells me that any deviation > 1000w, for me, is suspicious.

[Robert Chung]

On May 28, 5:57 am, Mark Liversedge <liverse...@gmail.com> wrote:
> I agree. Whilst working on the rideFile spike issue I realised that I
> needed to ignore negative power deltas and just look at increases (since
> like you say, stopping pedalling gives a huge delta).

Wait a second. You need the negative deltas to tell you how long the
spike lasted. I thought we were talking about transient spikes that
suddenly jump up, last only a record or two, then just as suddenly
return to the neighborhood of the pre-spike level. It sounds to me
like you're trying to identify spikes only by how high they jump, not
by how high they jump *and* how long they last.

[Mark Liversedge]

Hi Robert,

I thought I was just implementing your suggestion! :-)

I'm looking at each sample individually and comparing it to the 30s
moving average. If there are multiple spiked samples then they will
skew the moving average and be less likely to be identified as spikes.

For SRM/Quarq power this seems sensible to me, since those spikes
typically only appear as one, two or in extreme cases three samples
and are typically caused by double triggered reed switches.

For hr it will fail, since they tend to last 'longer' as they are
caused by electrical interference, similarly for speed and in some
cases cadence data. I was going to look at that next.

For PT power spikes I am less certain - not sure what the cause of
those spikes are and how they manifest.

For what its worth a lot of the code is to diagnose/assess the problem
not neccessarily to fix it!

Cheers,
Mark

[Mark Liversedge]

Re-reading my post I ought to add that I am ignoring samples that are
lower than the moving average. i.e when power goes from 1000w to zero
that is not a spike, it is someone stopping pedalling.

So I am deliberately ignoring -ve deviations from the moving average
when looking for outliers in ridefile power data.

[Fernando Maldonado]

If you can identify spikes as indicated can you identify data drops? It is not uncommon for the 705 to drop data for 1 to 3 seconds then resume.

[Mark Liversedge]

On May 28, 4:55 pm, Fernando Maldonado

<fernando.j.maldon...@gmail.com> wrote:
> If you can identify spikes as indicated can you identify data drops? It is
> not uncommon for the 705 to drop data for 1 to 3 seconds then resume.
>

I cannot distinguish between a rider freewheeling and data drops when
just looking at power. I have coded up a post processor for handling
gaps in recording as a separate function. It has a user definable
threshold (in seconds) to decide when to interpolate and when to set
to zero. It doesn't use cadence to see check for power drops but that
could easily be added.

I suspect that once a good set of algorithms are found for identifying
anomalies/outliers that we will be able to re-use them to provide
inputs into all sorts of applications like handling recording gaps,
finding matches burnt, fixing power spikes, highlighting poor/
exceptional performance etc.

Will keep plugging away.

Anyone wanna help me understand Fourier analysis? ;-)

Cheers,
Mark

[Mark Liversedge]

Just a quickie, I've coded up the power spike fixer. To my surprise it
found one in my data (which I thought was clean). See attached
screenshot. As DailyEvil requested, I've also added some metadata fields
for spikes which I charted to look for problems, see other screenshot.

I had to set smoothing to 1s to show on the plot (which needed a change
to the allplot code).

Which begs the question -- why is the smoothing lower limit 2s and not
1s??? I think now we have the stacked view we should let the user see
the data without smoothing?

Cheers,
Mark

[Robert Chung]

On May 29, 3:22 am, Mark Liversedge <liverse...@gmail.com> wrote:
> Just a quickie, I've coded up the power spike fixer. To my surprise it
> found one in my data (which I thought was clean). See attached
> screenshot. As DailyEvil requested, I've also added some metadata fields
> for spikes which I charted to look for problems, see other screenshot.

Very nice.

[Mark Liversedge]

I've coded up a 'spreadsheet-style' ride data editor which uses the new
anomaly detection stuff - they're highlighted with a wiggly-red line.
Will post a patch when I've nailed the bugs.

Any thoughts on useful editor functions?

[Mitsukuni]

Hi Mark,

It's great stuff!

I think about your approach is good.

I needed to fix powerspike, so coded filter for TCX (by python).
Which is not need cause your patch will merge to GC :-)

> Screenshot-Mark Liversedge.png
> 43K表示ダウンロード

[dkrenik]

With my SRM (using PCV) my power spikes are always accompanied by
cadence spikes and last 1 to 2 samples (1 sec). From Marks "Editor"
this doesn't seem to be the case with Garmins, correct?

This is great stuff. Even with getting my SRM upgraded to the dual
reed switch config, I still get a significant number of power/cadence
spikes and edit them out of WKO+.

Dave

On May 28, 6:23 am, Fredrik Lingvall <f...@ifi.uio.no> wrote:
> I think both cadence and power can change can change very fast (i.e., faster
> than we sample) but one can at least set some resonable tresholds to catch
> the most obvious cases.
>

> /Fredrik

[Sean Rhea]

This is *rad*.

It would be cool if the dialog was like the split-ride one. You
select menu Ride->Remove power spikes, or something like that, and up
pops a dialog with all the points during the ride that seem to be
bogus, maybe including the two or three points before or after each
one to make it clear that they are outliers. Then you select the
checkboxes next to the points you want to remove, and click the
"Remove checked outliers" button.

Did that make sense?

Sean

> --
> _______________________________________________
> Golden-Cheetah-Users mailing list
> golden-che...@googlegroups.com
> http://groups.google.com/group/golden-cheetah-users?hl=en

--
"I refuse to accept the idea that the ‘isness’ of man’s present nature
makes him morally incapable of reaching up for the eternal ‘oughtness’
that forever confronts him." --MLK

[Mark Liversedge]

On 07/06/10 03:05, Sean Rhea wrote:
> This is *rad*.
>
> It would be cool if the dialog was like the split-ride one. You
> select menu Ride->Remove power spikes, or something like that, and up
> pops a dialog with all the points during the ride that seem to be
> bogus, maybe including the two or three points before or after each
> one to make it clear that they are outliers. Then you select the
> checkboxes next to the points you want to remove, and click the
> "Remove checked outliers" button.
>
> Did that make sense?
>
> Sean
>

:-) Yes.

I have coded it up slightly differently. There are three parts to the
solution:

1. A new metric that highlights how many spikes/drops there are, so you
can glance at the metrics plot to see if you have any problematic ride
files.
2. A post-processor that can be run automatically or manually to just
fix the problem (with user configurable options like variance thresholds).
3. A ride editor that highlights anomalies - you can work through them
all, fixing as you go.

I really will sort a patch out, but I'm ironing out bugs.

I'd like to get #2 to work in co-operation with #3 but can't quite
decide the best way (technically)...

Cheers,
Mark