Data Fields Bulk Edit

[Tim Cary]

I've added a bunch of custom data fields.

Is there any way to populate them in bulk (like export to csv) and then re-import?  The goal would be avoiding going back into every ride and populating the field, because the data is pretty much the same that I want to populate (bike or workout purpose, for example).

I see how we can export metrics, and the ride data, but neither contain those data fields I have set up.

Thanks.

[Mark Liversedge]

Short version: No. 

Long version:

I thought about allowing you to edit the values in the navigator, but it was just asking for trouble on a feature I'm not sure many would use and we would need to be able to select multiple activities to do it, like an id3 tag editor. This would be extremely complicated.

When we have a mechanism for simple tagging (like the new OSX finder tags) and a mechanism for selecting multiple rides (for compare), then this feature may be easier to implement!

Mark

[Jean Div]

With all the new filter 'logic' that we can now use, is this still not possible..?

[Mark Liversedge]

We could possibly use that to help; e.g. filter ride list and then a dialog to update fields updates them all.

[Mark Liversedge]

I've added a set() function that can be used in the search/filter box.

I would recommend testing the query condition before using it and if unsure take a backup.

The update is not saved until you save the file, but that will happen via prompt on exit.

It is a shotgun, don't shoot your foot off !

Here is the commit message:

commit 48fe226b229195d92c6342b56cb8a5226b4177af

Author: Mark Liversedge

Date:   Fri Sep 4 21:12:14 2015 +0100

    Formula set(field, value, condition)

    

    .. allow users to use a set command to add

       an override or set metadata under certain

       conditions.

    

       e.g. set(Workout_Code, "HT", IF>0.9)

            set(TSS, 65, TSS <= 0 && Route = "Fave Loop")

    

    .. we should probably add an unset() command

       to do the reverse and clear values under

       conditions too.

    

       USE WITH CAUTION -- TAKE A BACKUP BEFORE

       MAKING ANY RADICAL CHANGES TO YOUR DATA

Cheers

Mark

[Mark Liversedge]

I've now added;

unset(field, conditions)

isset(field)

So you can unset a metric override or metadata value and also filter for rides with a specific metadata tag or override.

isset(Duration) 

is useful for filtering out manual rides.

Mark

[Mark Liversedge]

Sorry, I also mean't to add;

isset(Workout_Code) = 0

in the search/filter box will obviously limit the sidebar to activities that don't have a workout code set.

you can then use that to navigate and update them manually if you are uncomfortably using the set() function.

Mark 

[Jean Div]

WOW! Talk about a speedy turnaround.. comment on the 4th, committed by the 5th!

Was thinking about this on my ride this morning.. about the 'shotgun' nature of it actually. Reason is that my first use for it would be to drop my indoor trainer CP (I know that you don't agree with it, but I have found a consistent 15% discrepancy in my experience with an industrial fan blowing in my face). That was less worrying, because there are less files. Then my next bigger use was to think about taking the 4%-7% discrepancy out for my OSymmetric chainrings.. which would be pretty much every Road and TT ride since 2012.. BIG potential for harm!

My simple head will have to get around the syntax, and I'll definitely make a backup first.. but very glad to see that this comes with a bit of safety now in the 'unset'.

The other thing that I thought about is that I may have had a different CP this time last year, and bulk editing to a hard number wont really allow for this for files across a longer timespan. Would we be able to use an index/% when adjusting.. so in the case of the chainrings for instance, that it takes off 7% of what my CP was at the time, rather than making it a fixed '340W' for the past 4 years..?

Just a thought..

Ta,

Jean

[Will]

I'm having a hard time with this.  I'm trying to set all entries to "Bike" in the sport, and I am trying this command:

set(Sport,"Bike",IF="")

I am getting a syntax error.  I've also tried IF=0, but since it is configured as a text box I didn't think it'd work.

Any help would be appreciated.  Also, I just want to say thanks for 3.2 -- it is really incredible.

Thanks,

Will

[Mark Liversedge]

Its been put into the code for v3.3.

[Will]

Ah, that explains it. Sounds good. Thank you!

Will

[de...@mountainridgept.com]

Hi,

What would I type in the search/filter field to change all blank "sport" metadata fields to "Run"? I'm new to so much in GC that I'm ignorant to the commands. 

I've tried many things like: 

set(Sport, "Run", IF="")

Is there something I should type beyond the "set" command? 

It's the condition or expression definition that I don't quite get, I think. 

Sorry for the beginner question. 

Thank you. 

[Ale Martinez]

El domingo, 11 de diciembre de 2016, 16:05:42 (UTC-3), de...@mountainridgept.com escribió:

Hi,

What would I type in the search/filter field to change all blank "sport" metadata fields to "Run"? I'm new to so much in GC that I'm ignorant to the commands. 

I've tried many things like: 

set(Sport, "Run", IF="")

Is there something I should type beyond the "set" command? 

It's the condition or expression definition that I don't quite get, I think. 

In this case you can use isset(Sport)=0 as the condition to select all activities where Sport is no set, the full command would be: set(Sport, "Run", isset(Sport)=0) 

[de...@mountainridgept.com]

Oh my gosh. That fixed it. You just saved me from manually needing to modify 895 run files! Thank you!

[Will]

On the topic of user interface, it would be useful to have an easier way to do things like this. It's a powerful feature that is not easy to use.

[Mark Liversedge]

On Sunday, 11 December 2016 23:06:26 UTC, Will wrote:

On the topic of user interface, it would be useful to have an easier way to do things like this.  It's a powerful feature that is not easy to use.

Its a bit of an edge case as it stands, but I think its closely related to on-boarding new users -- they are doing this to "adapt" their data on import, to get it ready for GC. Most of the time it seems to be about setting sport or workout codes...

So perhaps the generic and difficult way should remain, but a user-friendly way of managing key fields on first data import could be incorporated into the on-boarding process and/or the activity import process (if not the same)

Mark 

[Ale Martinez]

When other sports were introduced I thought it would be useful to show the detected sport on the import dialog and let the user the chance to change it, similar to date/time, but I never took the time to see how to implement it. 

[Mark Liversedge]

Great idea ! 

I'd like to let folks edit metadata at the same time too, since most users import and then edit them anyway.

Mark 

[Chris Cleeland]

On Mon, Dec 12, 2016 at 5:54 AM, Mark Liversedge <liver...@gmail.com> wrote:

I'd like to let folks edit metadata at the same time [as importing] too, since most users import and then edit them anyway.

Imitation is the most sincere form of flattery, and I think Strava's workflow for this aspect (importing files) is brilliant.  It's obvious what you're doing and what you're supposed to do, a most of the time the user just gets it right.

I'd be cautious about spending a lot of effort making the on-boarding experience easier because that's a one-time activity. Not that you should make it difficult, but folks may be a bit more forgiving in regards to difficulties during the initial import.

--

Chris Cleeland

[Ward Muylaert]

This was not working as-is for me. In case someone else has similar issues, the following did:

set(Sport, "Run", isRun<>1)

This sets *everything* to be a run, not just the blank ones. That was ok for me since the only activities in it were runs anyway.

[Nigel Van de Velde]

Hi,

I have some questions on this set(field,value,condition) function.

Is it possible to set multiple conditions? So for example something like 'set(field,value,Sport="Bike" and TriScore=0)'. I tried adding brackets and quotes, but I can't get it to work properly.

Does the 'value' field need to be alphanumerical? Or can I refer to another metric? I would like to copy Trimp_Zonal_points into the TriScore field.

Kind regards,

Nigel

[Ale Martinez]

El miércoles, 31 de mayo de 2017, 15:36:21 (UTC-3), Nigel Van de Velde escribió:

Hi,

I have some questions on this set(field,value,condition) function.

Is it possible to set multiple conditions? So for example something like 'set(field,value,Sport="Bike" and TriScore=0)'. I tried adding brackets and quotes, but I can't get it to work properly.

Yes, the filter can be any expression, but "and" should be &&, see https://github.com/GoldenCheetah/GoldenCheetah/wiki/UG_Special-Topics_Formula-Syntax-and-Expressions

Does the 'value' field need to be alphanumerical? Or can I refer to another metric? I would like to copy Trimp_Zonal_points into the TriScore field.

No, value is another expression to be evaluated, it can be another metric. 

[Nigel Van de Velde]

Thanks, it worked. The problem seemed to be a bracket present in the custom metric of the "value" field.

[achim...@gmail.com]

Similiar to the original question.

I want to manually set the user-defined metric (say myTSS) via Excel since there are too many activities without power data and load These into GC.

Can I export the activties, set MyTSS and upload this metric back into GC?

[Ale Martinez]

El martes, 5 de febrero de 2019, 13:48:25 (UTC-3), achim...@gmail.com escribió:

Similiar to the original question.

I want to manually set the user-defined metric (say myTSS) via Excel since there are too many activities without power data and load These into GC.

Can I export the activties, set MyTSS and upload this metric back into GC?

I don't think so but, if you are willing to write a script to avoid going one for one, you could update the corresponding json files in activities folder when GC is closed to add an override for the metric. 

[Will]

I believe GC can estimate TSS From HR data if you have HR data. Also, you could use the bulk edit functions as described above in this thread if you want to set an arbitrary amount based on the duration of the ride, the route (if you assign a route), one of the tags for the workout, etc.

Will

[Piotr Wysocki]

Set seem not work in v3.5 build 3991, syntax is ALWAYS not correct, no matter how I try. Can you check?

I just want to change all "Walk" into "Walking" field as preferred by some Garmin 3rd party App. No luck. I have hundreds of "Walk" records, result of walking with my dog.

[Ale Martinez (Please don't email or cc me)]

El sábado, 8 de agosto de 2020, 19:47:17 (UTC-3), Piotr Wysocki escribió:

Set seem not work in v3.5 build 3991, syntax is ALWAYS not correct, no matter how I try. Can you check?

It works.

 

I just want to change all "Walk" into "Walking" field as preferred by some Garmin 3rd party App. No luck. I have hundreds of "Walk" records, result of walking with my dog.

set(Sport, "Walking", Sport="Walk")

[piotr.wys...@gmail.com]

Thanks, it worked. For some mysterious reason it didn't work before, but after putting all spaces AFTER comma as in your input, it worked. Thank you! You saved me a lot of time.

[Guy Melville]

Hi,

I wish to set the Subsport of my trainer rides to VirtualRide to syc older rides with Strava.  I'm trying: 

set(SubSport, "VirtualRide", Device="Zwift")

Nothing changes.

I've trained Subsport and SubSport and spaces around the = sign.

I'm using v3.6 January build.

Please can you advise?

Also, how can I search between two dates?

Best regards,

Guy

[Ale Martinez]

El martes, 2 de marzo de 2021 a la(s) 10:43:43 UTC-3, guy.t.m...@gmail.com escribió:

Hi,

I wish to set the Subsport of my trainer rides to VirtualRide to syc older rides with Strava.  I'm trying: 

set(SubSport, "VirtualRide", Device="Zwift")

Nothing changes.

I've trained Subsport and SubSport and spaces around the = sign.

I'm using v3.6 January build.

Please can you advise?

The recommended way to use set is: first test the filter part (3rd parameter) alone to check it selects the activities you want, then use that filter in set(Field, Value, filter)

 

Also, how can I search between two dates?

Using Date, see the wiki: https://github.com/GoldenCheetah/GoldenCheetah/wiki/UG_Special-Topics_SearchFilter

[Guy Melville]

Hi Ale,

Thank you for your reply.

The Device = "Zwift" Subsport = "VirtualRide" filters works, but the set command doesn't appear to.

Thanks

Guy

--
_______________________________________________
Golden-Cheetah-Users mailing list
golden-che...@googlegroups.com
http://groups.google.com/group/golden-cheetah-users?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "golden-cheetah-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golden-cheetah-users/orBY1d3LKlU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golden-cheetah-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golden-cheetah-users/b5b9977f-dead-47a0-8928-85876700e316n%40googlegroups.com.

[Guy Melville]

It's working, I think.

I didn't have it in filter

Best regards,

Guy

On Tue, 2 Mar 2021, 5:56 pm Ale Martinez, <amtri...@gmail.com> wrote:

--

[Salvatore Scafidi]

Where can I use it?

I am trying to do some auto-processing like in my own metric

# calculate metric value at end

value {

if(Device_Info contains "Assioma")

{

       SSPBike <- 1;

    #set(Trainer,0,1);set(field, value, expr)

       set(Bike, "SL4", 1);

       

}

that will help greatly to categorize rides semi-auto

Can you help

KR
S

[Ale Martinez]

El jueves, 12 de mayo de 2022 a la(s) 01:59:37 UTC-3, ssca...@gmail.com escribió:

Where can I use it?

See https://github.com/GoldenCheetah/GoldenCheetah/wiki/UG_Special-Topics_SearchFilter

 

I am trying to do some auto-processing like in my own metric

Metrics cannot modify data or metadata, they just compute values 

[Ale Martinez]

There’s an entry in FAQs with a set example, and which also illustrates linked defaults, which are useful to automatically set metadata on import: https://github.com/GoldenCheetah/GoldenCheetah/wiki/FAQ-DATA#how-can-i-set-metadata-fields-based-on-conditions

[Salvatore Scafidi]

Thank you @Ale

But How then can I pre process data. I have too many athletes I can't go 1 by one and characterize them.

I need some power to preprocess and only then filter

Thanks

[Steve Edmonds]

To remove identifying information from some activity files I have written a bash script that will parse the .json activity files and will do this recursively through subdirectories so it will parse all .json files for all athletes if need be. It is a bit clunky and in hindsight might have been better in perl or python but is simple and works.

For instance one replacement set I do assigns the parameters as below;

tag="Workout Code"
match="MTB"
replace="G"

What OS are you using.

--

_______________________________________________
Golden-Cheetah-Users mailing list
golden-che...@googlegroups.com
http://groups.google.com/group/golden-cheetah-users?hl=en
---

You received this message because you are subscribed to the Google Groups "golden-cheetah-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golden-cheetah-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golden-cheetah-users/685842c3-4ecb-40ed-8e0a-d564b28401ffn%40googlegroups.com.

[Salvatore Scafidi]

Windows-

ThoughI think preprocessing and post processing capabilities should be part of specification

[Ale Martinez]

El domingo, 15 de mayo de 2022 a la(s) 01:16:15 UTC-3, ssca...@gmail.com escribió:

Thank you @Ale

But How then can I pre process data. I have too many athletes I can't go 1 by one and characterize them.

Automatic pre-processing happens at import time, via default rules and data processors configured to run at import time.

Afterwards it is a manual task at the athlete level.

 

I need some power to preprocess and only then filter

 If the purpose of such preprocessing is just to set some metadata fields to be used in filters, that step could be avoided using the preprocessing criteria directly in filters, without intermediary metadata fields.

[Steve Edmonds]

Processing the activity at import to GC is the most consistent way to customise the fields.
My script will run easily on Linux or Mac but not on Windows and most useful for parsing multiple folders of activities with multiple changes.

As Ale has linked to below, the search/filter bar on GC 3.6 when toggled to filter will allow you to bulk modify tag data in the activities in the activity folder of an athlete (but not all athletes at once).
For instance set(SubSport, "DHS", Workout_Code = "E") changes the value for SubSport to DHS for all activities where Workout_Code = "E".

In your example set(SubSport, "VirtualRide", Device="Zwift") will set SubSport to "VirtualRide" for all activities in the athletes folder where Device="Zwift".

As is mentioned, test the "where" part of the expression first to make sure the correct activities are going to be changed by entering just Device = "Zwift" in the filter bar, there is no undo to the changes by the filter and if unsure backup the contents or your activities folder first.

To view this discussion on the web visit https://groups.google.com/d/msgid/golden-cheetah-users/33b2d416-4393-49c1-b0c4-917bc715243cn%40googlegroups.com.

[Karl Billeter]

On Sun, May 15, 2022 at 04:45:06PM +1200, Steve Edmonds wrote:
> To remove identifying information from some activity files I have written a
> bash script that will parse the .json activity files and will do this
> recursively through subdirectories so it will parse all .json files for all
> athletes if need be. It is a bit clunky and in hindsight might have been
> better in perl or python but is simple and works.

jq is another useful tool for this sort of work.

K

[Paul J]

I'm thinking it would be good to implement an option to run a selected python fix on all activities, currently I think this is only available on:

i) those activities imported or saved, or

ii) manually on individual activities. 

So maybe an additional sub-menu option could be added to run them on all activities, with a "are you really sure confirmation?", for powerful selection and modification of already imported activities?

alternatively (and probably better) an extra button could be added to the manage python fixes menu below the Delete Button (see below) to run the fix selected python on all activities, with a confirmation dialog ?

I'm happy to work on this if it considered useful and the access method is agreed.

Paul.

On Sunday, 30 June 2013 at 09:54:47 UTC+1 liver...@gmail.com wrote:

Short version: No. 

Long version:

I thought about allowing you to edit the values in the navigator, but it was just asking for trouble on a feature I'm not sure many would use and we would need to be able to select multiple activities to do it, like an id3 tag editor. This would be extremely complicated.

When we have a mechanism for simple tagging (like the new OSX finder tags) and a mechanism for selecting multiple rides (for compare), then this feature may be easier to implement!

Mark

[Ale Martinez]

El lunes, 16 de mayo de 2022 a la(s) 15:39:21 UTC-3, paulj...@gmail.com escribió:

I'm thinking it would be good to implement an option to run a selected python fix on all activities, currently I think this is only available on:

i) those activities imported or saved, or

ii) manually on individual activities. 

https://github.com/GoldenCheetah/GoldenCheetah/wiki/UG_Special-Topics_SearchFilter#data-processor-functions 

[Paul J]

Hi Ale,

Thank you for the link, I wasn't aware of the postprocess functionality to select then run a python fix.

I have coded the selection criteria in the python fix along with the modification I wish to make, and just run the fix, so that's why I though it might be useful.

Thanks

Paul.

[Ale Martinez]

El lunes, 16 de mayo de 2022 a la(s) 16:12:53 UTC-3, paulj...@gmail.com escribió:

Hi Ale,

Thank you for the link, I wasn't aware of the postprocess functionality to select then run a python fix.

I have coded the selection criteria in the python fix along with the modification I wish to make, and just run the fix, so that's why I though it might be useful.

Both functions apply to all DP, not just Python fixes, just like set/unset are very flexible, but a graphical UX would make the easier to use.

At some point I thought we could extend the Bulk Export UI, which shows all filtered activities and allows further manual selections, to this operations available only via filters, and deletion which is a long-standing request, but never progressed to implement it.

[Paul J]

Hi Ale,

  I see the Bulk Export UI shows all filtered activities, and as you say allows manual selections, so is the following what you are suggesting ?

  Renamed the access menu option from  "Batch export..." to  "Batch Processing...." or similar,

  Rename the "Activity Batch Export" dialog to "Batch Processing", then add a radio box at the top with the following three options:

    i) Export as    xxxxxxx     to    xxxxx  Browse  overwrite  xx

 

    ii) Run data processor xxxxxxxxx 

    iii) Delete all selected

update the Action column to display the option selected and change the Export button to Run or Execute.

[Ale Martinez]

El martes, 17 de mayo de 2022 a la(s) 16:09:24 UTC-3, paulj...@gmail.com escribió:

Hi Ale,

  I see the Bulk Export UI shows all filtered activities, and as you say allows manual selections, so is the following what you are suggesting ?

  Renamed the access menu option from  "Batch export..." to  "Batch Processing...." or similar,

  Rename the "Activity Batch Export" dialog to "Batch Processing", then add a radio box at the top with the following three options:

    i) Export as    xxxxxxx     to    xxxxx  Browse  overwrite  xx

 

    ii) Run data processor xxxxxxxxx 

    iii) Delete all selected

update the Action column to display the option selected and change the Export button to Run or Execute.

Exactly, plus iv) Set metadata field (drop down?) to value (constant or formula?) for all selected 

[Paul J]

Thanks for the feedback, I'm happy to work on this, so I have created issue: Update Batch export... to Batch Processing.... #4223, to capture the changes.

[Alan Benstead]

I have been using the Batch Processing feature for a while now and find it a very useful addition. I batch process HRV data using Filter R-R Outliers and then batch delete those activity files. Everything seems to work as expected except for the Processing failed for an unknown reason message which appears both after the Filter R-R Outliers and the deletion processes have run, see attached image.

[Paul J]

Hi Alan,

Thanks for the testing, I'll try and repeat the issue you are having, it might possibly be working correctly and just the error message is wrong, I'll take a look...

Paul.

[Paul J]

Ok, so the error system was the last thing I updated, and there is an error in the Python, Core data processors and the Delete function they (I mean I) failed to capture the return value, I'll update the PR in the next hour.

So they are working correctly, just failing to display the right success message, so thanks for spotting the incorrect message.