GoldenCheetah Computer Spec Bottleneck..?

[Jean Div]

Hey All

Just a quick question on what the key bottleneck is likely to be on GC once an Athlete library reaches a reasonable size (+1000 files, multiple athletes). When moving between workouts and Tabs, there is a delay (spinning ball cursor) of anywhere between 5s-25s. Not the end of the world and I can appreciate that there are a ton of files that GC is reading, but I was wondering what the particular bit of the laptop is taking strain.

My current Macbook Pro spec is:

Processor - Intel i5 2.5ghz

Ram - 16GB (Crucial)

HDD - 240GB SSD (OCZ Vector)

I cant see it being the RAM @ 16GB, so I'd guess that it's either the processor or the read speed on the HDD.

So, if I had to upgrade with the specific aim to improve GC performance, would I spend money on a RAID SSD to bump up read/write speed, or an i7 chip to bump up the processing performance..?

Ta,

Jean

[Mark Liversedge]

You need to be more specific. Which charts. What activity. What date range.

Almost all data is pre-computed so number of workouts is irrelevant once GC is up and running.

Mark

[Jean Div]

Opening a new user usually takes the longest -15-30s. This is also for initial load time, which is a bit longer even. Even with users with only c.100 files.

Closing an open user tab takes about 5s.

Going to 'Trends' view always takes about 13s the first time round, then it can be quicker after that (Just timed that to the 'Summary' tab). Once Trends is open, its a lot quicker (almost instant).

Diary is about 4s on first time for an athlete, quicker afterwards (1s).

Calendar is also quick (1s).

Moving between open files in the Activity view is quick (c.1s).

Hope thats helpful..?

[Jean Div]

..and to answer the other questions, it doesn't seem to be dependant on date range or activity. 

[Mark Liversedge]

Yes. When you open things for the first time it needs to load the cache's etc Once you are in it will be fine.

If you want it to be quicker I'd suggest an SSD, its the best upgrade I have ever done on my Mac.

Mark

[Mark Liversedge]

I see you already have an SSD .. maybe just accept the first time in will take a handful of seconds !

Mark

[Jean Div]

Yeah, already have 1 SSD and it made a massive difference. My MBP is the mid 2012 model, so I have the option to install a data doubler and run SSD RAID to double the read/write speed, but it doesn't seem like a worthwhile investment to save 10s when I open a GC user. Perhaps spend it on carbon parts rather ;-)

[Henrik Johansson]

Not very familiar with OSX but there should be some kind of sleep and hibernation modes to be used instead of really turning the computer off (most of the times).

Then one can just keep GC on for longer periods and "starting" it should be as fast as the SSD & SDRAM swaps in the GC memory pages instead of the normal start delay.

This will not help much if you often need to power cycle the computer for other reasons or if a GC release is less stable.

rgs
Henrik

[Chris Cleeland]

On Wed, Jun 15, 2016 at 4:22 AM, Mark Liversedge <liver...@gmail.com> wrote:

Yes. When you open things for the first time it needs to load the cache's etc Once you are in it will be fine.

Does it load the caches by parsing json, or are they serialized in some binary format?

Does it load caches all the way back to the beginning of time, or just for the selected date range/period?

I'm wondering if it's we could make the cache loading a little smarter and maybe use something like mmap for efficiency.

[Jean Div]

Hey Henrik

Its a good suggestion.. I had a few issue with battery drainage so I was shutting down but I'll 'sleep' the MBP more often, if its on the power source.

GC still crashes *fairly* regularly for me with heavy use, and that makes me go through the lag again.. but c'est la vie!

Ta,

Jean

[Mark Liversedge]

We could definitely speed things up by using binary formats like HDF5 to support parallel read/write operations into the data. We could use mmap to help once we did that. At present the JSON is read in serial from top to toe, so mmap not likely to help much (?).

I'm avoiding that for now since using JSON makes it much easier for others to work with the data (although you could argue that's already covered with R and the RESTful API).

Cheers

Mark

[Mike Patton]

The only issue with the suggestion to not close GC is if I am sharing files via dropbox with an athlete (via symlink) and the athlete uploads a new file, I have to reload GC in order for the file to become visible. If the file list could be refreshed without closing GC, that would solve that quite nicely.

[Chris Cleeland]

On Wed, Jun 15, 2016 at 11:14 AM, Mark Liversedge <liver...@gmail.com> wrote:

We could definitely speed things up by using binary formats like HDF5 to support parallel read/write operations into the data. We could use mmap to help once we did that. At present the JSON is read in serial from top to toe, so mmap not likely to help much (?).

Agreed--mmap likely won't help since I would guess that the delay is dominated by the parsing and setup of internal (binary) structures.

 

I'm avoiding that for now since using JSON makes it much easier for others to work with the data (although you could argue that's already covered with R and the RESTful API).

I wouldn't advocate abandoning the JSON format--just crafting the binary structures such that they could be (a) easily mmap'd back in and (b) validated that the information they contain remains consistent with the json content, e.g., using file hash (even crc32 might be sufficient) and keeping that in the binary structure for comparison.  Only parse the json if the hashes don't match.

--

Chris Cleeland

[Claus Assmann]

On Wed, Jun 15, 2016, Chris Cleeland wrote:

> I wouldn't advocate abandoning the JSON format--just crafting the binary
> structures such that they could be (a) easily mmap'd back in and (b)

Isn't that what the files (.cpx) in cache/ are for?
If those files exist (and are "up to date") does GC really (need
to) read/parse the .json files from activities/ ?

--
Reply-To: is set...

[Chris Cleeland]

I have no idea.

[Mark Liversedge]

CPX

There is a .cpx file for each ride. It contains pre-computed mean maximals and distributions for all data series in a ride file. It is a binary format. They are cpu intensive to compute, and doubly so when you request for a date range when we read and aggregate them (and then cache the aggregate for reuse).

In practice you won't notice CPX i/o that much since we have optimised the format to make it possible to jump straight to the data you need. For example to get 20m peak power for a ride we read the CPX header (<128 bytes), seek to the point where 20m peak is stored and read 8 bytes for the (float) value.

RideDB.json

The rideDB.json contains the pre-computed metrics for the ride and all the intervals. At startup the integrity of the RideDB are checked and refreshed as necessary. We read in the RideDB into memory at startup, its what the 0-100% indicator is telling you.

Improving the performance of the rideDB load is where using a binary format will help, but at the cost of openness. I guess this is important if you are opening multiple athletes as it becomes time consuming to get going.

Mark 

[Mark Liversedge]

On Wednesday, 15 June 2016 17:10:53 UTC+1, Jean Div wrote:

GC still crashes *fairly* regularly for me with heavy use, and that makes me go through the lag again.. but c'est la vie!

Please do share the OSX dumps, we want to fix them.

I rarely get a crash, but probably touch different features to you, so am interested in seeing them.

Mark 

[Mark Liversedge]

On Wednesday, 15 June 2016 17:15:00 UTC+1, Mike Patton wrote:

The only issue with the suggestion to not close GC is if I am sharing files via dropbox with an athlete (via symlink) and the athlete uploads a new file, I have to reload GC in order for the file to become visible. If the file list could be refreshed without closing GC, that would solve that quite nicely.

We could add a file system watcher .. is it just the activities directory you update?

Mark

[Eric Ragot]

I have incredibility long startup and processing times. I am running the latest 3. Version. I have an AMD quad core running at 2.0ghz and 8gb ram and an SSD.

Startup to get to the athlete choice screen is at least a minute.

Auto import is on and that takes manyinutes for just a few files. Moving between trends and my main screen(diary?) Takes about two minutes as well. Saving files takes 30sec or so. Closing is back into the minutes range.

2.x used to run so fast. 3.0 started the downward slide.

Any ideas?

[Mark Liversedge]

Do you have elevation correction or some other data processor active ? (this was a common problem)

Are you running 3.3?

Mark 

[Claus Assmann]

On Wed, Jun 15, 2016, Eric Ragot wrote:
> I have incredibility long startup and processing times. I am running
> the latest 3. Version. I have an AMD quad core running at 2.0ghz and
> 8gb ram and an SSD.

> Startup to get to the athlete choice screen is at least a minute.

Similar here... (but without SSD, for GC 3.x and current).

No "auto-correction" of anything enabled, it's "just slow" to read
and process all those files (500-1200 rides, depending on "athlete").

That's one of the reasons why I "hate" GUIs that "grab the focus"
as I try to do other things while GC is starting (or reading files
later on).

[Eric Ragot]

Unless it is turned on by default, no correction.

I still love GC but can't wait to work with client files. Thus a little less great.

[Mark Liversedge]

On Thursday, 16 June 2016 23:36:49 UTC+1, Eric Ragot wrote:

Unless it is turned on by default, no correction.

I still love GC but can't wait to work with client files. Thus a little less great.

Why not go and check: Options/Preferences > Data Fields > Processing. 

And which version are you using?

Mark

[Jean Div]

Hey Mark

As requested, a collection of crash files from the past few weeks. About 4/5 are unique. there are a few that occurred when GC kept crashing trying to start a Train session.

Hope thats helpful..?

Ta,

Jean

[Mark Liversedge]

Not really that helpful, but the OSX crash reports would be useful.

Also, have you tried 4.0 beta ?

Mark

[Jean Div]

Crap.. I've been saving the wrong things! Where would I get the crash files from?

I've been beta testing 4.0 in a Virtualbox running Ubuntu. Didn't want to mess around with dual installs in OSX, and I saw on your 'How To' vids that you had used Ubuntu, so just assumed that it would be more stable.

Perhaps I'll try OSX beta then.. unless you tell me I'm crazy to do it? (Accepted disclaimer: Beta, backups, no responsibility, et al :-) ).

[Mark Liversedge]

The OSX crash logs are all saved. http://www.cnet.com/uk/news/tutorial-an-introduction-to-reading-mac-os-x-crash-reports/

V4.0 has some bug fixes and would certainly recommend giving it a try.

Mark

[Jean Div]

Hey Mark

The new Dev version is definitely a LOT more stable.. but I'm picking up a crash after trying to use the 'Linked Fields' in defaults as described by Ale here.

Crash Logs attached.