Different outcomes with different GGIR versions

[Julia]

Dear all,

I first used GGIR script version 2.3-0 for our activity and sleep data collected with geneactiv watch. Since there have been several updates, I am currently using version 3.0-9. Due to these updates/improvements, the results on physical activity and sleep data that I obtained with the latest version differs from the results obtained using 2.3-0 version. I used the same criteria for settings and filtering etc

I noticed that total PA in mg does not differ much for all days. Most differences are within inactivity and MVPA (we used 1 minute bouts). (see added figure)

Are these differences reasonable, did someone may experience the same? Do you have any idea how it is possible that there is most variation in inactivity and MVPA, but light PA seems somewhat comparable between versions?

Thank you, looking forward to your response!

Kind regard,

Julia

 

Code:

#### Input needed from user ####

f0 = 1

f1 = length(file_paths)

mode = c(1,2,3,4,5)

 

GGIR(

   mode = mode,

  datadir = datadir,

  outputdir = outputdir,

  studyname = studyname,

  f0 = f0,

  f1 = f1,

  overwrite = TRUE,

  idloc = 2,

  print.filename = TRUE,

  storefolderstructure = FALSE,

 

  ##### PART 1 Parameters #####

  windowsizes = c(5,900,3600),

  do.cal = TRUE,

  do.enmo = TRUE,

  do.anglez = TRUE, 

  chunksize = 0.5,

  printsummary = TRUE,

 

  do.enmoa = FALSE,

  lb = 0.2, hb = 15,

  do.anglex = FALSE,

do.angley = FALSE,

  #acc.metric = "ENMO",

 

  ##### PART 2 Parameters #####

  do.imp = TRUE,

  strategy = 1,

  hrs.del.start = 0,

  hrs.del.end = 0,

  maxdur = 8,

  includedaycrit = 20,

  #M5L5res = 10,

  #winhr = c(5,10),

  #qlevels = c(c(1380/1440),c(1410/1440)),

  qwindow = c(0,8,21,24),

  ilevels = c(),

  iglevels = c(seq(0,4000,by=25),8000),

  mvpathreshold = c(100),

  bout.metric = 4,

 

  ##### PART 3 Parameters #####

  anglethreshold = 5, 

timethreshold = 5,

  ignorenonwear = TRUE,

  do.part3.pdf = TRUE,

 

  ##### PART 4 Parameters #####

  excludefirstlast = FALSE,

  includenightcrit = 16,

  def.noc.sleep = c(1), outliers.only = FALSE,

  relyonsleeplog = FALSE,

  do.visual = TRUE,

  nnights = 7,

 

  ##### PART 5 Parameters #####

  excludefirstlast.part5 = FALSE,

  save_ms5rawlevels = FALSE,

  threshold.lig = c(40),  threshold.mod = c(100),

  threshold.vig = c(400),  boutcriter = 0.8,

  boutcriter.in = 0.9,  boutcriter.lig = 0.8,

  boutcriter.mvpa = 0.8,

  boutdur.in = c(1,10,30),

  boutdur.lig = c(1,10),

  boutdur.mvpa = c(1),

  timewindow = c("MM", "WW"),

 

  ##### Report generation #####

    do.report = c(2,4,5),

  visualreport = TRUE,

  dofirstpage = TRUE,

  viewingwindow = 2)

[Vincent van Hees]

Dear Julia,

Thanks for your question. Other people also approached me with this question and one group (led by Pedro in cc) even wrote a paper about it: https://www.nature.com/articles/s41598-023-29872-7.

I understand your frustrations and it may be good to take the time to publicly explain why GGIR has changed over time. Not just for you, but also because it will make it easier for me to reference this response in the future when people approach me with the same question:

1. Innovation is impossible without breaking consistency. You cannot expect us to keep everything consistent and at the same time keep improving the code. GGIR is not an implementation of a set of algorithms that were agreed upon by the community in the past but it is based on an active research area where new insights lead to modifications to algorithms. For example, you can do an OLS regression analysis and expect it to be the same as 10 years ago but GGIR is not like that. The research community has never clearly agreed upon an algorithm on how to detect bouts of behaviour irrespective of what classifier is used (machine learning, cut-point or otherwise), so it took me from 2012 to 2020 to iteratively develop an algorithm that I think best reflects what we all assume to be a bout.
2. GGIR makes use of so-called, unit-tests, which is code specifically intended to test that GGIR functionality remains consistent between releases. This attempts to prevent accidental breaks in consistency. Nevertheless, GGIR is developed by a small group of humans by which mistakes are sometimes made and later corrected. The more people are involved in reporting and/or fixing issues the easier it will be to spot those mistakes early.
3. From time to time we add a prototype algorithm and then optimise it based on user feedback. This makes it a lot more effective and cheap to develop than if we would try to develop the functionality outside GGIR and wait for it to be perfect before merging. Note that not all work done on GGIR is paid for by which sometimes things start as an unpaid side project and then evolve into a more serious functionality once funding or external support becomes available. We could do this more professionally but that requires funding.
4. A lot of the algorithms originally designed for count data in 1990s turn out to not be directly applicable to modern data, which then means that efforts are needed to critically review and revise these algorithm. For example, the IV and IS algorithm seem simple to implement at first sight but it actually is far from straightforward as the underlying count calculation was not described in the original paper. Only now after IV and IS have been part of GGIR for 3+ years I have been able to prepare a hopefully final revision with help from others and supported by a lot of research funding (to be merged in a GGIR release later this year).
5. GGIR is open source research software with a relatively small group of people committed to maintaining and advancing GGIR. This is different from popular generic libraries like ggplot2 where dozens of highly skilled R developers are keen to help. As a result, the development of GGIR has been relatively slow although it gained momentum in recent years.
6. Some researchers and companies have copied parts of GGIR for their own internal work without contributing back to GGIR. This is of course allowed according to the open source license, under the condition that GGIR is cited, but it does not help the advancement of GGIR if everyone has their own version with their own bugs and their own challenges without communicating about them. The more collaborative we all can be the easier it is to advance the field.
7. Some of the changes in output can be traced back to the difficulties to read in the accelerometer files and account for accelerometer brand specific issues. Some brands have provided open source R packages to read their data (kudos do ActivInsights and Movisens), while other manufacturers such as ActiGraph and Axivity have not made that  level of effort for the R community, which meant that it took a while before we had figured out how to read in their data in a fast and correct way (kudos to Tuomo who wrote read.gt3x for that). I think we are now at a point that all these data formats are read in correctly, but it remains important that the community keeps critical about even those aspects.
8. Open source software is poorly supported by the research community. Everyone likes free research software but academic conferences and journals are still very active in endorsing the researchers and manufacturers who do not share their code. It is very hard to develop open source research software in this landscape as funding and career opportunities are diverted from the good open science to bad closed science. Addressing this requires a culture change where we keep insisting on openness in all our interactions with those who are not committed.

As a final note: If you want 100% consistent scientific results use the same software version. This applies to any software you use in research.

Kind regards,

Vincent

PS. I wrote this message unpaid because I believe in open science and the importance of advocating for it.

Dr. Vincent van Hees | Independent consultant | https://accelting.com/

--
You received this message because you are subscribed to the Google Groups "R package GGIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to RpackageGGIR...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/RpackageGGIR/d8d6e546-1c1d-4d8d-b543-c18becc4f7f8n%40googlegroups.com.

[Vincent van Hees]

Just to add to my previous message:

In https://cran.r-project.org/web/packages/GGIR/news/news.html you will find a technical log of all key changes per GGIR release.

In https://www.accelting.com/updates/ you will find blog posts that aim to complement this with a non-technical summary.

Kind regards,

Vincent

To view this discussion on the web, visit https://groups.google.com/d/msgid/RpackageGGIR/OK9dEVJbwd7BhorZgmayqcrxDIiuGMfKkkM1DWt_LG9Cvh01uR62_OGA1azahOu9hNF88yAjAFo06eVyjFh7-UhSEZ9Oshk0tgeCmWybzvc%3D%40accelting.com.

[Marques-Vidal Pedro Manuel]

Dear Julia, dear Vincent

 

Indeed, we noted differences between versions regarding physical activity. Looking at your table, the differences for Sleep are much larger, which raises some issues when we study sleep. We now systematically report in our publications the version used to process the data. As we are collecting data almost continuously, we decided to reprocess all the data collected regularly using an updated version of GGIR, so that we can have homogeneous results throughout the study period.

 

Hope this helps.

 

Regards

 

 

CHUV
centre hospitalier universitaire vaudois

Pedro Marques-Vidal - MD, PhD, FESC.

Professeur ordinaire
Département de Médecine

Médecine Interne
BH 10/642
Rue du Bugnon 46, CH-1011 Lausanne
Switzerland
+41 (0)21 314 09 34 TEL
+41 (0)21 314 09 51 FAX
+41 (0)79 556 72 11 MOBILE
Pedro-Manuel....@chuv.ch
http://www.chuv.ch/

 

 

 

[Vincent van Hees]

In terms of comparability with 2.3-0: In release 3.0-10 (GitHub only) I reversed some changes to sleep detection that I made in 3.0-7. So, if you want more comparable results with 2.3-0 then probably you want to use 3.0-10 and not 3.0-9. My aim is to make a new release 3.1-0 for CRAN and GitHub later this month such that these updates are available everywhere.

Also when using metric ENMO I would advise now using minloadcrit = 240, which is now the default.

Kind regards,

Vincent

[Michael Rueschman]

Apologies if I missed this in the above discussion, but another important factor to keep in mind when comparing old to new(er) versions of GGIR is the nonwear_approach parameter (options are 2013 and 2023). In version 3.0-0 and above the default is 2023.

Here's a prior discussion about the implications of the nonwear_approach revision: https://groups.google.com/g/rpackageggir/c/U0qWaaBLFcU/m/WZmw1A-mAQAJ

As a quick test, I chose four (4) random, 14-day recordings and ran them through with a very basic GGIR (version 3.1-0) command using both the 2013 and 2023 nonwear_approach options. I then quickly eyeballed the part5_personsummary files. Two of the recordings appeared to match exactly; these two recordings also appeared to have minimal nonwear time on visual inspection. The other recordings, which did have periods of nonwear interspersed in the recording, ended up with noteworthy differences in key metrics (e.g., 38 and 17 minute differences in average sleep duration (dur_spt_sleep_min_pla)). 

[Julia]

Thank you all for the considerations!

Kind regards, 

Julia 

Op woensdag 8 mei 2024 om 16:52:04 UTC+2 schreef Michael Rueschman:

[Rajan Shrestha]

Dear all,

I encounter a problem with the same GGIR version. While running the same code at different times, it is giving me different values. I have attached a picture of part2summary result.

--
You received this message because you are subscribed to the Google Groups "R package GGIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to RpackageGGIR...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/RpackageGGIR/ab960155-e801-4e0d-b22c-02f0bb5a77d7n%40googlegroups.com.

--

With Regards,

Rajan Shrestha

PhD Student

Aarhus University

Department of Public Health