Part5 Time-series CSV Output inconsistency

[Abdul Haleem Butt]

Hi Vincent,

I observed there is a difference in timestamps in the part 5 CSV file. the output time-series CSV started 2 hours before the actual recording and also estimates all metrics with Acceleration. could you please explain this to me? why and how CSV shows results with respect to time which was not recorded in the actual file. 

for example, part% csv time-series output started at 20:15

but actually, the raw part2 csv time series started at 22:15

Looking forward to hearing from you.

Kind Regards, 

Abdul Haleem 

[Abdul Haleem Butt]

Hi all, I am facing a lot of inconsistency in the CSV file obtained. According to the CSV time series, acceleration was high even in non-wear time and sometimes

class_id said wake-IN (1) and invalid epoch said non-wear (1), which is very confusing. Anyone explain will be appreciated.   

  

Kind Regards, 

Abdul Haleem 

[Vincent van Hees]

I am not able to reproduce this.

Were both files created in the same GGIR process on a single computer using the same time zone setting? 

If not, then make sure you keep this standardised.

If yes, can you share a minimum reproducible example, e.g. 1 RData files from the meta/basic file and an R script with the GGIR call that is supposed to produce the error?

Thanks,

Vincent

--
You received this message because you are subscribed to the Google Groups "R package GGIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to RpackageGGIR...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/RpackageGGIR/CAEYmEhhdq4orx0CH8JpR0EiNRyFzjDwo1B1NUUVHaCbLYtDFbQ%40mail.gmail.com.

[Vincent van Hees]

In line with my reply a few minutes ago, could you share example file and the R code you used to run GGIR?

Thanks, Vincent

--

You received this message because you are subscribed to the Google Groups "R package GGIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to RpackageGGIR...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/RpackageGGIR/CAEYmEhjA62_mORtczALxNrN1UP7OXSnc1GOo8HHEDOF-qMRo4g%40mail.gmail.com.

[Vincent van Hees]

Regarding the timestamp difference you observed:

This is because you are doing timestamp conversion in Excel and Excel makes mistake.

If I go to R and convert the numeric time for the first timestamp I get:

> as.POSIXct(1694463300, desiredtz = "")
[1] "2023-09-11 22:15:00 CEST"

So, this is an Excel problem and not a GGIR problem.  I suspect that this is also why you see misalignment in your plots.

Regarding Axivity AX6:

This data should be processed identically to how AX3 accelerometer data is processed. Just make sure you have an up to date version of GGIRread as this is where the data is being loaded and we made various updates to it in the past year.

Regarding your script:

You are using a various duplicated arguments and GGIR warns you about this when you run it. Also you forgot to specify desiredtz by which it is impossible for me to retrieve what timezone settings you used for part 1. When sharing scripts make sure you create a minimum reproducible example to make it easier for others to help you. So, without all the lines that are not necessary to reproduce your issue. Do not dump your messy R code on my desk and expect me to clean it up for you.

Vincent

On Wednesday, 1 November 2023 at 3:40 PM, Abdul Haleem Butt <abdulhal...@gmail.com> wrote:

just, adding one more question, since I am using Axviity Ax6, but only accelerometer data, do you think it will affect the computation? mean pre-processing and non-wear detection, I am wondering, if acceleration should be close to zero in the obtained CSV file, which does not happen in my case.

On Wed, Nov 1, 2023 at 3:36 PM Abdul Haleem Butt <abdulhal...@gmail.com> wrote:

the CSV file obtained (ms5.outraw, added diary info for comparison) and meta/basic R file, and the code with sleeplog csv also attached.

`{R,eval=FALSE}
library(GGIR)
GGIR(mode=c(1,2,3,4,5),
datadir="sample_CWA",
outputdir="GGIR-master/vignettes",
do.report=c(2,4,5),
overwrite = TRUE,
do.imp=TRUE,
do.parallel = FALSE,
idloc=1,
print.filename=FALSE,
storefolderstructure = FALSE,
#=====================
# Part 1
#=====================
windowsizes = c(5,900,3600),
do.cal=TRUE,
do.enmo=TRUE,
do.anglex=TRUE,
do.anglez=TRUE,
do.angley=TRUE,
#do.neishabouricounts = TRUE,
chunksize=1,
printsummary=TRUE,
epochvalues2csv=TRUE,
#minimum_MM_length.part5 = 23,
#save_ms5rawlevels = TRUE,

#=====================
# Part 2
#=====================
strategy = 1,
ndayswindow=7,
hrs.del.start = 0,
hrs.del.end = 0,
cosinor = TRUE,
maxdur = 7,
includedaycrit = 16,
L5M5window = c(0,24),
M5L5res = 10,
winhr = c(5,10),
qlevels = c(c(1380/1440),c(1410/1440),c(1430/1440)),
qwindow=c(0,24),
ilevels = c(seq(0,400,by=50),8000),
mvpathreshold =c(100,120), # (100,120),
#cosinor = TRUE,
#=====================
# Part 3 + 4
#=====================

timethreshold = c(5),
anglethreshold = 5,
# nonwear_approach = "2013",
ignorenonwear = TRUE,
outliers.only = TRUE,
criterror = 4,
do.visual = TRUE,

# Part 4 parameters:
#-------------------------------
excludefirstlast = FALSE,
includenightcrit = 16,
def.noc.sleep = 1, # The time window during which sustained inactivity will be assumed to represent sleep
loglocation= "advance.csv",
# advance = TRUE,
outliers.only = FALSE,
criterror = 4,
relyonguider = FALSE,
sleepwindowType="TimeInBed",
colid=1,
coln1=2,
HASPT.algo = "HDCZA",
# HASIB.algo = "VANHESS",
#HASPT.ignore.invalid = FALSE,
Sadeh_axis = "Z",
do.visual = TRUE,
nnights = 5,
do.sibreport = TRUE,
#=====================
# Part 5
#=====================
threshold.lig = c(30),
threshold.mod = c(100),
threshold.vig = c(400),
boutcriter = 0.8, boutcriter.in = 0.9, boutcriter.lig = 0.8,
boutcriter.mvpa = 0.8, boutdur.in = c(10,20,30), boutdur.lig = c(1,5,10),
boutdur.mvpa = c(1,5,10),
includedaycrit.part5 = 2/3,
frag.metrics="all",
save_ms5rawlevels = TRUE,
ave_ms5raw_format = "csv",
frag.metrics="all",
part5_agg2_60seconds=TRUE,
save_ms5raw_without_invalid=FALSE,

#=====================
# Visual report
#=====================
timewindow = c("MM"),
epochvalues2csv = TRUE,
visualreport=TRUE)
```

On Wed, Nov 1, 2023 at 3:27 PM Abdul Haleem Butt <abdulhal...@gmail.com> wrote:

thank you for your feedback. all files are created on the same computer with the same time zone setting. in the attachment, I am sharing the meta/basic file with the R script.

``{R,eval=FALSE}
library(GGIR)
GGIR(mode=c(1,2,3,4,5),
datadir="E:/Bologna/AGT-IT/Open_Source/R_PAckges/GGIR-master/vignettes/sample_CWA",
outputdir="E:/Bologna/AGT-IT/Open_Source/R_PAckges/GGIR-master/vignettes",
do.report=c(2,4,5),
overwrite = TRUE,
do.imp=TRUE,
do.parallel = FALSE,
idloc=1,
print.filename=FALSE,
storefolderstructure = FALSE,
#=====================
# Part 1
#=====================
windowsizes = c(5,900,3600),
do.cal=TRUE,
do.enmo=TRUE,
do.anglex=TRUE,
do.anglez=TRUE,
do.angley=TRUE,
#do.neishabouricounts = TRUE,
chunksize=1,
printsummary=TRUE,
epochvalues2csv=TRUE,
#minimum_MM_length.part5 = 23,
#save_ms5rawlevels = TRUE,

#=====================
# Part 2
#=====================
strategy = 1,
ndayswindow=7,
hrs.del.start = 0,
hrs.del.end = 0,
cosinor = TRUE,
maxdur = 7,
includedaycrit = 16,
L5M5window = c(0,24),
M5L5res = 10,
winhr = c(5,10),
qlevels = c(c(1380/1440),c(1410/1440),c(1430/1440)),
qwindow=c(0,24),
ilevels = c(seq(0,400,by=50),8000),
mvpathreshold =c(100,120), # (100,120),
#cosinor = TRUE,
#=====================
# Part 3 + 4
#=====================

timethreshold = c(5),
anglethreshold = 5,
# nonwear_approach = "2013",
ignorenonwear = TRUE,
outliers.only = TRUE,
criterror = 4,
do.visual = TRUE,

# Part 4 parameters:
#-------------------------------
excludefirstlast = FALSE,
includenightcrit = 16,
def.noc.sleep = 1, # The time window during which sustained inactivity will be assumed to represent sleep
outliers.only = FALSE,
criterror = 4,
relyonguider = FALSE,
sleepwindowType="TimeInBed",
colid=1,
coln1=2,
HASPT.algo = "HDCZA",
# HASIB.algo = "VANHESS",
#HASPT.ignore.invalid = FALSE,
Sadeh_axis = "Z",
do.visual = TRUE,
nnights = 5,
do.sibreport = TRUE,
#=====================
# Part 5
#=====================
threshold.lig = c(30),
threshold.mod = c(100),
threshold.vig = c(400),
boutcriter = 0.8, boutcriter.in = 0.9, boutcriter.lig = 0.8,
boutcriter.mvpa = 0.8, boutdur.in = c(10,20,30), boutdur.lig = c(1,5,10),
boutdur.mvpa = c(1,5,10),
includedaycrit.part5 = 2/3,
frag.metrics="all",
save_ms5rawlevels = TRUE,
ave_ms5raw_format = "csv",
frag.metrics="all",
part5_agg2_60seconds=TRUE,
save_ms5raw_without_invalid=FALSE,

#=====================
# Visual report
#=====================
time-window = c("MM"),
epochvalues2csv = TRUE,
visualreport=TRUE)

```

I am looking forward to hearing from you.

Kind Regards,

Abdul Haleem

To view this discussion on the web, visit https://groups.google.com/d/msgid/RpackageGGIR/zTaRsf7K1kn3meTsmTFWqQRfqylIcjPPfvUlXVCCqczL6TO8SbOSCWgOpHodLxIYa-dWN5YUia0XrPRbUlSSIV1ofX-pECO0_5771JTXGAs%3D%40accelting.com.

[Abdul Haleem Butt]

Dear Vincent,

I really appreciated your prompt response and patience . Let me try your advice, and then I will let you know if I need help. 

Kind Regards, 

Abdul Haleem 

[Abdul Haleem Butt]

Dear Vincent,

I really appreciated your prompt response and patience . Let me try your advice, and then I will let you know if I need help. 

Kind Regards, 

Abdul Haleem 

[Abdul Haleem Butt]

Dear Vincent,

I really appreciate your help. The results improved significantly in the worn time after rectifying the timezone error. But still, I can see the high acceleration in non-wear, which can be due to some in-herited noise but should not be too high. As you indicated it may be due to insufficient worn time of the accelerometer and based on non-wear criteria labeled as non-weared. 

I would request to see if there is anything missing in the script. I also included the desired time zone in this script.   

library(GGIR)
    GGIR(mode=c(1,2,3,4,5),
    datadir="p1.cwa",
    outputdir="xyz",
    rmc.desiredtz = "Europe/Italy",
    do.report=c(2,4,5),
    overwrite = FALSE,

    do.imp=TRUE,
    do.parallel = FALSE,
    idloc=1,
    print.filename=FALSE,
    storefolderstructure = FALSE,
    #=====================
    # Part 1
    #=====================
    windowsizes = c(5,900,3600),
    do.cal=TRUE,
    do.enmo=TRUE,

    do.angley=TRUE,
    chunksize=1,
    printsummary=TRUE,
    epochvalues2csv=TRUE,  

    #=====================
    # Part 2
    #=====================
    strategy = 1,
    ndayswindow=7,

    use.temperature=FALSE,
      ----

    #=====================
    # Part 3 + 4
    #=====================
     timethreshold = c(5),
    anglethreshold = 5,

    nonwear_approach = "2023",

    ignorenonwear = TRUE,
    outliers.only = TRUE,
    criterror = 4,
    do.visual = TRUE,
   
    # Part 4 parameters:
    #-------------------------------
    excludefirstlast = FALSE,
    includenightcrit = 16,
    def.noc.sleep = 1,   # The time window during which sustained inactivity will be assumed to represent sleep
 loglocation= "advance.csv",

    advance = TRUE,
    outliers.only = FALSE,  

    criterror = 3,

    relyonguider = FALSE,
    sleepwindowType="TimeInBed",
    colid=1,
    coln1=2,

    coln2=3,

    HASPT.algo = "HDCZA",
    HASIB.algo = "Sadeh1994",
    Sadeh_axis = "Y",

    #=====================
   # Part 5
   #=====================
   threshold.lig = c(30),
   threshold.mod = c(100),
   threshold.vig = c(400),
   boutcriter = 0.8,      boutcriter.in = 0.9,     boutcriter.lig = 0.8,
   boutcriter.mvpa = 0.8, boutdur.in = c(10,20,30), boutdur.lig = c(1,5,10),
   boutdur.mvpa = c(1,5,10),
   includedaycrit.part5 = 2/3,
   frag.metrics="all",
   save_ms5rawlevels = TRUE,
   ave_ms5raw_format = "csv",
   frag.metrics="all",
   part5_agg2_60seconds=TRUE,
   save_ms5raw_without_invalid=FALSE,
   
   #=====================
   # Visual report
   #=====================
   timewindow = c("MM"),
   epochvalues2csv = TRUE,
   visualreport=TRUE)

Kind Regards, 

Abdul Haleem 

[Abdul Haleem Butt]

Just, one more clarification I replaced desiredtz = "CET". 

Kind Regards, 

Abdul Haleem 

[Vincent van Hees]

You need to use the full timezone database name, e.g. "Europe/Brussels", as that will also account for day saving time. Specifying "CET" will not do this and will introduce an one hour bias in the summer.

I am not able to reproduce your problem with GGIR. Please provide a minimum reproducible example based on code we both have access to, you are now referring to a visualisation you made outside GGIR.

Vincent

[Abdul Haleem Butt]

On Mon, Nov 6, 2023 at 12:41 PM Abdul Haleem Butt <abdulhal...@gmail.com> wrote:

Dear Vincent, 

thanks for your guidance. Actually, outside the GGIR I am working on Matlab, just converting numeric timestamps in the R (as you mentioned earlier) and importing the CSV file in Matlab for plotting, nothing else. In CSV I included an extra column of Diary, where 0 belongs to sleep-IN and 1 Belongs to any activity based on Class_ID.  For non-wear plotting, I am considering an invalid epochs column. 

The sensitivity and specificity between Diary and Class_ID was more than 90%, however, the non-wear labeling also seems fine but not aligned with the acceleration as expected. since the timezone problem was solved, and acceleration of non-wear in raw data was close to zero but in CSV (obtained from GGIR) is quite high in some points, still I need to consider the timezone issue or something else. 

here are the example figures

the first plotting is raw acceleration only from the x-axis

the second plot is processed by GGIR, if we look at 12:00 on 14 Sep there is a spike, and in raw it was zero. do you think it's related to computation since the angle computed from do.angley is based on Sadeh? or do we still consider timezone-related misalignment?

 

really appreciated your continued assistance. in the attached, I am sharing the CSV obtained in Part 2 and then Part 5 (both edited with diary(EU_Sadeh1994) and without diary).

Kind Regards, 

Abdul Haleem 

[Abdul Haleem Butt]

Here is the Matlab example if you want to reproduce the result with the files  (CSV) I sent earlier

% Load the data

load('Sadeh.mat');

time = EUSadeh1994ZC.timenum;

ACC = EUSadeh1994ZC.ACC;

class = EUSadeh1994ZC.class_id;

Dairy = EUSadeh1994ZC.Dairy;

SPT = EUSadeh1994ZC.SleepPeriodTime;

non_wear = EUSadeh1994ZC.invalidepoch;

unique_dates = unique(dateshift(time, 'start', 'day'));%, 'previous'));

num_rows = 3; % You can adjust these values

num_cols = 2;

% Determine the number of unique dates

num_dates = length(unique_dates);

% % Initialize variables to store sensitivity and specificity

sensitivity = zeros(num_dates, 1);

specificity = zeros(num_dates, 1);

% Calculate sensitivity and specificity based on the binary outcomes

true_positive = sum(class == 1 & Dairy == 1);

true_negative = sum(class == 0 & Dairy == 0);

false_positive = sum(class == 1 & Dairy == 0);

false_negative = sum(class == 0 & Dairy == 1);

sensitivity = true_positive / (true_positive + false_negative); % sleep periods

specificity = true_negative / (true_negative + false_positive); % activities

% % Display sensitivity, specificity, and binary outcomes over time

fprintf('Sensitivity: %.2f\n', sensitivity);

fprintf('Specificity: %.2f\n', specificity);

% Create a figure for subplots

figure;

for i = 1:num_dates

date = unique_dates(i);

date_indices = time >= date & time <= date + days(1);

% Create subplots for ACC, Diary, Class data, and metrics

subplot(num_rows, num_cols, i);

plot(time(date_indices), ACC(date_indices), 'b', 'LineWidth', 1.5);

%title(['Data for ', datestr(date, 'yyyy-mm-dd')]);

title('AX6 Sadeh_2013');

xlabel('Time');

ylabel('Angle in Degree');

% Plot Diary and Class data within the same plot

hold on;

yyaxis right;

plot(time(date_indices), Dairy(date_indices), 'g', 'LineWidth', 1.5);

plot(time(date_indices), class(date_indices), 'r', 'LineWidth', 1.5);

%plot(time(date_indices), SPT(date_indices), 'y', 'LineWidth', 2.5);

ylabel('0: Sleep 1: Wake-IN');

ylim([-0.1, 1.1]);

% Plot non-wear data using a shaded area

non_wear_indices = non_wear(date_indices) == 0;

area(time(date_indices), non_wear_indices * 1.2, 'FaceColor', [0.8, 0.8, 0.8]);

alpha(0.5);

% Add a legend to the subplot

legend('Acceleration', 'Dairy', 'Class', 'Non-Wear', 'Location', 'Best');

annotation('textbox', [0.15, 0.75, 0.1, 0.1], 'String', sprintf('Sens: %.2f\nSpec: %.2f', sensitivity, specificity), 'FitBoxToText', 'on', 'BackgroundColor', 'w');

end

[Abdul Haleem Butt]

for non-wear  one should look for 1 instead of 0   % Plot non-wear data using a shaded area

non_wear_indices = non_wear(date_indices) == 1;

area(time(date_indices), non_wear_indices * 1.2, 'FaceColor', [0.8, 0.8, 0.8]);

alpha(0.5);

[Vincent van Hees]

Hi Abdul,

I do not have a Matlab subscription and this question is going beyond basic support for GGIR which this google group is intended for. Maybe there are others here in the google group who can help you.

Otherwise I would be happy to free up some to help investigate this as a paid consultancy outside this group. Let me know if you are interested via personal message.

Kind regards,

Vincent

Dr. Vincent van Hees | Independent consultant | https://accelting.com/

[Abdul Haleem Butt]

Dear Vincent, 

thanks for your feedback, the issue was resolved, and the value I observed in non-wear was imputed. 

Kind Regards, 

Abdul Haleem