sameday filter - off by one?

[Jim Farrand]

I have recently created a tiddler, which has created timestamp: 20180720180051119.  I would naively assume that the filter:

[sameday:created[20180720]]

would return this tiddler, but it doesn't.  It is, however, returned by this filter:

[sameday:created[20180721]]

Is this just how sameday works, or do I have some sort of setup problem?  If sameday works this way, I would have thought this might be mentioned in the documentation for that operator, so I suspect I have some issue specific to me.  I'm currently in timezone PDT, whereas my "normal" timezone is BST, so I'm wondering if I have some confused configuration somewhere which is screwing things up.

(I'm trying to create a journal template that includes the list of tiddlers created/modified on that day.)

Thanks in advance,

Jim

[Mark S.]

I think it's a bug.

What it's doing is taking the string you gave it and appending zeros to make it represent that date as a UTC date at the very start. Then it converts that date into the local date and truncates the hours.

Then it compares that date against the field date (stored as UTC) converted to a local date truncated by hours.

The error is at the very start. By turning your string into the UTC date at the very start of the day BEFORE converting it into a local date, it has effectively moved the comparison date  BACK 7 hours. In this case, that means the actual date is 7/19 local time.  The UTC field date is still on 7/20.

The reason this error hasn't been noticed is because it works fine when the conversion compares against a date field (the majority of use cases) and rarely gives an error for anyone operating near GMT-0.

I think the code needs to be altered to assume that the target date is a local date (not UTC date) WHEN a short (non-field) date is used.

-- Mark

[cmari]

From a fellow PDT-er, here's the thread containing my workaround and the link to the still open ticket about this issue (hard to believe it's been an open ticket for nearly four years - clearly we PDT-ers need to both bulk up and recruit people more clever than I am!):

https://groups.google.com/forum/?hl=en#!search/sameday$20cmari/tiddlywiki/28DxHYLt6-0/JF3k6FFOKwQJ

cmari

[TonyM]

On my mobile the links not opening the discussion. I just thought of a work around as well, and would like to see yours.

Tony

[TonyM]

Jim,

In your examples you use a litteral date, surely you need it to come from a field a value or something else. How do you plan to actualy use sameday in your tiddlers?

I have been doing a lot with dates of late.

Regards
Tony

[Mark S.]

Tony,

You may be right, but that's not what the docs say. The docs promise that you can feed it an actual string and get expected results. So, either the code needs to change, or the docs need to change ;-)

-- Mark

[Jim Farrand]

On Friday, 20 July 2018 20:08:21 UTC-7, TonyM wrote:

Jim,

In your examples you use a litteral date, surely you need it to come from a field a value or something else. How do you plan to actualy use sameday in your tiddlers?

Hi Tony,

Yes, I simplified for the sake of the discussion.  In the real thing, I have the date stored in a field, and I'm using a macro to put it into the filter.  I have this all working, apart from the fact that I get the previous days tiddlers attached to each journal entry.

Looking through those tickets on github, I wonder if there is some analysis-paralysis happening.  The issue about sameday doing the wrong thing has been closed in favour of a much bigger and more controversial discussion about whether or not tiddlywiki should store timezones alongside dates.

The proposed storage change seems like a bit of a distraction.  Regardless of whether or not that change is made, sameday etc need to convert the query dates into the same timezone as the stored dates before doing the comparison.  Isn't is a pretty clear win to make sameday take an optional timezone (and treat argument as local time by default)?  There might be some small number of people who are relying on the buggy behaviour - they will have to update their code to explicitly mention UTC if that is the behaviour they really want.  But I think that number of people will be small compared to those who will benefit.

Regards,

Jim

[TonyM]

Jim,

The other way around I would think, all dates utc and when you view them the local timezone offset is applied. Including daylight.

This is nessasary for time zone shared wikis. I can say this since I live at +10

The view widget has the template parameter with the utc option and you can format the same full date code to the micro second. Perhaps using the view widget with or without utc before you feed it into your sameday filter in a variable, will solve this issue?

To me date field values are barly readable as they are so why not maintain them in utc and always filter and display them as local time.

Evans formula plugin and the rpn maths plugins provide a little more in the date, Maths and the pickaday plugin is good for date selection with a popup calendar.

Regards
Tony

[Mark S.]

This is an experimental fix to the sameday filter. ABSOLUTELY make a backup if you try this, because javascript errors can be insidious.

Drag and drop onto your tw (greenbar at top) and import. Save. Reload. Test.

The real fix would probably affect two core tiddlers, but for convenience I put the changes just into one.

-- Mark

On Friday, July 20, 2018 at 11:51:50 AM UTC-7, Jim Farrand wrote:

[Jim Farrand]

On Friday, 20 July 2018 21:21:16 UTC-7, TonyM wrote:

Jim,

The other way around I would think, all dates utc and when you view them the local timezone offset is applied. Including daylight.

This is nessasary for time zone shared wikis. I can say this since I live at +10

I agree, I think we are both suggesting the same thing.  As far as I can tell, the dates are already stored in UTC.  It's 20180720 22:29 here in PDT, and I just created a tiddler, and lookted at the timestamp in the file on disk is  201807210529 which is the time in UTC.  sameday needs to make the corresponding conversion to UTC so that it's comparing like to like (and as you say, display functions needs to convert either to local time, or to an explicit timezone if thay are given one).

Reading issue 2061, it seems that at least some people think that the timezone should also be stored on disk.  I disagree.  But it's also irrelevant because even if that change were to be made, we would still need to fix sameday etc.  Storing the timezone on disk is IMHO the wrong thing to do, but I don't see how it fixes the problem by itself, and it seems that no-one can be bothered to do it anyway.  So we might as well just fix sameday etc, which is easier, cleaner, and actually solves the problem.

I guess I'm hoping that a dev will pipe up and either say "go ahead and make this change, we'll probably accept the patch", OR... point out the missing line of reasoning that explains why it isn't the right solution.

 

The view widget has the template parameter with the utc option and you can format the same full date code to the micro second. Perhaps using the view widget with or without utc before you feed it into your sameday filter in a variable, will solve this issue?

I'm not sure how that would work.  What does the syntax look like to run a filter on the result of the view widget?

 

Regards,

Jim

[Jim Farrand]

On Friday, 20 July 2018 22:41:13 UTC-7, Mark S. wrote:

This is an experimental fix to the sameday filter. ABSOLUTELY make a backup if you try this, because javascript errors can be insidious.

Drag and drop onto your tw (greenbar at top) and import. Save. Reload. Test.

The real fix would probably affect two core tiddlers, but for convenience I put the changes just into one.

Hi Mark,

Thanks, this feels like progress!

For me your change errors with:

TypeError: tiddler.getFieldDay is not a function

I'm on 5.1.13 and I can't see that method in the source code for that version.  I notice that it is present in the code on github, so I guess maybe I just need to update my install to a new version.  Unfortunately it's late here in PDT, but I'll try to do this ASAP.

(Ironically I'm travelling back to BST tomorrow where this bug will impact me a lot less, but I'd still be glad to see if fixed.  :) )

Cheers,

Jim

[TonyM]

Jim,

Un-verified something like this

<$wikify name=new-date text="<View$ field="in-date" format="date" template="YYYY0MM0DDhh0mm0ss0XXX"/> >
<$list filter="[sameday:created<new-date>]">

</$list>
</$wikify>>

and if it gives you a different result 

template="[UTC]YYYY0MM0DDhh0mm0ss0XXX"

In your case "in-date" may be "created"

Using the aformentioned rpn or formulas plugin you could add or subtract the time zone.

Regards

Tony

[PMario]

Hi folks,

just go to github and +1 the "bump" ... :) https://github.com/Jermolene/TiddlyWiki5/issues/2061#issuecomment-406781727

-m

[Mark S.]

Yeah. The fix was designed around the latest version of TW.

-- Mark

[cmari]

This conversation has moved well beyond my very crude fix, but for what it is worth, my solution has been

1. Never use hours and minutes, just year, month, day.

2. Add 08 to the end of date string for all deadlines (or anything else that I want to be accurate in my time). So for example:

sameday[2018072108]

I think that technically I would need to change to 07 during daylight saving time, but (as people in continental Europe have noted) the hour difference is rarely noticeable.

cmari