Sameday filter operator requiring tomorrow's date to match <<now>> of just created tiddler field

[KML]

I'm having difficulties using the sameday operator to select tiddlers based on a date+time field. For example, the following code results in the newly created tiddler being listed under the next day's date (hard-coded into this example) rather than today's date. 

<$button>Make Tiddler with ~MyDateField=<<now "YYYY0MM0DD0hh0mm">>

<$action-sendmessage $message="tm-new-tiddler" title="Tiddler made now" tags="Test" MyDateField=<<now "YYYY0MM0DD0hh0mm">>/>

</$button>

__Tiddlers with sameday in ~MyDateField as <<now "YYYY0MM0DD">>__<br>

<$list filter="[tag[Test]sameday:MyDateField<now>]" >

<$link><<currentTiddler>></$link>   {{!!modified}}  {{!!MyDateField}}

</$list>

__Tiddlers with sameday in ~MyDateField as 20170106__<br>

<$list filter="[tag[Test]sameday:MyDateField[20170106]]" >

<$link><<currentTiddler>></$link>   {{!!modified}}  {{!!MyDateField}}

</$list>

__Tiddlers with sameday in ~MyDateField as 20170107__<br>

<$list filter="[tag[Test]sameday:MyDateField[20170107]]" >

<$link><<currentTiddler>></$link>   {{!!modified}}  {{!!MyDateField}}

</$list>

I understand that there have been some issues around how UTC vs. local time are used, such as those discussed in the "TW5 created/modifed info is UTC/GMT? (odd results in Recent Tiddlers list)" thread. I'm not certain, but I didn't notice this issue on TW version 5.0.9ish. For my immediate purposes, it would be sufficient to just match the first eight characters of a date+time field. I've tried to construct a regular expression filter with the parameter coming from transclusion of another tiddler's field, but am stumbling on the concatenation of the "^" anchor with the transclusion of another tiddler's field.  Is there some way to apply something like the prefix operator to fields (such as my custom date+time) other than titles?

Thanks,

Kevin

[Tobias Beer]

There are a couple of things here.

Some general rules and some more things to consider:

1. fieldnames are mandatorily lowercase, not sure why, but they are:
   
   so, MyDateField is invalid whereas mydatefield is valid
   
   
2. dates are always stored as GMT / always store dates are as GMT!
   
   therefore, you cannot actually create a meaningful custom date using the now macro,
   at least not until the macro provides a second parameter that reads GMT:yes
   returning the date in GMT rather than local time
   
   
3. always view dates in local time, because why would you care about the GMT equivalent?
   
   looking at a date field using plain transclusion, e.g. via {{!!modified}}
   only helps insofar as you need to be aware that you're looking at GMT,
   otherwise use the ViewWidget to get the date displayed in local time
   
   
4. using the <<now>> macro without formatting does not yield a valid tiddler date format
   
   use a $set or $vars widget first, to get the value for <<now>> formatted appropriately, first
   
   
5. you cannot use regexp with dates, esp. not when in GMT, because you're not dealing with strings,
   this is more ore less the same issue as 3.
   
   use or invent proper date handling filters

So, here's an updated example for you, however, not necessarily solving all your problems:

<$button>Make Tiddler with mydatefield=<<now "YYYY0MM0DD0hh0mm">>
<$action-sendmessage $message="tm-new-tiddler" title="Tiddler made now" tags="Test" mydatefield=<<now "YYYY0MM0DD0hh0mm">>/>
</$button>

__Tiddlers with sameday in mydatefield as <<now "YYYY0MM0DD">>__<br>
<$vars rightNow=<<now "YYYY0MM0DD">>>
<$list filter="[tag[Test]sameday:mydatefield<rightNow>]" >
<$link><$view field="title"/></$link><br>
<$view field="modified" format="date" template="YYYY0MM0DD0hh0mm"/><br>
<$view field="mydatefield" format="date" template="YYYY0MM0DD0hh0mm"/><br>
</$list>
</$vars>

__Tiddlers with sameday in mydatefield as 20170106__<br>
<$list filter="[tag[Test]sameday:mydatefield[20170106]]" >
<$link><$view field="title"/></$link><br>
<$view field="modified" format="date" template="YYYY0MM0DD0hh0mm"/><br>
<$view field="mydatefield" format="date" template="YYYY0MM0DD0hh0mm"/><br>
</$list>

__Tiddlers with sameday in mydatefield as 20170107__<br>
<$list filter="[tag[Test]sameday:mydatefield[20170107]]" >
<$link><$view field="title"/></$link><br>
<$view field="modified" format="date" template="YYYY0MM0DD0hh0mm"/><br>
<$view field="mydatefield" format="date" template="YYYY0MM0DD0hh0mm"/><br>
</$list>

Best wishes,

Tobias.

[KML]

Tobias, 

Thank you very much for the pointers and changes to the example. I ran your modified code this morning (in the UTC - 7 time zone), and it produced the following result:

6th January 2017 at 8:09am

Make Tiddler with mydatefield=201701060809

Tiddlers with sameday in mydatefield as 20170106

Tiddlers with sameday in mydatefield as 20170106

Tiddlers with sameday in mydatefield as 20170107
Tiddler made now
201701060809
201701060109

Thus the tiddler created on 2016-Jan-06 (both here and in London) is still being filtered by sameday with a mydatefield matching 2016-Jan-07.  Thus I believe there is a bug in sameday.

Perhaps I should try to skin this problem a different way. I understand the following is contrary to your point #5, but I can create a javascript macro <<nowlocal ...>> that produces a string with the local date and time. I can use a regexp:mydatefield operator to filter for the tiddlers I want to retrieve, but I'm unsure of how to get the regular expression parameter with an initial anchor (^) followed by the contents of yet another tiddler field into a variable that is fully rendered before it gets used in the filter.  In case it is helpful, I'll explain the context, although I don't think this is critical to the solution.  

I include macros in my journal tiddlers that list tasks scheduled for that day as well as work done on that day. To help with this, each journal tiddler has a field named date-journal in the format YYYY0MM0DD matching the date that the tiddler journals. The work tiddlers include two fields, datetime-start and datetime-stop in the format YYYY0MM0DD0hh0mm0ss that are populated by button presses when work is started and stopped, respectively. (These fields are used to calculate elapsed time for the work.)  Within each journal tiddler I would like to list the work tiddlers with datetime-start fields that begin with the 8 digit date corresponding to date-journal. 

Could you kindly suggest how to construct "^{{!!date-journal}}" in such a way that the transclusion occurs before using this string in a filter that includes "tag[Work]regexp:datetime-start[^{{!!date-journal}}]"?  I've sometimes succeeded in creating helper macros, but am stumbling on this one.

Thanks for reading. By the way, I make use of your :let SUM filter to total hours in my work and task tiddlers. Thanks for that contribution of yours as well.

Cheers,

Kevin

[Mark S.]

KML, I'm about an hour off your timezone.

I put the created date in the mydatefield field and it showed up as sameday tomorrow 1/7.

It appears that sameday is assuming that the date field is local time, and rolling the clock forward to make it UTC time. I don't know if that was the planned behavior. If it's a bug, it would be almost invisible to anyone living within a hour of UTC.

[KML]

Mark, 

Thanks for checking and commenting on your result that sameday is matching when the parameter of sameday is one day later. I'm not sure that this is just a local vs. UTC time issue since the result I posted above was done at 08:09 in UTC-7 and thus 15:09 in UTC, both still on 2017-Jan-06. 

I do note that using $view with date format on the string from the tiddler field did subtract 7 hours from it, even though I intended it to be a local time. Perhaps this emphasizes Tobias's point #2 that dates should always be stored as UTC, even when being generated by users or user tiddlers.

Tobias, with respect to your point #5, the reason I believed I should be able to work directly with dates as strings stored in fields, including doing regexp filtering on them, is because when I get info on a tiddler from the drop down menu, it shows the created and modified fields as date+time in a YYYY0MM0DD0hh0ss... format. 

To illustrate the point I'm stuck at in trying to use tiddler fields (such as my date-journal field) to act as part of a regexp filter parameter (to be pre-pended with the "^" anchor), I've put the following example code into a tiddler that has a field named 'date-search' which is equal to '20160622', a date when some tiddlers in TiddlyWiki version 5.1.13 at tiddlywiki.com were modified.

\define concat(1,2,3,4,5)

$1$$2$$3$$4$$5$

\end

\define anchorSearchField()

<$macrocall $name=concat 1="^" 2={{!!date-search}} >

\end

\define wikifiedAnchorSearchField()

<$wikify name=wASF text=<<anchorSearchField>> output=text >

<<wASF>>

</$wikify>

\end

THE FOLLOWING SHOWS THAT USING A LITERAL ANCHORED DATE WORKS AS THE regexp PARAMETER <br>

<$vars dateRegExp="^20160622" >

__List of tiddlers with 'modified' field matching `"^20160622"`=<<dateRegExp>>__ <br>

<$list filter="[regexp:modified<dateRegExp>]">  

<$link><<currentTiddler>></$link>  {{!!modified}}  <br>

</$list>

</$vars>

THE FOLLOWING SHOWS THAT USING TRANSCLUSION OF THE DATE DOES NOT WORK <br>

<$vars dateRegExp=<<anchorSearchField>> >

__List of tiddlers with 'modified' field matching `<<anchorSearchField>>`=<<dateRegExp>>__ <br>

<$list filter="[regexp:modified<dateRegExp>]">  

<$link><<currentTiddler>></$link>  {{!!modified}}  <br>

</$list>

</$vars>

... EVEN IF THE MACRO OUTPUT IS WIKIFIED AS SEEN BELOW. <br>

<$vars dateRegExp=<<wikifiedAnchorSearchField>> >

__List of tiddlers with 'modified' field matching `<<wikifiedAnchorSearchField>>`=<<dateRegExp>>__ <br>

<$list filter="[regexp:modified<dateRegExp>]">  

<$link><<currentTiddler>></$link>  {{!!modified}}  <br>

</$list>

</$vars>

The results of this code are shown below:

ToolsReferencesTaggingListListedFieldsAdvancedSources

created	20170108001700345
date-search	20160622
modified	20170108004923404
tags
title	Tiddler to test regexp date selection

7th January 2017 at 5:49pm

THE FOLLOWING SHOWS THAT USING A LITERAL ANCHORED DATE WORKS AS THE regexp PARAMETER
List of tiddlers with 'modified' field matching "^20160622"=^20160622
Generating Static Sites with TiddlyWiki Wed Jun 22 2016 05:13:43 GMT-0600 (Mountain Daylight Time)
GettingStarted Wed Jun 22 2016 05:13:49 GMT-0600 (Mountain Daylight Time)
HTML in WikiText Wed Jun 22 2016 05:22:59 GMT-0600 (Mountain Daylight Time)
PermaLinks Wed Jun 22 2016 05:13:55 GMT-0600 (Mountain Daylight Time)
Translate TiddlyWiki into your language Wed Jun 22 2016 08:14:41 GMT-0600 (Mountain Daylight Time)

THE FOLLOWING SHOWS THAT USING TRANSCLUSION OF THE DATE DOES NOT WORK
List of tiddlers with 'modified' field matching <<anchorSearchField>>=^20160622

... EVEN IF THE MACRO OUTPUT IS WIKIFIED AS SEEN BELOW.
List of tiddlers with 'modified' field matching <<wikifiedAnchorSearchField>>= ^20160622
 

I conclude that even if I try to wikify (and thus in my mind fully render) the transclusion of the {{!!date-search}} reference, I just can't concatenate the contents of a field with other characters and use the result as a filter parameter.

Of course, I welcome any suggestions on how to combine the contents of a field with other constant strings to create filter parameters.

Thanks,

Kevin

[CL]

Well, I am just end user and I couldn't understand entirely
but if you are trying to overcome problem with UTC time,
then you could use plugin called Tiddlytime.

http://tiddlytime.tiddlyspot.com/

If you are in UTC+01:00 zone, then use this macro:

<<dt "YYYY0MM0DD0hh0mm0ss" "-60">>

If you are in UTC-01:00 zone, then use this macro:
<<dt "YYYY0MM0DD0hh0mm0ss" "60">>

to generate time in UTC

I am not sure this will help you to solve the problem

Best wishes,
CL

[Tobias Beer]

As a general note,

Make sure not to mix up UTC and GMT.

UTC is a standard used for storing date-time information.

GMT, however, refers to a timezone in UTC.

The problems in TiddlyWiki stem from storing dates in GMT, so that all users around the globe have the same date reference when looking at a tiddler.

In other words, when you look at a plain date value in a tiddler, e.g. via {{!!modified}} it is not shown in "your time", but simply as plain text.

If you want it to be displayed in your time or selected compared to your time then you first need to have it converted into your time in order for it to be displayed accordingly, or compared to some other date in local time you wish to compare some date to, for example. That is why regexp is not your friend, unless you first manage to translate the date you wish to compare against to GMT, which more or less just switches what ends of the puzzle you need to flip so as to solve it.

Practically speaking:

tiddler-date: 20161213235959

query-date: 20170101000000

Supposer "tiddler-date" is a date value as stored in a tiddler and "query-date" is a date value as you happen to make it up so as to say "give me everything before that date", e.g. via some filter.

Now, suppose that you have no clue about GMT and so "query.date", for you, is defined as the time you see on your clock. If your use regexp to run that filter, and you're ass is not sitting on top of a GMT couch, then you may or may not actually yield the desired result, since you would first need to figure out what your query date translates to for GMT. Or, put differently, someone sitting in India looking at your wiki, doing your exact query, must inevitably get different results... since 0 o clock on Jan 1 for them isn't 0 o clock on Jan 1 for you.

However, the above assumes that you actually managed to store "tiddler-date" in some custom "date field"... properly, in GMT. I would suspect most implementations, including the one where you type that in by hand, do not respect the convention that the date eventually needs storing in GMT. Now, this produces a total mess, since you have core dates stored in GMT and some user defined dates stored in whatever time their clock was showing when typing. The only way you can fix this, is to somehow batch.convert the dates stored wrong to GMT and to then store them properly in GMT, so that standard date fields and custom date fields speak the same language. Let's just hope and assume, you didn't enter some of your custom dates while you were in Tokyo and others while you were in Guatemala. Then you pretty much need to restart with your efforts recording time, or fix every date manually.

Best wishes,

Tobias. 

[KML]

Tobias,

Thanks for your continued pointers. I'll need to digest what you wrote below, but in the meantime, I found that your $setvars widget (http://tobibeer.github.io/tw5-plugins/#setvars) solved my transclusion (if not my GMT comprehension) problem.  After finding a related prior thread, [TW5] Passing a Transcluded Field Value to a Filter, I was able to get the following code to filter for tiddlers that had the first 8 characters of their modified field matching the contents of !!date-search.

<$setvars _anchor="^" _date={{!!date-search}} dateRegExp="anchor date" >

__List of tiddlers with 'modified' field matching variable defined with setvars =<<dateRegExp>>__ <br>

<$list filter="[regexp:modified<dateRegExp>]">  

<$link><<currentTiddler>></$link>    <$view field=modified format=date template="YYYY0MM0DD" />  <br>

</$list>

</$setvars>

While I'm not clear on all the implications of timezone conversions, I do wonder if part of my confusion is how TW interprets an 8 digit date field when there is no time information present.  For example, would a date field containing only "20170108" always refer to this 8th day of January regardless of time zone, or could TW possibly also take it to be the 7th day in the western hemisphere because it pads it out to be 20170108000000 GMT = 20170107170000 in GMT - 7 for example?  I find it more intuitive to interpret "20170108" as a local date, without regard to time zone, always associated with my journal tiddler from the 8th, but I can understand why the mechanics of TW would want to handle 20170108 the same as 2017010800000.

This discussion does bring up a conceptual distinction question in my mind about the behavior of the sameday operator. Are the two tiddler fields 20170101000001 and 20170101235959 in the same day everywhere as evaluated by sameday[...], or does the answer depend on the time zone? If the answer depends on the time zone, then I guess I need to record the "date only" field in my journal tiddler to be YYYYMMDD070000 if I'm in UTC - 7. And then I'd have to dynamically change this field if I shifted time zones. Perhaps a better alternative is always using a macro that subtracts the timezone offset from (adding 7 hours in my case) a specified 8 digit date before using it as the parameter of the sameday operator.

I am under the impression that GMT and UTC both refer to a time rather than a format, and that they are within a fraction of a second of each other. For example, the header to your message below, generated by Google for better or worse, gives a time in UTC - 7.

Thanks again for your continued insights,

Kevin

On Sunday, January 8, 2017 at 6:51:33 AM UTC-7, Tobias Beer wrote:

As a general note,

Make sure not to mix up UTC and GMT.

UTC is a standard used for storing date-time information.

GMT, however, refers to a timezone in UTC.

[...]