[NoRM MongoDB] DateTime, Epoch, UTC, and you.

[Andrew Theken]

This is just a heads up that I made a "potentially breaking change" to
the BSON Serializer.

All dates will now be stored and retrieved using UTC. I am not sure,
but I believe that the actual timezone information is not stored in
MongoDB, but rather the date time is stored as a long (milliseconds
since EPOCH).

What this means is that BEFORE you pull and use my latest (master or
unstable), you'll need to migrate up your dates, which could end up
being non-trivial, if you run into a migration issue, please post it
here and maybe we can collectively work out appropriate offset - date/
time is hard.

--att

--
You received this message because you are subscribed to the Google Groups "NoRM mongodb" group.
To post to this group, send email to norm-m...@googlegroups.com.
To unsubscribe from this group, send email to norm-mongodb...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/norm-mongodb?hl=en.

[jcgrubbs]

I just ran into an issue with this but not related to any dates stored
with previous version of NoRM but rather what I put in the database is
different than what I get out. When I run the following:

var calendarItem = new CalendarItem {
Name = "CalendarItemA",
Start = DateTime.Now,
End = DateTime.Now.AddDays(1)
};
_calendarItemRepo.Save(calendarItem);

I see the following date in the database for the Start property: Sun
May 02 2010 16:09:08 GMT-0500 (Central Daylight Time)

However, when I pull this back out of the database, what I get is this
(in 12hr time): 5/2/2010 9:09 PM

So, does the change you made account for this difference? Is this a
bug or do I need to handle this on my own in my client code?

Thanks,
JC Grubbs

[Andrew Theken]

Can you do me a favor and tell me what the timezone of the machine you're on is set to? I have a guess - I think that this is mongodb being "helpful"

//Andrew Theken

[Andrew Theken]

Actually, we are returning this to you in UTC, and I think that the DateTime that is deserialized should be in UTC, so you might have to account for the offset.

//Andrew Theken

On Sun, May 2, 2010 at 5:24 PM, jcgrubbs <grub...@gmail.com> wrote:

[jcgrubbs]

My machines timezone is: (UTC-06:00) Central Time (US & Canada)

JC

> > norm-mongodb...@googlegroups.com<norm-mongodb%2Bunsubscribe@google groups.com>
> > .

> > > For more options, visit this group athttp://
> > groups.google.com/group/norm-mongodb?hl=en.
>

> > --
> > You received this message because you are subscribed to the Google Groups
> > "NoRM mongodb" group.
> > To post to this group, send email to norm-m...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > norm-mongodb...@googlegroups.com<norm-mongodb%2Bunsubscribe@google groups.com>
> > .

[jcgrubbs]

This seems kind of like an odd design choice. What essentially
happens is that I save a document with a given date/time, the database
stores that exact date/time but when retrieved it's a different date?
I could somewhat understand if when I save the entity the database
converts it on the way in, but it's strange to me that what's in the
database and what I get as the result of a query are different. What
was the reasoning behind the DateTime change here?

Thanks,
JC

> > norm-mongodb...@googlegroups.com<norm-mongodb%2Bunsubscribe@google groups.com>
> > .

> > > For more options, visit this group athttp://
> > groups.google.com/group/norm-mongodb?hl=en.
>

> > --
> > You received this message because you are subscribed to the Google Groups
> > "NoRM mongodb" group.
> > To post to this group, send email to norm-m...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > norm-mongodb...@googlegroups.com<norm-mongodb%2Bunsubscribe@google groups.com>
> > .

[Andrew Theken]

There was no choice - the BSON Specification requires it to be stored as UTC http://bsonspec.org/#/specification

What I think you're seeing in the Mongo console is that they're reporting it in the current timezone on the server, but the spec doesn't allow us to store the timezone information, therefore we can only deserialize to UTC - its the same "time" just different timezone information.

//Andrew Theken

[Michael Kennedy]

Hi guys,

I'm working on my project which uses MongoDB & NoRM here. I noticed some time issues that have to be related to this UTC business.

Imagine I have a class like this:

public class LogEntry {

   public DateTime DateCreated {get; set;}
}

Setting 

DateCreated = DateTime.Now.Date 

shows it at midnight in memory. But after a save / load, it's now 7am that day - 12am adjusted for Pacific Daylight Time.

That just seems odd to me. I know the spec says store time in BSON in UTC. But can't we go to / from UTC time using these two functions at the serializer level?

DateTime.ToLocalTime();

DateTime.ToUniversalTime();

I can see one argument for canonicalization for why you might just always write in UTC and leave it up to the app to normalize the time to their timezone.

So I'm about to change my app to do that - treat time as UTC and transform it. But if there's a chance it'll be put back into local time automatically I don't want to do that twice. :)

Thoughts?

Best regards,
Michael

[Adam Schroder]

I was dealing with this exact issue last night and at one point changed it to use the .ToLocalTime() on deserialization. 

I was a little hesitant at making the change permanent because I wasn't sure of the consequences. (and still aren't)

Would love to hear others thoughts?

Adam

[Ken Egozi]

imo an application should do all it's DateTime related work in UTC terms.

LocalTime is only relevant within Presentation Layer

--
Ken Egozi.
http://www.kenegozi.com/blog
http://www.delver.com
http://www.musicglue.com
http://www.castleproject.org
http://www.idcc.co.il - הכנס הקהילתי הראשון למפתחי דוטנט - בואו בהמוניכם

[Andrew Theken]

I am not really sure what to to, even UTC can produce issues. It's
actually a pretty simple change, I'm open to doing it either way,
depending on the prevailing wisdom of the group.

-att

On Tuesday, May 11, 2010, Ken Egozi <ego...@gmail.com> wrote:
> imo an application should do all it's DateTime related work in UTC terms.LocalTime is only relevant within Presentation Layer

>
> On Tue, May 11, 2010 at 10:04 AM, Adam Schroder <adamsc...@gmail.com> wrote:
> I was dealing with this exact issue last night and at one point changed it to use the .ToLocalTime() on deserialization.
> I was a little hesitant at making the change permanent because I wasn't sure of the consequences. (and still aren't)
>
> Would love to hear others thoughts?
> Adam
>
> On Tue, May 11, 2010 at 1:26 PM, Michael Kennedy <mkennedy...@gmail.com> wrote:
> Hi guys,
> I'm working on my project which uses MongoDB & NoRM here. I noticed some time issues that have to be related to this UTC business.
>
>
> Imagine I have a class like this:
>
>
> public class LogEntry {   public DateTime DateCreated {get; set;}
> }
> Setting
> DateCreated = DateTime.Now.Date
> shows it at midnight in memory. But after a save / load, it's now 7am that day - 12am adjusted for Pacific Daylight Time.
>
>
>
>
> That just seems odd to me. I know the spec says store time in BSON in UTC. But can't we go to / from UTC time using these two functions at the serializer level?
> DateTime.ToLocalTime();
>
>
>
> DateTime.ToUniversalTime();
> I can see one argument for canonicalization for why you might just always write in UTC and leave it up to the app to normalize the time to their timezone.
>
>
>
>
> So I'm about to change my app to do that - treat time as UTC and transform it. But if there's a chance it'll be put back into local time automatically I don't want to do that twice. :)
>
>
>
>

> Thoughts?Best regards,

> Michael
>
> On Mon, May 3, 2010 at 4:45 AM, Andrew Theken <ath...@gmail.com> wrote:
>
>
>
>
> There was no choice - the BSON Specification requires it to be stored as UTC http://bsonspec.org/#/specification
> What I think you're seeing in the Mongo console is that they're reporting it in the current timezone on the server, but the spec doesn't allow us to store the timezone information, therefore we can only deserialize to UTC - its the same "time" just different timezone information.
>
>
>
>
>
> //Andrew Theken
>
>
> On Sun, May 2, 2010 at 10:42 PM, jcgrubbs <grub...@gmail.com> wrote:
>
>
>
>
>
> This seems kind of like an odd design choice.  What essentially
> happens is that I save a document with a given date/time, the database
> stores that exact date/time but when retrieved it's a different date?
> I could somewhat understand if when I save the entity the database
> converts it on the way in, but it's strange to me that what's in the
> database and what I get as the result of a query are different.  What
> was the reasoning behind the DateTime change here?
>
> Thanks,
> JC
>
> On May 2, 4:33 pm, Andrew Theken <athe...@gmail.com> wrote:
>> Actually, we are returning this to you in UTC, and I think that the DateTime
>> that is deserialized should be in UTC, so you might have to account for the
>> offset.
>>
>> //Andrew Theken
>>
>>
>>
>>
>>
>> On Sun, May 2, 2010 at 5:24 PM, jcgrubbs <grubbs...@gmail.com> wrote:

>> > I just ran into an issue with this but not --

> Ken Egozi.
> http://www.kenegozi.com/blog
> http://www.delver.com
> http://www.musicglue.com
> http://www.castleproject.org
> http://www.idcc.co.il - הכנס הקהילתי הראשון למפתחי דוטנט - בואו בהמוניכם
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "NoRM mongodb" group.
> To post to this group, send email to norm-m...@googlegroups.com.
> To unsubscribe from this group, send email to norm-mongodb...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/norm-mongodb?hl=en.
>

--
//Andrew Theken

[Björn Graf]

On Tue, May 11, 2010 at 1:15 PM, Andrew Theken <ath...@gmail.com> wrote:
> I am not really sure what to to, even UTC can produce issues. It's
> actually a pretty simple change, I'm open to doing it either way,
> depending on the prevailing wisdom of the group.

What about leaving the conversion issue to the user and just store the
time as if it were given in UTC by just stripping any timezone
information (e.g. 2010-05-11 17:05:00+02:00 would be stored as
2010-05-11 17:05:00) when moving the value to BSON? Doing the
conversion implicitly will cause issues for some users sooner or
later: doing the DateTime. ToUniversalTime()/ToLocalTime() dance when
the user stores values for timezones different than the local one
would result in “unexpected” (for the user) values.

Another option could be to just do the ToUniversalTime during storing
and leave it up to the user to deal with her timezone needs…

The “We give up” (and not so serious) solution would be to just throw
an exception when the DT ain’t in UTC ;)

In the end I just hate date and time as much as anyone else :)

-- bg

[Andrew Theken]

We implicitly convert it to UTC when sending it to Mongo. This will not change (unless Mongo/BSON adds support for datetime with timezone, which I don't expect to happen in the short term, as the current mechanism is adequate for most purposes).

The only real issue is on the inbound from Mongodb. We could (and currently do) return as UTC, understanding that the value received by the client may be (conceptually) "different" than the one they stored. Or, do a .ToLocalTime(), which *still* might be a value (conceptually)"different" than the one they sent to MongoDB.

Normally I wouldn't like to expose clients to the internals of the persistence layer.. However, in this case I lean towards automatically converting to UTC on serialization, and returning UTC on deserialization.

//Andrew Theken

[James Avery]

I agree with that, I think we save as UTC and return as UTC is the best bet. I think doing this is a best practice anyway, so forcing people into a best practice isn't a bad thing.

-James

--
James Avery
Zerk Media - http://zerkmedia.com
Infozerk Inc. - http://www.infozerk.com

[jcgrubbs]

What's the approach for queries then? For example, if I save an
entity with a DateTime property which gets converted to UTC I assume
any date related query arguments would also need to be converted to
UTC before being passed in? Just one more issue a client code
developer will need to consider. This is a much stickier problem than
I originally thought and I'm starting to come around to storing and
returning UTC across the board...still a pain in the arse though.

Thanks,
JC

On May 11, 10:32 am, James Avery <jav...@infozerk.com> wrote:
> I agree with that, I think we save as UTC and return as UTC is the best bet.
> I think doing this is a best practice anyway, so forcing people into a best
> practice isn't a bad thing.
>
> -James
>
>
>
>
>
> On Tue, May 11, 2010 at 11:31 AM, Andrew Theken <athe...@gmail.com> wrote:
> > We implicitly convert it to UTC when sending it to Mongo. This will not
> > change (unless Mongo/BSON adds support for datetime with timezone, which I
> > don't expect to happen in the short term, as the current mechanism is
> > adequate for most purposes).
>
> > The only real issue is on the inbound from Mongodb. We could (and currently
> > do) return as UTC, understanding that the value received by the client may
> > be (conceptually) "different" than the one they stored. Or, do a
> > .ToLocalTime(), which *still* might be a value (conceptually)"different"
> > than the one they sent to MongoDB.
>
> > Normally I wouldn't like to expose clients to the internals of the
> > persistence layer.. However, in this case I lean towards automatically
> > converting to UTC on serialization, and returning UTC on deserialization.
>
> > //Andrew Theken
>

> > On Tue, May 11, 2010 at 11:14 AM, Björn Graf <bjoern.g...@gmail.com>wrote:

>
> >> On Tue, May 11, 2010 at 1:15 PM, Andrew Theken <athe...@gmail.com> wrote:
> >> > I am not really sure what to to, even UTC can produce issues. It's
> >> > actually a pretty simple change, I'm open to doing it either way,
> >> > depending on the prevailing wisdom of the group.
>
> >> What about leaving the conversion issue to the user and just store the
> >> time as if it were given in UTC by just stripping any timezone
> >> information (e.g. 2010-05-11 17:05:00+02:00 would be stored as
> >> 2010-05-11 17:05:00) when moving the value to BSON? Doing the
> >> conversion implicitly will cause issues for some users sooner or
> >> later: doing the DateTime. ToUniversalTime()/ToLocalTime() dance when
> >> the user stores values for timezones different than the local one
> >> would result in “unexpected” (for the user) values.
>
> >> Another option could be to just do the ToUniversalTime during storing
> >> and leave it up to the user to deal with her timezone needs…
>
> >> The “We give up” (and not so serious) solution would be to just throw
> >> an exception when the DT ain’t in UTC ;)
>
> >> In the end I just hate date and time as much as anyone else :)
>
> >> -- bg
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "NoRM mongodb" group.
> >> To post to this group, send email to norm-m...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> norm-mongodb...@googlegroups.com<norm-mongodb%2Bunsubscribe@google groups.com>
> >> .

> >> For more options, visit this group at
> >>http://groups.google.com/group/norm-mongodb?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "NoRM mongodb" group.
> > To post to this group, send email to norm-m...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > norm-mongodb...@googlegroups.com<norm-mongodb%2Bunsubscribe@google groups.com>
> > .

> > For more options, visit this group at
> >http://groups.google.com/group/norm-mongodb?hl=en.
>
> --
> James Avery
> Zerk Media -http://zerkmedia.com

> Infozerk Inc. -http://www.infozerk.com

>
> --
> You received this message because you are subscribed to the Google Groups "NoRM mongodb" group.
> To post to this group, send email to norm-m...@googlegroups.com.
> To unsubscribe from this group, send email to norm-mongodb...@googlegroups.com.

> For more options, visit this group athttp://groups.google.com/group/norm-mongodb?hl=en.

[Adam Schroder]

The dates will get converted to UTC before being sent to mongo for a query.

[Michael Kennedy]

Good to hear Adam. So it sounds to me like everyone is leaning towards this:

* Store all data in Mongo by converting the source date to UTC.

* Returned values are in UTC without conversion

* Query values are converted to UTC.

So really it's the developer who needs to be careful that all their entities and making sure it's clear what the timezone is. Things like: 

   public DateTime CreatedDateUtc {get; set;} 

might be a good idea to drill home the requirement.

Any thoughts to code by convention? So if I have two properties:

DateTime CreatedDateUtc {get; set;}

and

DateTime CreatedDate {get; set;}

would it be evil to see the "Utc" ending and keep that date always as UTC and throw an exception if it's in local time before saving it. Whereas on the other hand, without the Utc suffix the driver could auto translate to/from UTC.

Best regards,
Michael

Michael Kennedy
http://www.michaelckennedy.net

[James Avery]

I don't think you want to do magic stuff based on the naming of the property, I think you are right on with your three bullet points though. That sounds like the best plan.

-James

Zerk Media - http://zerkmedia.com

Infozerk Inc. - http://www.infozerk.com

[Ken Egozi]

guys, dates are dates. they represent a moment in time, not the representation of that moment.

Therefore, having two properties is weird at best.

now to make things jolly and simple - always do *everything* in your app layer and beyond in Utc terms, and have the presentation layer (or whatever surface you have) make sure that inputs are converted to Utc, and outputs are string.formatted according to the user's preference. 

--
Ken Egozi.
http://www.kenegozi.com/blog
http://www.delver.com
http://www.musicglue.com
http://www.castleproject.org
http://www.idcc.co.il - הכנס הקהילתי הראשון למפתחי דוטנט - בואו בהמוניכם

[Andrew Theken]

yeah, I think we're all pretty much settled on the stuff outlined below.

that being said, and mostly as a total asside, doing everything in UTC still has pitfalls.

why didn't our forefathers realize the mess they were creating for us when they came up with the units that yet did.

// Andrew Theken