Lucene - WhereLess/GreaterThanOrEqual on Arabic DateTime

[Alan Ciantar]

First of all, sorry, I know this isn't really a RavenDB issue.

Anyway, when I want to query documents which say, are created on before today, i'd use, .WhereLessThanOrEqual(x=>x.CreatedOn, DateTimeOffset.UtcNow)

Now, apparently Arabic calendar uses some sort of Hijri system, which dates like 24/09/2013 appear as 18/05/1834 (invented values).

So the query would be CreatedOn less than 18/05/1834 instead of the actual date.

From what I've found, I'll have to .ToString() without a culture, but .WhereLessThanOrEqual() only takes a DateTime, not string.

Will I have to write the query myself? ("CreatedOn":"[* To DateTimeOffset.UtcNow.ToString(CultureInfo.InvariantCulture]") (or something similar)

How to get DateTimeOffset.UtcNow without a culture for querying?

[Mircea Chirea]

Why is your thread default culture anything other than invariant?

Thread.CurrentCulture = CultureInfo.Invariant;

Thread.CurrentUICulture = CultureInfo.Invariant;

CultureInfo.DefaultThreadCurrentCulture = CultureInfo.Invariant;

CultureInfo.DefaultThreadCurrentUICulture = CultureInfo.Invariant;

Of course, the real bug is RavenDB not using the ToString overload taking IFormatProvider with CultureInfo.Invariant.

P.S.: There are my assumptions, I haven't checked anything.

[Alan Ciantar]

I'm not really good with these things - so here's what I know:

Current thread is the same one as the one being executed on our Front-end, which has to be Arabic as the site has to be Arabic.

Ideally, I know, that back-end and front-end should be separated so back-end is culture invariant on another thread; but alas, not the case with this application.

Any ideas?

[Itamar Syn-Hershko]

How about indexing the actual milliseconds since epoch value?

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Alan Ciantar]

@Itamar: I've been told that wouldn't be very human readable

---

Since I barely have an idea of Threads/Cultures, I tried the following:

                var currentCulture = Thread.CurrentThread.CurrentCulture;
                Thread.CurrentThread.CurrentCulture = CultureInfo.GetCultureInfo("en");
                currentDateTime = DateTimeOffset.UtcNow;
                Thread.CurrentThread.CurrentCulture = currentCulture;

.. And of course it didn't work :p

So is this a RavenDB bug?

[Itamar Syn-Hershko]

I said indexing, not storing. If you deal with multiple cultures in your DB, thats the only way date search would really make sense anyway

--

[Kijana Woodard]

Surely there must be a way to translate the user supplied date to an iso date instance for storage and querying...or is this something else.

It seems to me the culture setting to format the date should be in the outermost parts of the app.

Or does in need to be stored in the 'Arabic format' for some reason?

Never had to deal with this case, so am curious.

[Itamar Syn-Hershko]

The problem is Lucene indexes dates as numeric fields, but RavenDB still favors storing them as strings. The former can handle timezones and cultures (just store the milliseconds from epoch), the latter can't.

The easiest way out of it is to explicitly convert the date to a numeric value (dateTimeOffset.Milliseconds or similar) in your index definition.

[Mircea Chirea]

Not really. You should manually and religiously pass your target culture to all string conversions. You want the code and app to work in the invariant culture, but use Arabic when displaying to the user - and only then.

If you set the default culture to Arabic, weird things start happening - for example you can't pass a float to an MVC action as "12.45" if the culture has a different decimal separator (such as a comma) - and that is just the tip of the iceberg.

[Kijana Woodard]

This is what I was thinking. Was wondering if there was something I was missing since I don't normally have to deal with these issues in c# code.

--

[Matt Johnson]

Alan,  I think I answered your question on StackOverflow already, but I didn't realize you were coming from a RavenDB perspective.

RavenDB stores any DateTime or DateTimeOffset you give it as an ISO8601 formatted string.  You can't represent a date in ISO format without using the ISO calendar.  (The ISO calendar is essentially the same as the Gregorian calendar, but as if it always existed.)

So when you store a document in RavenDB that has a DateTime or DateTimeOffset field, Raven does .ToString("o") on it.  Regardless of culture settings, this will always give timestamps in the ISO format with the ISO calendar.

Can you actually provide some code that demonstrates how you are getting Hijri dates stored in your documents?  That should not happen.

[Matt Johnson]

@Itamar - RavenDB doesn't index dates as numeric fields.  It indexes them as ISO8601 formatted strings.  For DateTimeOffset fields, it normalizes to UTC during indexing, but it is still stored as a string in the index.  I think you were thinking of TimeSpan values, which are indexed numerically by their .Ticks property.

The idea of indexing DateTime as a pure integer is not a bad one, as long as you are talking about UTC values.  It would work, but it is not very straightforward.  For DateTimeOffset, you would have to normalize to UTC first.  But why bother when Raven will do this for you already?

See also: http://ravendb.net/kb/61/working-with-date-and-time-in-ravendb

[Matt Johnson]

I've done some testing of my own and there is indeed a bug in RavenDB.  The document is getting saved just fine, and it's being indexed properly.  But the query that is being built is affected by the culture and it shouldn't be.

https://gist.github.com/mj1856/6705516

http://issues.hibernatingrhinos.com/issue/RavenDB-1377

[Alan Ciantar]

Thanks for all the help, Matt :)

[Alan Ciantar]

As well as others, haha! 

[Alan Ciantar]

Any updates on when Matt's fixes will be released in a build?

[Oren Eini (Ayende Rahien)]

Already have been

On Mon, Sep 30, 2013 at 12:29 PM, Alan Ciantar <alan.c...@casasoft.com.mt> wrote:

Any updates on when Matt's fixes will be released in a build?

--

[Alan Ciantar]

Oh, Thanks :)

[Itamar Syn-Hershko]

I know that, and this is what I was saying