Timezones and price fetcher

[Martin Blais]

I have an interesting quandry right now about price fetchers and timezones, which explains why the build is broken.

I was a bit sloppy and never defined precisely the timezone of the input date to the price fetcher (when you use bean-price ... -d <date>) nor what ought to be the timezone of the date it outputs.

That's why it fails from Europe, for instance.

If you're in a timezone other than that of the market (e.g. you're located in Japan and fetching the price of an instrument trading in the USA), what should be the date that the price fetcher prints?

One school of thought (say, A) would be that the date should be that of the market where the instrument is traded. One problem is that this is well defined for stocks and bonds, but not so for FX / currency exchange rates. So you end up treating these things differently (e.g., for FX you input and output the date as of the user's timezone). Moreover, of the four price sources I have, only Yahoo outputs the exchange's timezone. We can assume IEX is all USA products. I wrote to Quandl about this - their instruments are global and they don't provide a timezone. OANDA is all FX.

The other schools of though (say, B) would be that since all the transactions are recorded in the user's timezone, the prices should be printed with dates relative to that timezone. So a closing price at 2018-04-09 in the USA market might render as 2018-04-10 when fetched from Tokyo (depending on the time it is fetched). This difference is somewhat inelegant, but on the other hand it makes the treatment of timezones consistent - you always convert date/timestamps from/to the user's timezone.

(B) is easy to implement and well-defined, but it means the dates may not match that seen on the particular markets (though they always line up with the user's transactions).

(A) is trickier to implement - each instrument has to have an associated timezone and I have no authoritative source for that - but seems more coherent.

An idea to resolve (A)'s problems would be to use optional metadata on the Commodity directives to let the user provide the timezone and if the timezone isn't provided, to assume the instrument in the timezone of the user and compute the date that way.

(I think a similar issue occurs about accounts. For instance, if you're in Japan and you access an account in the US, do you use US dates? For sure you do - especially if you use importers. It doesn't have the same impact though, because inter-account transactions already have limbo time because of posting vs. settlement dates.)

Any thoughts welcome. I lean toward (B), but I'm annoyed.

[Martin Michlmayr]

* Martin Blais <bl...@furius.ca> [2018-04-10 22:07]:

> Any thoughts welcome. I lean toward (B), but I'm annoyed.

Maybe I misunderstand the problem, but with option (A) even if I don't
know what timezone they use, I should get a consistent result if I ask
the API for an exchange rate for a specific time (X) on a specific
day. Maybe I don't know what timezone X is (as you say, only one
publishes that info), but X is always going to be X from the POV of
the source (i.e. the API). (Similarly, if I in Europe ask the API for
a rate at, let's say, 14:00, and you in New York ask for the rate at
14:00, we should get the same result. And I don't mean asking at
14:00 hours but asking for the rate at 14:00)

If I understand (B) correctly, someone moving from Japan to the US
would suddenly get different results, and I think making the result
dependent on where you are is a bad idea.

--
Martin Michlmayr
http://www.cyrius.com/

[Martin Blais]

On Wed, Apr 11, 2018 at 3:23 PM, Martin Michlmayr <t...@cyrius.com> wrote:

* Martin Blais <bl...@furius.ca> [2018-04-10 22:07]:
> Any thoughts welcome. I lean toward (B), but I'm annoyed.

Maybe I misunderstand the problem, but with option (A) even if I don't
know what timezone they use, I should get a consistent result if I ask
the API for an exchange rate for a specific time (X) on a specific
day.

What do you mean by "consistent"? 

Consistently one day off if it is?

 

  Maybe I don't know what timezone X is (as you say, only one
publishes that info), but X is always going to be X from the POV of
the source (i.e. the API).  (Similarly, if I in Europe ask the API for
a rate at, let's say, 14:00, and you in New York ask for the rate at
14:00, we should get the same result.  

And I don't mean asking at
14:00 hours but asking for the rate at 14:00)

You ask at 14:00 in which timezone? US 14:00 or Europe 14:00?

If you don't know the instrument trades in the US, you ask at 14:00 in Europe timezone?

It also depends how the API expects its input. If it expects a date, it's implicitly that of the instruments.

If it expects a timestamp, there's no date, it's up to you to take the user's input date and convert it to a timestamp, assuming some timezone.

Our current API doesn't support you providing a time, just a date. 

You request for a price at a particular date.

The output is also coerced to be only a date too (the Price directive, like all other directives, supports only a date).

(I don't think adding time support to all directives would be a reasonable compromise just for this.)

If I understand (B) correctly, someone moving from Japan to the US
would suddenly get different results,

Yes, the date might differ.

 

and I think making the result
dependent on where you are is a bad idea.

I agree, but then I need the timezone of those instruments, and some instruments don't have a timezone (FX).

I don't think there's any good solution for this than to input/output time from those fetchers, but I do need to choose how to convert timestamps to dates and dates to timestamps consistently (that's why unit tests are broken if you run them from Europe right now).

--
Martin Michlmayr
http://www.cyrius.com/

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20180411192331.nnbpxcox6jepe3ro%40jirafa.cyrius.com.
For more options, visit https://groups.google.com/d/optout.

[Justus Pendleton]

I don't see "lining up with markets" as a necessary condition since that's poorly defined for dealing with other timezones anyway.

If I run bean-price when the market is open I'm going to get a quasi-real time quote that is delayed an arbitrary and possibly unknown amount of time anyway. It won't **really** be the price at 12:07pm. (This will vary depending on the source and the price fetcher, of course.)

Just using the user's timezone seems to make the most since to me. But what happens with downloaded prices when the user moves to a different timezone? Or is the timezone going to be stored with the fetched price?

[Martin Blais]

Nothing changes.

 

Or is the timezone going to be stored with the fetched price?

No, unless there's a good reason to.

That would break things (dates changing unexpectedly).

Want to keep it simple.

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/c6450704-b02f-4698-9b5f-b0dc8d699f75%40googlegroups.com.

[Martin Michlmayr]

* Martin Blais <bl...@furius.ca> [2018-04-11 16:29]:

> You ask at 14:00 in which timezone? US 14:00 or Europe 14:00?
> If you don't know the instrument trades in the US, you ask at 14:00 in
> Europe timezone?

Sorry, I thought you told the API "I want the exchange rate at 14:00
time". If so, we'd all get the same answer regardless of our local
timezone. It would be nice to know what the APIs timezone is so we
know what "14:00" really means but that doesn't matter for beancount.

If you only tell the API "I want the exchange rate now", then I think
beancount should store the time in UTC.

> Our current API doesn't support you providing a time, just a date.
> You request for a price at a particular date.

Ok, (I think) I see the problem.

If the API queried by beancount only provides 1 rate per day, we could
easily find out what a "day" is according to them by doing a query
every hour for 24 hours.

Or just pick mid-day UTC.

[Martin Blais]

On Sun, Apr 15, 2018 at 12:51 PM, Martin Michlmayr <t...@cyrius.com> wrote:

* Martin Blais <bl...@furius.ca> [2018-04-11 16:29]:
> You ask at 14:00 in which timezone? US 14:00 or Europe 14:00?
> If you don't know the instrument trades in the US, you ask at 14:00 in
> Europe timezone?

Sorry, I thought you told the API "I want the exchange rate at 14:00
time".  If so, we'd all get the same answer regardless of our local
timezone.  It would be nice to know what the APIs timezone is so we
know what "14:00" really means but that doesn't matter for beancount. 

If you only tell the API "I want the exchange rate now", then I think
beancount should store the time in UTC.

If you store the timezone, there's no ambiguity.

I made a couple decisions staring at this long enough:

- The API will accept a time, not a date. bean-price will be in charge of deciding how to convert that (and for now, using the local time, because problems highlighted in prior discussion).

- All the datetime instances have to be aware. No more usage of naive datetime instances anywhere. (I'm coming around to think that support naive datetimes in the Python datetime module API is a design flaw.)

 

> Our current API doesn't support you providing a time, just a date.
> You request for a price at a particular date.

Ok, (I think) I see the problem.

If the API queried by beancount only provides 1 rate per day, we could
easily find out what a "day" is according to them by doing a query
every hour for 24 hours.

Or just pick mid-day UTC.

--
Martin Michlmayr
http://www.cyrius.com/

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20180415165103.zaixqwtthtn35fuu%40jirafa.cyrius.com.

[Martin Blais]

Fixed:

https://bitbucket.org/blais/beancount/commits/2171ff658cc4ed507376fed1811b9e9b798fb30f

Green CI build now:

https://bitbucket.org/blais/beancount/addon/pipelines/home#!/results/17

--
You received this message because you are subscribed to the Google Groups "Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beancount+unsubscribe@googlegroups.com.
To post to this group, send email to bean...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/20180415165103.zaixqwtthtn35fuu%40jirafa.cyrius.com.