Time Zones

[Kevin Mendel]

Hi. Noob here. 

I have been thinking about how to make Blueflood work with different time zones, if possible.

Something like different tenants belonging in different time zones, which may not be the roll-up module's time zone.

I saw a post by Jeeyoung Kim that touched on this. "binning at UTC boundary doesn't sound too useful" looks like it sums up my issue. 

I would want hourly and daily aggregations more aligned with local time for the tenant.

So ... how far from hopeless is this? Does anyone have any approaches that they can offer?

Yes, sharding will be needed as well. I mention this because this code looks most like it assumes all slots have the same time ranges. 

[Richard Sartor]

Hi, Kevin. Thanks for your question. I don't think there's any facility yet for calculating rollups from time zones other than the server's. I'm not as familiar with the component in question as other devs, so maybe there is. Unfortunately, the dev team have all gone home for the day. I'll discuss it with them in the morning (Central Time) and we'll get a more thorough answer for you then.

Sincerely,
Richard Sartor

[Richard Sartor]

Hello again.

I've talked it over with the other devs, and the consensus is that blueflood does not support per-tenant time zones, nor time zones other than UTC.

Sincerely,
Richard Sartor

[Kevin Mendel]

Hello back to you, and thank you.

I guess I didn't explain well. I do know that blueflood doesn't support time zones as it stands.

I was wondering how hard it would be to get there.

I was wondering if anyone had any experiences supporting multiple time zones in some way. 

I was wondering if any clever ideas had come to light over time. 

In other words, I am wondering if this is short of completely and utterly impossible.

Blueflood is a great thing in my experience, but I'd hate to lose out to one hard requirement. 

Thanks for any responses. 

[Lakshmi N Kannan]

Kevin, 

I am not an active committer anymore. So I am speaking from my experience with the project which might be outdated. So truth is the code itself. 

It's a good question and one we had thought about a bit. 

>I saw a post by Jeeyoung Kim that touched on this. "binning at UTC boundary doesn't sound too useful" looks like it sums up my issue. 

I would want hourly and daily aggregations more aligned with local time for the tenant.

It indeed is a problem. Supporting a per tenant timezone would be hard. The rollup scheduler works with the single time reference (UTC IIRC) and all query modules use timestamp as UTC timestamps. 

To support a per tenant timezone, you need to make sure two things happen:

1. Rollup scheduler need to work with all time zones registered in the time zone. And then for each of the shards, iterate through the list of time zones (probably persisted in db) and invoke rollups. When performing the rollup for appropriate time zone, the slot boundary changes. It would be sweet if all persisted columns in db still used UTC for consistency but the translation happened at higher layers. This way operationally things would be much easier (speculation). 

2. The query modules which service the API calls need to get metadata about tenant, get their time zone info and then perform appropriate translation to UTC and then query. 

To make these changes, it would be good to focus on two different timezones as a start and work your way through the code. This is a serious exercise because it would touch pretty much every piece of code. As an alternate, you can write a tool that simply queries the raw metrics and performs timezone based rollups and writes to a row tenant.metric.timezone. This would be a good approach if you are new to the code base but would still like the functionality. 

Long story short, doable but tricky. I hangout on #blueflood and might be able to answer some questions depending on how much time I got on hand. 

--
You received this message because you are subscribed to the Google Groups "Blueflood Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blueflood-disc...@googlegroups.com.
Visit this group at https://groups.google.com/group/blueflood-discuss.
To view this discussion on the web visit https://groups.google.com/d/msgid/blueflood-discuss/6afc76a5-8726-4f62-8812-5adde9880cc9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Best,

Lakshmi

[Kevin Mendel]

Lakshmi, 

Hanging out on #blueflood and not seeing you. 

I am confused by your last response.

|| Rollup scheduler need to work with all time zones registered in the time zone. 

"all time zones registered in the time zone" ? Did you mean "all tenants" ?

Even then I still have no idea what this means. 

 

|| And then for each of the shards, iterate through the list of time zones (probably persisted in db) and invoke rollups.

It sounds like you think the rollups need to be handled on a time-zone-by-time-zone basis.

But I am not sure how you can find all the metric data in a specific time zone.  

By tenant ID?

On Tuesday, March 1, 2016 at 5:59:53 PM UTC-5, Kevin Mendel wrote:

[Kevin Mendel]

I understand a bit more after reviewing the information you provided in IRC.

My main question is now this:

1. Are you suggesting that each shard should have metrics for only a single timezone? Or are you suggesting that, within each shard, all metrics for a specific timezone should be rolled up, and slot states updated, before proceeding to metrics from a different time zone? I think the latter, but I wish to confirm before proceeding.

Other questions:

1. To confirm my understanding, it seems that shards are purely logical, and do not reside in different tables. Also, any ingester may produce a metric for any shard.Yes?
2. I am having trouble understanding how metrics_state table functions, since state is part of the table key, and every shard/gran/slot has both an "A" and an "X" row.
3. My modified code results in the metrics_state table having over 11,000 rows for each shard. Almost every combination is present. This feels wrong but I do not know a concrete reason why it is. Is it?

Thanks, 

Kevin 

On Tuesday, March 1, 2016 at 5:59:53 PM UTC-5, Kevin Mendel wrote:

[Greg Sharek]

Hi Kevin,

I'm new the project, but I'll take a stab at your questions.

1. Are you suggesting that each shard should have metrics for only a single timezone? Or are you suggesting that, within each shard, all metrics for a specific timezone should be rolled up, and slot states updated, before proceeding to metrics from a different time zone? I think the latter, but I wish to confirm before proceeding.

It's only my opinion, but I think the latter makes much more sense.

1. To confirm my understanding, it seems that shards are purely logical, and do not reside in different tables. Also, any ingester may produce a metric for any shard.Yes?

Yep

 

1. I am having trouble understanding how metrics_state table functions, since state is part of the table key, and every shard/gran/slot has both an "A" and an "X" row.

Yep.  I would say that each shard/gran/slot has both an "A" and an "X" version of its key, but I might not be on top of the row/key terminology.  This is to prevent locking between 2 services updating a record.

 

1. My modified code results in the metrics_state table having over 11,000 rows for each shard. Almost every combination is present. This feels wrong but I do not know a concrete reason why it is. Is it?

The metrics_static table will have a lot of entries.  I believe it should the following

# of shards * # of slots across all tables * 2 (A & X).  Depending on the TTL of the data, that can be a lot of slots.  For our setup, I believe we have

4032 - 5m slots

1008 - 20m slots

336 - 60m slots

84 - 240m slots

14 - 1440m slots

Our total is 1,401,344.

Greg

 

[Kevin Mendel]

Posting this at vinny_ly's request: 

Originally, slots were assigned to times as follows:
        return (int)((millis % (BASE_SLOTS_PER_GRANULARITY * MILLISECONDS_IN_SLOT)) / MILLISECONDS_IN_SLOT);
  
The requirement for time zone support is to group minutes into hours, and hours into days, based on local time, not UTC time.

In order to achieve timezone-based aggregation, I changed it to use local time:
        long localMillis = millis + zone.getRawOffset();
        return (int)((localMillis % (BASE_SLOTS_PER_GRANULARITY * MILLISECONDS_IN_SLOT)) / MILLISECONDS_IN_SLOT);

Then I had an epiphany.

The issue with this simplistic approach is that, for small granularities, 
it changes the slot number to be something very far away from where it would otherwise be.
Time zone offsets range from -12 hours to +14 hours.
So at 5 min granularity, the localtime slot index selected will be -144 to +168 from the UTC-based slot.
If you have many tenants in many time zones, this spreads ingested datapoints over a range of 300+ slots. 
In other words, every 5 minutes, 300+ 5m slots could be dirty.
[Correction: it would be far fewer slots, due to number of different time zones, but it would be spread over 300+ slots.]

This does not need to be. We can reduce this to 2 dirty slots, regardless of granularity,
and still group datapoints based on local time. 

We do not need to add the entire timezone offset to the UTC time.
We can add (offset % granularity.milliseconds) and achieve the same results. 
For 60m datapoints, we only need to add either +0m or +30m.
For 1440m datapoints, we only need to add +0m to +1440m (exclusive).
For 5m slots and Full slots, the slot number won't be changed at all, because timezone offsets are always multiples of 30m.
This will achieve the grouping that we need. 

Using this approach, when the slot number is calculated, it will either be 
the same slot as the UTC slot, or the next one after that. 
Which means no matter how many time zones are actively collecting metrics,
only 2 slots at a time will be dirty. 

[Kevin Mendel]

Another post, to clarify what I am trying to achieve. 

Suppose I am a user living in Austin, TX, which is in CST time zone, or GMT-06:00. 

I consider my day to be midnight to midnight local time.

But Blueflood considers my day to be 6:PM to 6:PM local time, 

because that is midnight to midnight UTC time.

That's not good for me.

First of all, that's not a natural way for me to visualize things happening in the same day. 

From my point of view, data generated at 5PM, and then other data generated later at 7PM 

should be considered happening on the same day, be rolled up into the same average. 

But they aren't.

And when I look at a datapoint that says "Friday", some of it is really from what I consider to be Thursday.

That's very confusing. 

Second of all, that's not when I expect my daily roll-ups to be complete. 

I expect them to be available shortly after midnight local time,

because my day ended at midnight.

I don't expect daily roll-ups to be ready shortly after 6PM local time. 

That's very confusing. 

I have very little conception of what UTC time means. 

Your explanation that a "day" is based on UTC time only befuddles me.

If I live in the Caracas, or India, or Adelaide, or some other places, 

my time zone is GMT+03:30 or GMT+06:30 or GMT+09:30.

So Blueflood doesn't even consider "an hour" to be what I expect it to be.

I consider my hour to be from :00 to :00 local time.

But Blueflood, using UTC time, considers an hour to be from :30 to :30 local time.

That's VERY confusing.

So what I would like to have blueflood do is group timestamps together based on my local time,

not based on UTC time. 

This IS NOT changing the timestamp on any ingested datapoint.

This IS changing how finer datapoints are aggregated into courser datapoints

specifically it is changing the timerange that the courser datapoint corresponds to.

This also changes the timestamps on rolled-up datapoints.

For example, a 1440m datapoint, using the CST timezone,

will encompass data from 18:00 to 18:00 UTC time,

which is 00:00 to 00:00 local time, 

and it will be stamped as 18:00 UTC time,

which will be displayed as 00:00 local time. 

[Kevin Mendel]

On Tuesday, March 8, 2016 at 2:11:00 PM UTC-5, Kevin Mendel wrote: 

Then I had an epiphany.

We do not need to add the entire timezone offset to the UTC time.

We can add (offset % granularity.milliseconds) and achieve the same results. 

Okay, this was a good idea.

However, it breaks the Granularity.slotFromFinerSlot() design.

Timestamps would not "line up" courser to finer as they need to.