Sample data not showing/graphing in grafana, but data results in templates shows

[Billy Teves]

Hi Everyone,

This time, I was able to use the pushgateway properly, I can now push data more than 500 per request. I did some data randomizer to insert data. This time, I have sample data in my mongodb and try to push them to prometheus db (the actual data, since this is the data that we will be pushing). The data is very old, about 3 months ago. I have successfully pushed all the data to pushgateway.

So far so good. I also tried to create a template in grafana. Which looks like this:

$label1 = label_values(mysample_metrics, sample_label1)     //include all option set custom all value to ".*"
$label2 = label_values(mysample_metrics, sample_label2)     //include all option set custom all value to ".*"

And it shows all the unique data inside the sample_label1 and sample_label2. Then I tried to do a query in the query editor of grafana:

count(mysample_metrics{sample_label1=~"$label1",sample_label2=~"$label2"})

But there is no data to be graph. I also set the range up to now-90days even now-120days just to make sure that the data will show since they are all 3 months old. Still not showing anything. I tried to show all the data in /graph or prometheus and I still get nothing. Since I didn't get any result other than the template data result, I tried to delete the contents of the metrics including the data in pushgateway. Upon deleting the data using the delete API, I was able to delete more than 700+.  I am not sure what's going on on my data that was inserted. Please advise.

Note: The data are in UTC then converted to PDT once inserted in prometheus. (Ex 2016-03-16 17:16:11 -0700 PDT ) 
Not sure how, I think prometheus did the changes depending on the machine's timezone

Regards,
Billy

[Billy Teves]

hi again.

I tried to changed all the data's timestamp to the current time upon insertion and refresh my grafana dashboard. Data is now showing. Is there something wrong if the data is 3months old? Have you guys also encountered this?

[Björn Rabenstein]

Just a few points:

- There is no clean way to import historical data as of now, cf.
https://github.com/prometheus/prometheus/issues/535 . What you are
doing is a hack, and all kind of things can go wrong.

- Default retention time is 15d. You have to change the flag
-storage.local.retention to something else, otherwise your older data
points will not show and gradually get deleted.

- Prometheus has no notion of time zones. Everything is UTC.

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[Billy Teves]

Hi Björn,

I see and noted on this. I wasn't aware on #535.

- Default retention time is 15d. You have to change the flag
-storage.local.retention to something else, otherwise your older data
points will not show and gradually get deleted.

Is there a way to set the data retention per metric?
Something like in InfluxDB where you can set specific data retention policy per database.

- Prometheus has no notion of time zones. Everything is UTC.

Noted also on this. Since I am using a pushgateway, I noticed that each datum's timestamp is in e.g PDT which is my machine's timezone is set to PDT.
The data I am passing in pushgateway is from MongoDB and has a defined timestamp. The timestamp (e.g isodate 2016-03-17T02:06:52.0Z) is returned to date->sec in PHP. To show properly in pushgateway, I have to append "000", if not it shows something like this : 1970-01-17 13:02:09.34 -0800 PST. But wiith . "000": 2016-03-16 04:55:40 -0700 PDT  . I just really wanted to make sure that when prometheus pulls the data from pushgateway it will be in UTC. Kindly correct me if there is something wrong on what I did in the timestamp.
 
Again, thank you for helping me out.

[Björn Rabenstein]

On 25 May 2016 at 03:40, Billy Teves <billy...@gmail.com> wrote:
> Is there a way to set the data retention per metric?

No. At least not yet. I believe this is a a feature that will happen
eventually, but there are concerns among the Prometheus developers how
to do it in a sane way.

> The data I am passing in pushgateway is from MongoDB and has a defined
> timestamp. The timestamp (e.g isodate 2016-03-17T02:06:52.0Z) is returned to
> date->sec in PHP. To show properly in pushgateway, I have to append "000",
> if not it shows something like this : 1970-01-17 13:02:09.34 -0800 PST. But
> wiith . "000": 2016-03-16 04:55:40 -0700 PDT . I just really wanted to make
> sure that when prometheus pulls the data from pushgateway it will be in UTC.
> Kindly correct me if there is something wrong on what I did in the
> timestamp.

The Prometheus timestamp is milliseconds since epoch, i.e. 1970-01-01
00:00:00 UTC, excluding leap seconds.

[Björn Rabenstein]

On 27 May 2016 at 06:57, Billy Teves <billy...@gmail.com> wrote:
> Regarding the bulk import of historical data again Björn, for now, is there
> a way to do this?

There is no sane way to do this. People came up with hacky solutions
but those are essentially setups to shoot yourself in the foot.