Skip to first unread message
Andrew Student
May 25, 2018, 9:11:57 AM5/25/18
to Warp 10 users
Hi everyone,
I'm actually facing a really strange error with the Wrap10-plateform behavior.
I'm trying to feed the database with my point. But I have a issue when the timestamp is older than a certain arbitrary value.
Here is my procedure, giving a totally empty database
```
$ curl 'http://mywarpaddress:8080/api/v0/fetch?token=READ_TOKEN&selector=~.*\{\}&start=2000-01-01T00:00:00.000000Z&stop=2018-05-25T15:00:00.000000Z' -o out && echo "" && cat out
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
avalleteau@PC9168 MINGW64 ~
$
```
So, no result here, everything is normal, then, I post some data using the Wrap10 plateform:
```
1527000000000000// test{steps=100} -0.6133061918698982
1527000000000000// test{steps=100} 0.9228427144511169
1527000000000000// test{steps=100} 0.1301889411087915
1527000000000000// test{steps=100} -0.9909979074466949
1527000000000000// test{steps=100} 0.38860822449533416
1527000000000000// test{steps=100} 0.7875576742396682
1527000000000000// test{steps=100} -0.8009024241854527
1527000000000000// test{steps=100} -0.36827736651082893
1527000000000000// test{steps=100} 0.993699252984253
1527100000000000// test{steps=100} -0.6133061918698982
1527100000000000// test{steps=100} 0.9228427144511169
1527100000000000// test{steps=100} 0.1301889411087915
1527100000000000// test{steps=100} -0.9909979074466949
1527100000000000// test{steps=100} 0.38860822449533416
1527100000000000// test{steps=100} 0.7875576742396682
1527100000000000// test{steps=100} -0.8009024241854527
1527100000000000// test{steps=100} -0.36827736651082893
1527100000000000// test{steps=100} 0.993699252984253
```
Notice that the difference in the timestamps between the two parts.
So, now, I resend my fetch request:
```
avalleteau@PC9168 MINGW64 ~
$ curl 'http://mywarpaddress:8080/api/v0/fetch?token=READ_TOKEN&selector=~.*\{\}&start=2000-01-01T00:00:00.000000Z&stop=2018-05-25T15:00:00.000000Z' -o out && echo "" && cat out
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 396 100 396 0 0 396 0 0:00:01 --:--:-- 0:00:01 3633
1527100000000000// test{steps=100,.app=io.warp10.bootstrap} -0.6133061918698982
=1527100000000000// 0.9228427144511169
=1527100000000000// 0.1301889411087915
=1527100000000000// -0.9909979074466949
=1527100000000000// 0.38860822449533416
=1527100000000000// 0.7875576742396682
=1527100000000000// -0.8009024241854527
=1527100000000000// -0.36827736651082893
=1527100000000000// 0.993699252984253
avalleteau@PC9168 MINGW64 ~
$
```
Here we face the weird part, the exact same data. And only the one dating from after 1527100000000000 is registered.
Anyone already face the same situation before ?
Mathias Herberts
May 25, 2018, 10:56:12 AM5/25/18
to Warp 10 users
Hi,
your timestamps appear to be truncated in your message, they are only two different ones and Warp 10 does not allow for multiple values to be stored at the same timestamp, could you provide the actual timestamps you used?
Andrew Student
May 25, 2018, 11:18:44 AM5/25/18
to Warp 10 users
Hi Mathias !
I don't understand, the timestamp didn't look truncated to me, they are following the microsecond format.
To be sure that the error didn't come from a 'timestamp overriding' error, I've changed my UPDATE query to have a unique timestamp, this is a copy/paste of what I put in the plateform 'UPDATE' section:
1527000000000001// test{steps=100} -0.61330619186989821527000000000002// test{steps=100} 0.92284271445111691527000000000003// test{steps=100} 0.13018894110879151527000000000004// test{steps=100} -0.99099790744669491527000000000005// test{steps=100} 0.388608224495334161527000000000006// test{steps=100} 0.78755767423966821527000000000007// test{steps=100} -0.80090242418545271527000000000008// test{steps=100} -0.368277366510828931527000000000009// test{steps=100} 0.9936992529842531527100000000010// test{steps=100} -0.61330619186989821527100000000011// test{steps=100} 0.92284271445111691527100000000012// test{steps=100} 0.13018894110879151527100000000013// test{steps=100} -0.99099790744669491527100000000014// test{steps=100} 0.388608224495334161527100000000015// test{steps=100} 0.78755767423966821527100000000016// test{steps=100} -0.80090242418545271527100000000017// test{steps=100} -0.368277366510828931527100000000018// test{steps=100} 0.993699252984253So, I have a unique timestamp for each of my record. That is the result I get:
avalleteau@PC9168 MINGW64 ~$ curl 'http://mywarpaddress:8080/api/v0/fetch?token=READ_TOKEN&selector=~.*\{\}&start=2000-01-01T00:00:00.000000Z&stop=2018-05-25T15:00:00.000000Z' -o out && echo "" && cat out % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 396 100 396 0 0 396 0 0:00:01 --:--:-- 0:00:01 26400
1527100000000010// test{steps=100,.app=io.warp10.bootstrap} -0.6133061918698982=1527100000000011// 0.9228427144511169=1527100000000012// 0.1301889411087915=1527100000000013// -0.9909979074466949=1527100000000014// 0.38860822449533416=1527100000000015// 0.7875576742396682=1527100000000016// -0.8009024241854527=1527100000000017// -0.36827736651082893=1527100000000018// 0.993699252984253PS: I got the exact same problem when I try to push the example data present on: http://www.warp10.io/getting-started/
1449212583312000/48.395776999999995:-4.509472/ example.drone.gps.gts5{host=ubuntu, number=5, type=gps.state} true
1449212593312000// example.drone.captor.gts1{host=ubuntu, number=1, type=captor.time} 8
1449212593312000/48.395776999999995:-4.479472/ example.drone.gps.gts1{host=ubuntu, number=1, type=gps.state} true
1449212593312000// example.drone.captor.gts2{host=ubuntu, number=2, type=captor.time} 5
1449212593312000/48.355777:-4.489472/ example.drone.gps.gts2{host=ubuntu, number=2, type=gps.state} true
1449212593312000// example.drone.captor.gts3{host=ubuntu, number=3, type=captor.time} 6
1449212593312000/48.355777:-4.479472/ example.drone.gps.gts3{host=ubuntu, number=3, type=gps.state} true
1449212593312000// example.drone.captor.gts4{host=ubuntu, number=4, type=captor.time} 5Mathias Herberts
May 25, 2018, 11:58:09 AM5/25/18
to Warp 10 users
Can you push your data using a curl command and not the Web interface?
curl --data-binary @file.gts -H 'X-Warp10-Token: WRITE_TOKEN' http://myaddress:8080/api/v0/update
then issue a call to /fetch to see if you have the issue.
Andrew Student
May 28, 2018, 4:26:21 AM5/28/18
to Warp 10 users
The same problems occur with the curl command.
```
username@PC9168 MINGW64 ~
$ cat data.gts
1527000000000001// test{steps=100} -0.6133061918698982
1527000000000002// test{steps=100} 0.9228427144511169
1527000000000003// test{steps=100} 0.1301889411087915
1527000000000004// test{steps=100} -0.9909979074466949
1527000000000005// test{steps=100} 0.38860822449533416
1527000000000006// test{steps=100} 0.7875576742396682
1527000000000007// test{steps=100} -0.8009024241854527
1527000000000008// test{steps=100} -0.36827736651082893
1527000000000009// test{steps=100} 0.993699252984253
1527100000000001// test{steps=100} -0.6133061918698982
1527100000000002// test{steps=100} 0.9228427144511169
1527100000000003// test{steps=100} 0.1301889411087915
1527100000000004// test{steps=100} -0.9909979074466949
1527100000000005// test{steps=100} 0.38860822449533416
1527100000000006// test{steps=100} 0.7875576742396682
1527100000000007// test{steps=100} -0.8009024241854527
1527100000000008// test{steps=100} -0.36827736651082893
1527100000000009// test{steps=100} 0.993699252984253
1527200000000010// test{steps=100} -0.6133061918698982
1527200000000011// test{steps=100} 0.9228427144511169
1527200000000012// test{steps=100} 0.1301889411087915
1527200000000013// test{steps=100} -0.9909979074466949
1527200000000014// test{steps=100} 0.38860822449533416
1527200000000015// test{steps=100} 0.7875576742396682
1527200000000016// test{steps=100} -0.8009024241854527
1527200000000017// test{steps=100} -0.36827736651082893
1527300000000018// test{steps=100} 0.993699252984253
1527300000000010// test{steps=100} -0.6133061918698982
1527300000000011// test{steps=100} 0.9228427144511169
1527300000000012// test{steps=100} 0.1301889411087915
1527300000000013// test{steps=100} -0.9909979074466949
1527300000000014// test{steps=100} 0.38860822449533416
1527300000000015// test{steps=100} 0.7875576742396682
1527300000000016// test{steps=100} -0.8009024241854527
1527300000000017// test{steps=100} -0.36827736651082893
1527300000000018// test{steps=100} 0.993699252984253
username@PC9168 MINGW64 ~
$ curl "http://warpipaddr:8080/api/v0/fetch?token=READ_TOKEN&selector=test\{steps=100\}&start=1970-01-01T00:00:00.000000Z&stop=2019-01-01T15:00:00.000000Z" -o out && echo "" && cat out
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
username@PC9168 MINGW64 ~
$ curl -H "Transfer-Encoding: chunked" -H "X-Warp10-Token: WRITE_TOKEN" --data-binary "@data.gts" "http://warpipaddr:8080/api/v0/update" % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1964 0 0 100 1976 0 1976 --:--:-- --:--:-- --:--:-- 61750
username@PC9168 MINGW64 ~
$ curl "http://warpipaddr:8080/api/v0/fetch?token=READ_TOKEN&selector=test\{steps=100\}&start=1970-01-01T00:00:00.000000Z&stop=2019-01-01T15:00:00.000000Z" -o out && echo "" && cat out
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 434 100 434 0 0 434 0 0:00:01 --:--:-- 0:00:01 27125
1527300000000018// test{steps=100,.app=io.warp10.bootstrap} 0.993699252984253
=1527300000000010// -0.6133061918698982
=1527300000000011// 0.9228427144511169
=1527300000000012// 0.1301889411087915
=1527300000000013// -0.9909979074466949
=1527300000000014// 0.38860822449533416
=1527300000000015// 0.7875576742396682
=1527300000000016// -0.8009024241854527
=1527300000000017// -0.36827736651082893
=1527300000000018// 0.993699252984253
```
But I've noticed an evolution. The limit of the oldest timestamp seem to stay on a range between NOW and the oldest time. Before it was `15271XXXX` now it is `15273XXXX`.
Mathias Herberts
May 28, 2018, 6:28:51 AM5/28/18
to Warp 10 users
What version of Warp 10 are you running and on what operating system?
Andrew Student
May 28, 2018, 7:15:43 AM5/28/18
to Warp 10 users
I'm running on a : Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-127-generic x86_64) server.
The warp10 is running inside a container with the latest docker-hub available image.
From the container logs:
warp10_1 | Install Warp10warp10_1 | Bootstrap..warp10_1 | Fix ownership..warp10_1 | WARP10_HOME: /opt/warp10warp10_1 | Fix permissions..warp10_1 | WARP10_DATA_DIR: /data/warp10warp10_1 | Warp 10 config has been generated here: /opt/warp10/etc/conf-standalone.confwarp10_1 | Install Sensisionwarp10_1 | Config file does not exist - Creating it from template...warp10_1 | Init Sensision..warp10_1 | Fix permissions..warp10_1 | SENSISION_DATA_DIR: /data/sensisionwarp10_1 | /data/sensision does not exist - Creating it...warp10_1 | Sensision: generate configuration with Warp10 conf/jar..warp10_1 | WARP10_CONF: /opt/warp10-1.2.17/etc/conf-standalone.confwarp10_1 | WARP10_JAR: /opt/warp10-1.2.17/bin/warp10-1.2.17.jarwarp10_1 | Create /var/run/sensision..warp10_1 | sed: unmatched '/'warp10_1 | sed: bad option in substitution expressionwarp10_1 | sed: bad option in substitution expressionwarp10_1 | Configuration File exists - Update Quantum port (8081)warp10_1 | Launch Warp10warp10_1 | Init leveldbwarp10_1 | ___ __ ____________warp10_1 | __ | / /_____ _______________ __< /_ __ \warp10_1 | __ | /| / /_ __ `/_ ___/__ __ \ __ /_ / / /warp10_1 | __ |/ |/ / / /_/ /_ / __ /_/ / _ / / /_/ /warp10_1 | ____/|__/ \__,_/ /_/ _ .___/ /_/ \____/warp10_1 | /_/So it's the warp10 version is: 1.2.17
Mathias Herberts
May 28, 2018, 7:47:36 AM5/28/18
to Warp 10 users
Thanks for the version details.
As it turns out, the docker image for 1.2.17 launches by default an in-memory version of Warp 10 with a configuration defining two chunks of 24hours. This explains why some of your data disappeared and also why in the output you pasted there were multiple identical timestamps.
This behavior of the docker image will be reversed in future releases, in the mean time I encourage you to either run a previous version of the image or modify manually the Warp 10 configuration file to disable the in-memory version by setting in.memory to false.
Xavier MARIN
May 28, 2018, 7:54:09 AM5/28/18
to Warp 10 users
Hi,
assuming you've mounted a Docker volume on your local file-system
--volume=/opt/warp10:/data
You can change the Warp 10 configuration file in /opt/warp10/warp10/etc/conf-standalone.conf with those following lines :
```
[...]
leveldb.home = ${standalone.home}/leveldb
[...]
in.memory = false
in.memory = false
```
Andrew Student
May 30, 2018, 9:15:02 AM5/30/18
to Warp 10 users
Hi,
That was the problem.
I've refactored my warp10 service into my docker-compose.yml with this configuration:
version: '2'
services: warp10: image: warp10io/warp10:latest ports: - "8080:8080" - "8081:8081" environment: - IN_MEMORY=false volumes:
- ./warp10data:/data restart: alwaysBy explicitly setting the 'IN_MEMORY=false' variable, I the init script generate a valid configuration. Which is resolving my bug.
Thank you for the help :)
Xavier MARIN
May 30, 2018, 9:16:06 AM5/30/18
to Warp 10 users
You are welcome :)
Reply all
Reply to author
Forward
0 new messages