web2py as Grafana endpoint

[Dave S]

I'm trying to implement a simple json endpoint for Grafana.  Does anyone here have experience with doing this?

<URL:https://github.com/grafana/simple-json-datasource/blob/master/README.md>

I'm looking at using @service.json, but I'm not sure how to get the body on a /query.

(The sample fake-simple-json-datasource is a server written in javascript!  It doesn't make clear if the /query is a POST or a GET, and if it's a GET how the body gets done.)

/dps

[Niphlod]

if the request comes in either as a POST or as a PUT it has a body (since all examples report a faily nested query structure, I'd say it's the case vs a simple querystring in a GET)

If the body carries the correct content-encoding, it's automatically parsed into request.vars. You can still read the raw body as usual in request.body .

[Dave S]

On Thursday, November 17, 2016 at 2:38:57 PM UTC-8, Niphlod wrote:

if the request comes in either as a POST or as a PUT it has a body (since all examples report a faily nested query structure, I'd say it's the case vs a simple querystring in a GET)

If the body carries the correct content-encoding, it's automatically parsed into request.vars. You can still read the raw body as usual in request.body .

Thanks, that helped me to know what to look for.  I got the Grafana stuff installed, it does the fake-json-source ok, and with FF developer tools  I'm seeing the traffic to my nascent endpoint.

When configuring the datasource URL, it does a GET to the base URL (the one just for "checking in", return an HTTP 200; I put a little status json into the response body just for grins).  When you configure a panel (the youtube videos they provide are important here) and select a metric , it will do an OPTIONS and then a POST.

I had put some CORS headers into my response, because FF is seeing me return data when it accesses the Gravana server.   [Sidebar question:  does the book cover anything about valid CORS?  I didn't find it if it's there; fortunately, there were some good posts in the GG archive.  The book does have little bits about providing protection against against black-hat CORS.]  The request to me seems to come from the client JS -- since it shows in the network panel -- but FF gives CORS errors if I don't have the headers.

So now I just have to make a good response to the metric requests.

/dps

[Dave S]

On Thursday, November 17, 2016 at 11:09:27 PM UTC-8, Dave S wrote:

On Thursday, November 17, 2016 at 2:38:57 PM UTC-8, Niphlod wrote:

if the request comes in either as a POST or as a PUT it has a body (since all examples report a faily nested query structure, I'd say it's the case vs a simple querystring in a GET)

If the body carries the correct content-encoding, it's automatically parsed into request.vars. You can still read the raw body as usual in request.body .

Thanks, that helped me to know what to look for.  I got the Grafana stuff installed, it does the fake-json-source ok, and with FF developer tools  I'm seeing the traffic to my nascent endpoint.

When configuring the datasource URL, it does a GET to the base URL (the one just for "checking in", return an HTTP 200; I put a little status json into the response body just for grins).  When you configure a panel (the youtube videos they provide are important here) and select a metric , it will do an OPTIONS and then a POST.

I had put some CORS headers into my response, because FF is seeing me return data when it accesses the Gravana server.   [Sidebar question:  does the book cover anything about valid CORS?  I didn't find it if it's there; fortunately, there were some good posts in the GG archive.  The book does have little bits about providing protection against against black-hat CORS.]  The request to me seems to come from the client JS -- since it shows in the network panel -- but FF gives CORS errors if I don't have the headers.

So now I just have to make a good response to the metric requests.

/dps

 

I may have to force the response to ASCII, and I don't remember how to do that.  My JSON is currently being decoded as 36 or so objects, each a single character, while the fake source response is being decoded as a single compound object (per the FF devtool network tab), and the Grafana display is saying "datapoints is undefined".

My  content header line says "application/json;encoding/utf-8", theirs says only "application/json".

[Dave S]

Any suggestions on how to force the response to ASCII instead of UFT-8?

/dps

[Massimo Di Pierro]

response from web2py controller? web2py returns bytes. It is up to you to put utf8 or ascii in there. web2y does not care.

[Dave S]

On Monday, November 28, 2016 at 3:23:59 PM UTC-8, Massimo Di Pierro wrote:

response from web2py controller? web2py returns bytes. It is up to you to put utf8 or ascii in there. web2y does not care.

Okay, I overrode the Content-Type header to remove the "charset: utf-8", and I set enforce_ascii in the simplejson.dumps(), but I still have 50 or so single-char objects rather than 1 hierarchical object.  I also tried applying str() before the dumps(), without any improvement.  Since I'm using FF's network tab, I can't see if there is any difference in the packets, only what FF decoded the packets as.  

This is a Ubuntu system (a VM); can I use tcpdump on localhost traffic?  I'm using the Rocket server on the web2py side; the fake-source from the Grafana people is using a js server.  (I'm not sure if Grafana itself is using a js server, but it does have an internal server to keep track of settings and such.)

/dps

[Dave S]

On Monday, November 28, 2016 at 11:11:48 PM UTC-8, Dave S wrote:

On Monday, November 28, 2016 at 3:23:59 PM UTC-8, Massimo Di Pierro wrote:

response from web2py controller? web2py returns bytes. It is up to you to put utf8 or ascii in there. web2y does not care.

Okay, I overrode the Content-Type header to remove the "charset: utf-8", and I set enforce_ascii in the simplejson.dumps(), but I still have 50 or so single-char objects rather than 1 hierarchical object.  I also tried applying str() before the dumps(), without any improvement.  Since I'm using FF's network tab, I can't see if there is any difference in the packets, only what FF decoded the packets as.  

Any suggestions here?

 

This is a Ubuntu system (a VM); can I use tcpdump on localhost traffic?  I'm using the Rocket server on the web2py side; the fake-source from the Grafana people is using a js server.  (I'm not sure if Grafana itself is using a js server, but it does have an internal server to keep track of settings and such.)

I'm going to guess a node.js stack.  Seems to use sqlite for the settings DB, although that probably isn't important to my problem.

[Dave S]

On Thursday, December 1, 2016 at 12:15:50 AM UTC-8, Dave S wrote:

On Monday, November 28, 2016 at 11:11:48 PM UTC-8, Dave S wrote:

On Monday, November 28, 2016 at 3:23:59 PM UTC-8, Massimo Di Pierro wrote:

response from web2py controller? web2py returns bytes. It is up to you to put utf8 or ascii in there. web2y does not care.

Okay, I overrode the Content-Type header to remove the "charset: utf-8", and I set enforce_ascii in the simplejson.dumps(), but I still have 50 or so single-char objects rather than 1 hierarchical object.  I also tried applying str() before the dumps(), without any improvement.  Since I'm using FF's network tab, I can't see if there is any difference in the packets, only what FF decoded the packets as.  

Any suggestions here?

Okay, moving my code to a different server lets me do tcpdump, and I see that I am indeed shipping ASCII.  What I can also see is that there are quotes (in the tcp packet body) around my json string, both for the hand-crafted json and the responses done with json.dumps() ... is this expected?

I'm not sure how to move the "fake data source" server elsewhere, even though I think it's node.js ... I've only watched such a critter being set up (once, 3 nights ago, as part of a MEAN stack demo), and not done it myself..

[Dave S]

On Friday, December 2, 2016 at 12:10:55 PM UTC-8, Dave S wrote:

On Thursday, December 1, 2016 at 12:15:50 AM UTC-8, Dave S wrote:

On Monday, November 28, 2016 at 11:11:48 PM UTC-8, Dave S wrote:

On Monday, November 28, 2016 at 3:23:59 PM UTC-8, Massimo Di Pierro wrote:

response from web2py controller? web2py returns bytes. It is up to you to put utf8 or ascii in there. web2y does not care.

Okay, I overrode the Content-Type header to remove the "charset: utf-8", and I set enforce_ascii in the simplejson.dumps(), but I still have 50 or so single-char objects rather than 1 hierarchical object.  I also tried applying str() before the dumps(), without any improvement.  Since I'm using FF's network tab, I can't see if there is any difference in the packets, only what FF decoded the packets as.  

Any suggestions here?

Okay, moving my code to a different server lets me do tcpdump, and I see that I am indeed shipping ASCII.  What I can also see is that there are quotes (in the tcp packet body) around my json string, both for the hand-crafted json and the responses done with json.dumps() ... is this expected?

I have access to a java-based server that I can get json from, and it doesn't seem to have the whole response wrapped in quotes.

[Dave S]

On Friday, December 2, 2016 at 5:53:46 PM UTC-8, Dave S wrote:

On Friday, December 2, 2016 at 12:10:55 PM UTC-8, Dave S wrote:

On Thursday, December 1, 2016 at 12:15:50 AM UTC-8, Dave S wrote:

On Monday, November 28, 2016 at 11:11:48 PM UTC-8, Dave S wrote:

On Monday, November 28, 2016 at 3:23:59 PM UTC-8, Massimo Di Pierro wrote:

response from web2py controller? web2py returns bytes. It is up to you to put utf8 or ascii in there. web2y does not care.

Okay, I overrode the Content-Type header to remove the "charset: utf-8", and I set enforce_ascii in the simplejson.dumps(), but I still have 50 or so single-char objects rather than 1 hierarchical object.  I also tried applying str() before the dumps(), without any improvement.  Since I'm using FF's network tab, I can't see if there is any difference in the packets, only what FF decoded the packets as.  

Any suggestions here?

Okay, moving my code to a different server lets me do tcpdump, and I see that I am indeed shipping ASCII.  What I can also see is that there are quotes (in the tcp packet body) around my json string, both for the hand-crafted json and the responses done with json.dumps() ... is this expected?

Because this is a service, I'm not doing anything explicit about a view file.  I have it decorated with @service.json.

 

I have access to a java-based server that I can get json from, and it doesn't seem to have the whole response wrapped in quotes.

Any suggestions here?  I seem to be spinning my wheels.

/dps

[Dave S]

So far, it's been easier to pick up node.js + express than to figure out what I'm doing wrong with my json, so that seems to be where the project is headed.

/dps