feeding python string object to d3.js csv

452 views
Skip to first unread message

greaneym

unread,
Jun 24, 2013, 4:02:35 PM6/24/13
to web...@googlegroups.com
Relating to d3.js and csv files, I'm having two problems trying to get a string object to load either

1. with d3.csv.parse or with

2. function() {return x; } 

right after the line,
g = new Dygraph( document.getElementById("div_pt"), 

that begins to generate a d3 graph.


The problem is similar to that on

but I don't understand how to make the url "have a text content".

Also, I am having a problem understanding what it means to use
the 
function() return x 
 
as the method to return data to d3.js. I want to load a string value into x
and have the graph display. When I use that and feed it {{=pt_data}} it looks like the view source output at the bottom of this page.

I am able to load a .csv file and produce a graph, but when I try to
create a string that can be loaded into function() {return x;} and it is formatted as a csv
file, the graph doesn't load. 
(disregard he timevalues).


The error console on the web inspector says,
syntax error, unexpected number, '00'
which means it is not parsing the date+time together.

Does the string have to be formatted differently from the CSV file? They
look the same to me. How should the pt_data format in the browser's 'view/source' look when it works?

For reference, I read the docs files in dygraphs-master docs/data.html as well as many other d3.js tutorials, but so far have not found an answer.
This project will also use json objects, but before that I want to get the csv as string object working first.


<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="X-UA-Compatible" content="IE=EmulateIE7; IE=EmulateIE9">
    <title>Summary</title>
    <!--[if IE]>
    <script type="text/javascript" src="../../static/dygraphjs/excanvas.js"></script>
    <![endif]-->

<script src="/enp8/static/dygraphjs/dygraph-dev.js" type="text/javascript"></script>
<script src="/enp8/static/dygraphjs/d3.v3.js" type="text/javascript"></script>

  </head>
  <body>
   <h6>pt_data is Date,ENWH
2011/09/10 00:05,0
2011/09/10 00:05,0
2011/09/10 00:05,0
2011/09/10 00:05,0
2011/09/10 00:05,32
2011/09/10 00:05,246
2011/09/10 00:05,250
2011/09/10 00:05,424
2011/09/10 00:05,500
2011/09/10 00:05,678
2011/09/10 00:05,898
2011/09/10 00:05,525
2011/09/10 00:05,400
2011/09/10 00:05,379
2011/09/10 00:05,184
2011/09/10 00:05,41
2011/09/10 00:05,0
2011/09/10 00:05,0
2011/09/10 00:05,0
2011/09/10 00:05,0 </h6>
    <p>Power Today Data:</p>
    <div id="div_pt" style="width:600px; height:300px;"></div>
    <script type="text/javascript">
       g = new Dygraph( document.getElementById("div_pt"),
         d3.csv.parse(=URL('power_today', 'index')}} ; }
    { }
     ); 


    </script>

<!--<table width="100%">-->
<table width="75%">
  <tr>
    <h2>Summary   2011-09-10 </h2>
    <td><h5><ul>Intervals:</h5></ul> 300</h5></td>
    <td><h5><ul>Power (W):</h5></ul> [0, 0, 0, 0, 32, 246, 250, 424, 500, 678, 898, 525, 400, 379, 184, 41, 0, 0, 0, 0]</h5></td>
  </tr>
  <br>
  <br>
  
</table>

<h5> JSON data formatted via python</h5>
  {u&#x27;first_interval_end_date&#x27;: u&#x27;2011-09-10T00:05-7:00&#x27;, u&#x27;production&#x27;: [0, 0, 0, 0, 32, 246, 250, 424, 500, 678, 898, 525, 400, 379, 184, 41, 0, 0, 0, 0], u&#x27;interval_length&#x27;: 300}



  </body>
</html>



Thanks a lot for any help or suggestions.

Margaret

Anthony

unread,
Jun 24, 2013, 4:12:15 PM6/24/13
to web...@googlegroups.com
Do you want to embed the data in the page, or do you want to serve it as a CSV file? If you want to embed it in the page, you might need to embed it in some Javascript as a Javascript data structure rather than as pure HTML. To serve a CSV file, see http://web2py.com/books/default/chapter/29/10#CSV (you might be able to set the content type to 'text/csv' rather than what is shown in that example). You would then provide D3 with the URL that serves the CSV file (note, it doesn't actually generate a CSV file on the server -- it just uses a cStringIO object and sends its content as the HTTP response).

Anthony

greaneym

unread,
Jun 24, 2013, 4:28:16 PM6/24/13
to web...@googlegroups.com
Anthony, thanks!   I will give the @service.csv  method a try.

greaneym

unread,
Jun 24, 2013, 5:04:06 PM6/24/13
to web...@googlegroups.com
Hi,

I tried the @service.csv example in the book and it does return values, but they are not in the form of a url but a download. I wonder if this is the expected behavior?  I am using mac os.
When I enter the path to the url including values, the system immediately returns the downloads interface and presents a file.  From reading the  manual, it looks like the service is supposed to render the values as a url?


all I did was copy and paste the example into the default.py controller and run the suggested call to the service.
What am I doing wrong? thanks


On Monday, June 24, 2013 3:12:15 PM UTC-5, Anthony wrote:

Anthony

unread,
Jun 24, 2013, 5:54:00 PM6/24/13
to web...@googlegroups.com
No, when you request the URL, it's supposed to return a CSV file. You provide the URL to D3, and it makes an Ajax request to the URL in order to retrieve the CSV file. See https://github.com/mbostock/d3/wiki/CSV. Looks like you can also pass a CSV string to D3.csv.parse(), so you might be able to just include the CSV content hidden in the HTML and extract the text node and pass it to D3. You might also consider using JSON, which will save D3 the trouble of parsing CSV.

Anthony

greaneym

unread,
Jun 24, 2013, 5:55:57 PM6/24/13
to web...@googlegroups.com
I am trying this example and get an error. Is this the way to set up the csv service
and call it? Do I need the .as_list() on the end of the select?


in default.py

def csvtab():
    rows = db(db.ptoday.id>0).select( ).as_list()
    for row in rows:
      data = row.csvdata
      ddata = ''.join(line.replace('\n', '\n') for line in data)
    return data

@service.csv
def exposed():
   mytable = csvtab()

If I want to serve csv tables as a web service, I call it from another function which
has the @service.csv decorator. Is this right?

I put this path in the uri

I tried this and get an error:
<type 'exceptions.AttributeError'> 'dict' object has no attribute 'csvdata'

thanks for any help


On Monday, June 24, 2013 3:12:15 PM UTC-5, Anthony wrote:

Anthony

unread,
Jun 24, 2013, 6:10:58 PM6/24/13
to web...@googlegroups.com
The method I first pointed out might be easier: http://web2py.com/books/default/chapter/29/10#CSV


def csvtab():
    rows = db(db.ptoday.id>0).select( ).as_list()
    for row in rows:
      data = row.csvdata

Note, .as_list() converts the Rows object to a list of dictionaries, so you cannot then do row.csvdata, because row is just a standard dict at that point (so it would be row['csvdata']). Anyway, .as_list() is not necessary here.
 
      ddata = ''.join(line.replace('\n', '\n') for line in data)
    return data

@service.csv
def exposed():
   mytable = csvtab()

The above is not how you would do a csv service in web2py. Please read the book section on this for the proper code: http://web2py.com/books/default/chapter/29/10#Remote-procedure-calls.

Anthony

greaneym

unread,
Jun 25, 2013, 10:05:27 AM6/25/13
to web...@googlegroups.com
Thanks for the answers Anthony, I am going to try to report a bug.
I am using 
web2py 
2.4.6-stable+timestamp.2013.04.06.17.37.38
the source version, 
on mac os
and 

when I copy and paste the csv as a service example from the book
section on Remote procedure calls,
to the default.py in the welcome application, and then run
the example to show the service, which I think is supposed to echo,
hello world
1,2

instead the application opens my Download application and creates a download file containing the hello world 1,2.

I am reporting this because I think the browser instead is supposed to display the text.

How do I fix this please?

The json as a web service works fine.

thanks,
Margaret

I will look on github to see where to report the possible bug.

Anthony

unread,
Jun 25, 2013, 10:21:48 AM6/25/13
to web...@googlegroups.com
Not a bug. I think @service.csv sets the content-type of the response to 'text/x-csv', so that will likely prompt the browser to do a download rather than display the text (which is typically what is desired). Have you tried it with D3 to see if it works as is?

Anthony

Andrew W

unread,
Jun 25, 2013, 3:59:48 PM6/25/13
to web...@googlegroups.com
Suggest you look at the book section "rendering a dictionary", I use the json approach with d3, and then reference the URL in the js script. Works great and should work for csv too. This approach allows you to get the file formatted in the way d3 expects it, without trying to do it in js code.

I'm still working out the best approach of embedding the d3 js code in an app, controller (perhaps a hybrid of https://github.com/mikedewar/d3py), in the view or a separate js file. ?

Anthony

unread,
Jun 25, 2013, 4:05:23 PM6/25/13
to web...@googlegroups.com
I would guess D3 would be faster if fed JSON (which is already Javascript) rather than having to first parse CSV.

Anthony

greaneym

unread,
Jun 25, 2013, 6:48:06 PM6/25/13
to web...@googlegroups.com
Anthony, I look forward to your incorporation of d3 into web2py!

Maybe you have enough d3 knowledge to help me make this work.

in my controller, powertoday.py there is a function,

def csvtab():
    #return dict(rows = db(db.ptoday.id>0).select( ))
    rows = db(db.ptoday.id>0).select( )
    for row in rows:
      data = row.csvdata
   #  d3data = data="".join(line.rstrip() for line in data) # all in one line
      ddata = ''.join(line.replace('\n', '\n') for line in data)
    return ddata

ddata is a typed as a string, according to the output of type(ddata)

then I have a view file csvtab.html
{{ =ddata }}

when viewed, ddata looks like this,
Date,ENWH 2011/09/10 00:05,0 2011/09/10 00:05,0 2011/09/10 00:05,0 2011/09/10 00:05,0 2011/09/10 00:05,32 2011/09/10 00:05,246 2011/09/10 00:05,250 2011/09/10 00:05,424 2011/09/10 00:05,500 2011/09/10 00:05,678 2011/09/10 00:05,898 2011/09/10 00:05,525 2011/09/10 00:05,400 2011/09/10 00:05,379 2011/09/10 00:05,184 2011/09/10 00:05,41 2011/09/10 00:05,0 2011/09/10 00:05,0 2011/09/10 00:05,0 2011/09/10 00:05,0

third, I have a .csv file for this view, csvtab.csv containing

{{
import cStringIO
stream=cStringIO.StringIO()
ddata.export_to_csv_file(stream)
response.headers['Content-Type']='text'
response.write(stream.getvalue(),escape=False)
}}

and when I view this file it looks like this:
Date,ENWH
2011/09/10 00:05,0
2011/09/10 00:05,0
2011/09/10 00:05,0
2011/09/10 00:05,0
2011/09/10 00:05,32
2011/09/10 00:05,246
2011/09/10 00:05,250
2011/09/10 00:05,424
2011/09/10 00:05,500
2011/09/10 00:05,678
2011/09/10 00:05,898
2011/09/10 00:05,525
2011/09/10 00:05,400
2011/09/10 00:05,379
2011/09/10 00:05,184
2011/09/10 00:05,41
2011/09/10 00:05,0
2011/09/10 00:05,0
2011/09/10 00:05,0
2011/09/10 00:05,0

Do you have any suggestions for how to get d3 to look at the csvtab.csv file?

I've tried many, many things (I want to get the csv as python solution working), including this,
      g = new Dygraph( document.getElementById("div_pt"),
           d3.csv({{=URL('powertoday','cvstab.csv')}}),  
        {}
     ); 

but that errors on the d3.csv line with "invalid flags supplied to reg exp constructor".

The URL points to a valid csv file.  I want to get a solution where a python obj is
providing the csv content. I don't want to embed in the html dom.
Have you tried this .csv method from the book?  

Any help very welcome.

margaret

Anthony

unread,
Jun 25, 2013, 7:02:52 PM6/25/13
to web...@googlegroups.com
I've tried many, many things (I want to get the csv as python solution working), including this,
      g = new Dygraph( document.getElementById("div_pt"),
           d3.csv({{=URL('powertoday','cvstab.csv')}}), 

Your URL needs to be in quotes:

d3.csv("{{=URL('powertoday','cvstab.csv')}}")

which will result in something like:

d3.csv("/yourapp/powertoday/cvstab.csv")

Anthony

Andrew W

unread,
Jun 26, 2013, 3:36:33 PM6/26/13
to web...@googlegroups.com
Not sure if its faster, but you'll write less js code if you use json, and it makes sense to use the controller to deliver the data in the expected format that is directly usable. Less js coding works for me!

Margaret, as Anthony suggests, try the json approach. I can post an example when I'm next at my computer.

By the way, Im working on a visualisation of the gluon library, once I figure out how to extract the module / class hierarchy as a json URL. Mike uses a software package for many of his examples ( http://bl.ocks.org/mbostock/4063550 )

greaneym

unread,
Jun 26, 2013, 4:03:47 PM6/26/13
to web...@googlegroups.com
This addition of quotes did nothing for me. Somehow the d3 does not load the url because I get a d3 error that there is no data found.
I can get  json to work with this,

json_data = sj.dumps(pt_data)

then in the view,
var jdata = {{=XML(response.json(pt_data))}}

      g = new Dygraph( document.getElementById("div_pt"),
           jdata,
           {} );

this loads the data with no problems and makes a graph.

BUT, what I want to be able to do is get a pythonic object for each method of putting data into d3, as described in the d3 wiki.  Except for json, I haven't been yet able to figure this out.

json is a string and so is the pt_data in my code. I need to learn more about d3 methods.  I really appreciate all the help with my questions,
and look forward to any incorporation of d3 into web2py.


Margaret

Anthony

unread,
Jun 26, 2013, 4:07:06 PM6/26/13
to web...@googlegroups.com
In your browser developer tools, can you watch the ajax request and see what web2py is returning?

greaneym

unread,
Jun 26, 2013, 5:21:16 PM6/26/13
to web...@googlegroups.com
yes, thanks, that is what I have been doing, and the web error console shows when d3 can't read the string. I didn't compile a list of the errors, but I intend to keep trying with the csv inputs and will post something when it works.  Thanks for all your help.

margaret

Anthony

unread,
Jun 26, 2013, 5:40:45 PM6/26/13
to web...@googlegroups.com
On Wednesday, June 26, 2013 5:21:16 PM UTC-4, greaneym wrote:
yes, thanks, that is what I have been doing, and the web error console shows when d3 can't read the string. I didn't compile a list of the errors, but I intend to keep trying with the csv inputs and will post something when it works.  Thanks for all your help.

Are you saying it doesn't even get as far as making the Ajax call to your URL because it can't read the URL to begin with? In that case, can you show the Javascript code that ends up in the HTML page?

Anthony 
Reply all
Reply to author
Forward
0 new messages