python+d3

[appy]

HI,

Can anybody suggest me some good tutorial for learning python with d3?I want to do data processing in python(i.e  backend) and then do the front end visualization with d3,css,html,javascript.

I look forward for suggestions.

Thanks in advance.

[Skip Montanaro]

> Can anybody suggest me some good tutorial for learning python with d3?

I've no tutorial to suggest. What little work I've done with d3 (all
job-related, so I can't show you an example) uses Python on the
server. The only real issue you have to solve is the data interchange
format. D3 supports both CSV and JSON out of the box. I started with
CSV because my original data on disk was in that format. I eventually
switched to JSON because it gave me some more flexibility (mostly the
ability to tack on a bit of metadata). I just use a very thin wrapper
around Python's json module and the third-party jsonext module to
encode my data (mostly Pandas dataframes).

Skip

[Thomas Murphy]

Hi,

I'm not sure about an all in one tutorial, but I think you have the right idea stack wise. Here are some thoughts:

• The gold standard for data wrangling/munging in Python is Pandas/Numpy with the utility csv & json modules for read/write and urllib2 for web crawling.
• This doesn't require your server to be in Python. It can follow the CGI pattern and your server can be in any language.(I personally use NodeJS a lot for this type of command line execution) If you wanted to stick pure Python though you could certainly use Django or Flask
• Since you control the front and back end I would recommend defining very specific RESTful verbs so that your client-side D3 can be as dumb as possible and not have to do any processing of the data before it draws.

--
You received this message because you are subscribed to the Google Groups "d3-js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to d3-js+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[appy]

I am newbie in python.Is it possible to process/compute data (like twitter data) from its api and show it it using d3,css without using python?

[Thomas Murphy]

Absolutely. Javascript is not a great language for vectorized data operations(which is how Pandas/Numpy think), but you can absolutely perform data-munging in Javascript. 

I would highly recommend utilizing the loDash library for a lot of these operations. It covers most set theory operations, the map/reduce paradigm, and a lot of relational style joins.

That said, heavy-duty data lifting should absolutely be done on the back end with the CGI paradigm, but I wouldn't want to deny you all the fun of building your first step MVPs with D3 and real data.

Best,

Thomas

--

[nick]

Good insights thus far.

+1 to getting on the scientific python track, after you get the hang of some of the standard libraries: numpy, pandas, scipy, etc. are probably the best way to leverage the tremendous community effort that is modern python, and really sets it apart from other general purpose languages.

If you just want to start hacking on data in a browser, you can use the IPython notebook:

http://ipython.org/notebook.html

 

With d3:

http://nbviewer.ipython.org/github/dboyliao/cookbook-code/blob/master/notebooks/chapter06_viz/04_d3.ipynb

If you have definitely decided you are going to build a web app, for "first steps" kind of web stuff, I like Flask:

https://realpython.com/blog/python/web-development-with-flask-fetching-data-with-requests/

It's no less powerful, but is less magical, than Django.

Then you'll do some REST, probably. Nothing wrong with hacking the headers and json.dumps'ing some stuff, but this is one of those places where a little convention will help a lot. Here's my current favorite for minimal REST:

https://github.com/toastdriven/restless

If you think you are going to get in to streaming data, tornado will eventually become a contender, and is not that much worse to start with:

http://danielfrg.com/blog/2013/02/05/d3-backbone-tornado-histogram-csv/

As an aside, having an evented data model like Backbone's on the frontend, while initially more complicated, will help substantially in thinking about your data when you want to actually start changing data on the backend, say, in response to a user click. On certain projects, I have ignored Backbone.View, and just used the Backbone.Model and Backbone.Collection classes, and used d3 as the view layer.

Have fun, let us know how it goes!

[Thomas Murphy]

Well, now that Nick's gone and dropped Backbone into the picture I have to throw my hat in and mention the use of AngularJS directives for building reusable data visualization components whose params you can declare and tweak directly from the view via attributes on element tags.

--

[Kyran Dale]

Hi,

I've just started writing a book for O'Reilly on this very subject (that would be a shameless plug caveat if there was anything tangible to plug) so it's nice to see an 'interested' datapoint. 

Some great points already made, I'll just add a few.

First off, it's worth emphasizing the for browser-based visualization Javascript writes its own ticket, often D3 shaped. So there's no alternative to working with it. Having said that, there are some rather exciting initiatives coming from the Python world to limit the amount of JS manipulation needed. The idea here is to process the data and craft the visualization in Python and then have a working JS-based viz pre-assembled. Pros and cons to this, but it can be mighty convenient if all you need is a fairly standard chart with limited interactivity. Those of us used to the expressive power of native D3 are probably going to find this a constraint too far, but it definitely has its use-cases. 

Here's something to whet your appetite. An Ipython notebook demonstrating the use of Plotly to create D3-based charts interactively as part of a Python processing workflow:

http://nbviewer.ipython.org/gist/msund/7ac1203ded66fe8134cc

Note how flexible plotly is - it will take a chart produced by any number of the Python charting libraries (often matplotlib based) and, with a single command, produce its JS counterpart. As mentioned, the UI is generic, but it's a very provocative demo. Of greater appeal to those who want to avoid having to learn JS/D3 but have a web-based charting itch to scratch. I'll mention bokeh here, as it's a bit more ambitious than the others, aiming for a grammar-of-graphicsy approach, which means more expressiveness. Canvas-based and not D3 but intriguing with some very cool demos. 

Plotly, matplotlibD3 et. al. are about as close as you're going to get to a 'python with D3' solution. To enjoy the full expressive power of D3 means a separation of powers, D3 on the visualization end, Python on the data scraping, munging, processing and delivery end. The latter involves assembling a tool-chain, and the key libraries there are, in my opinion:

Scraping: 

BeautifulSoup, lxml and the powerhouse Scrapy. For JS-locked data selenium and robobrowser. A nice intro here: http://blog.miguelgrinberg.com/post/easy-web-scraping-with-python and here http://brianabelson.com/open-news/2013/12/17/scrape-the-gibson.html

Cleaning+processing

 As mentioned, Pandas is the 500lb gorilla here. Well worth the effort getting to grips with. It's based on Python's numpy library so has the chops to do serious data-processing. Ipython Notebooks are a fantastic way to learn this stuff - it doesn't get a lot better than live, interactive code. Here are some Pandas notebook tutorials: https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks#pandas-for-data-analysis

Wes Kinney, the guy who wrote Pandas, has a useful book based on this subject with some accompanying Ipython notebooks: http://www.amazon.com/Python-Data-Analysis-Wrangling-IPython/dp/1449319793

To fully exploit Pandas you need to 'get' Numpy arrays and their syntax.

Delivery

To get your processed data to the browser (and I'd recommend using something like Pandas to get it as close to required form as possible) Flask is a great tool. One of Python's great plus points over other processing solutions like R is that it's a fully-fledged programming language with libraries for pretty much anything. With Flask you can roll a dev-server to deliver dynamic data in a few lines. And it's quite capable of doing production too. I'd avoid the bigger frameworks like Django unless you really need the things they're offering. For single-page apps, dynamic-data-driven visualizations etc., Flask is more than enough and a lot simpler to use: http://flask.pocoo.org/

And just to emphasise how wonderful Ipython Notebooks are for learning and implementing this stuff. You can interactively process and visualize your data, getting a feel for it before sending the right stuff to D3. It's a great workflow and tbh., notebooks almost justify the use of Python on their own. 

Hope that helps a bit - it's a big subject but I think Python + Javascript is the sweetspot for dataviz right now. It seems that every week a new, humongous library is being added to the Python canon and, as mentioned, JS is non-negotiable. Lucky for us that D3 et. al. give it the power to justify all that back-end data wrangling. 

Kyran 

[Thomas Murphy]

One point that Kyran made me think about is that if you're making static visualizations based on date passed from the client to the backend, you could actually run D3 on a node server on the backend and convert its output to JPG, then sending it back to the client.

--

[appy]

Hi all,

I want to develop now clone of application like facebook.Is it possible to implement Recommendation algo using python with d3 and visualize facebook friends and their likes?.I find learning Django/Flask is time consuming and not easy(in my opinion).

looking forward for any suggestions.

On Thursday, December 11, 2014 5:23:58 PM UTC+5:30, appy wrote:

[Russell Jurney]

We do Python/D3 apps in my book, Agile Data Science. Check it out.

--

You received this message because you are subscribed to the Google Groups "d3-js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to d3-js+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com

[Dan Frank]

I'm also interested in interactive d3 plotting from python. My current approach is using python to write the html and using pandas dataframes (which have a very convenient to_json method) 

Here's an example https://gist.github.com/danfrankj/ba7a9268747690250304

Would be great to hear what you guys think about this kind of approach.

On Thursday, January 1, 2015 at 3:31:24 AM UTC-8, rjurney wrote:

We do Python/D3 apps in my book, Agile Data Science. Check it out.

On Thursday, December 11, 2014, appy <sangee...@gmail.com> wrote:

HI,

Can anybody suggest me some good tutorial for learning python with d3?I want to do data processing in python(i.e  backend) and then do the front end visualization with d3,css,html,javascript.

I look forward for suggestions.

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "d3-js" group.

To unsubscribe from this group and stop receiving emails from it, send an email to d3-js+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Augustine Koh]

I saw this entire thread in a digest email a few weeks back, and haven't found the time to come round to this until now. But now I'm able to make a suggestion, here particularly having begun working on a data analysis project where I'm using a Python library that is called Bokeh http://bokeh.pydata.org/en/latest/

It lets you achieve d3 style plots without even writing d3, and that suits your python data processing backend very well. You can consider it for your project, and that will make for a nice flow that is entirely in python.

[nick]

Yeah, bokeh is indeed pretty amazing, and works great with the ipython notebook.

Mpld3 is another one:
http://mpld3.github.io

It's an implementation of the matplotlib api in d3.

You do write embedded JavaScript, but it is still quiet convenient, as you keep all of the matplotlib ui metaphors and controls.