best way importing and handling data? MySQL, PHP, D3, etc

[Nate Williams]

I am really excited about implementing D3JS to visualize one of my personal projects. I am a complete newbie

Question

I am trying to figure out the best way to import my data and in what format?

At what level should most of the data manipulation happen? MySQL, PHP, or D3JS?

Currently my app is just PHP/MySQL/CSS with no D3JS

Currently, I am pulling out all the data with simple MySQL queries (all columns in all queried rows) and then organizing it using PHP Arrays / custom functions etc (word counts,related queries, etc)

I know I could do lots of this by just writing MySQL statements (counts,etc), but that feels limited because I have to hit the database for each update.

So I guess my question is should I just pull out all the columns in a row in JSON format and then us D3JS to do all the counting or is it better do this to the data in PHP or MySQL before you import it into D3JS?

Below is the example of what type of data I am dealing with

each row has the following columns: 

• date - date/time
• rating - int (1-5)
• description - string
• categories - string (words separated by comma then end up in arrays)
• emotions - string (words separated by comma then end up in arrays)
  

[Shreeram Kushwaha]

Have a look over this link

http://www.jeromecukier.net/blog/2012/01/02/using-d3-with-a-mysql-database/

Not necessarily to go for a JSON format only, d3 supports many other formats as well.

And if possible if you describe actually what you want to do, because I am not able to get what you are asking.

--
You received this message because you are subscribed to the Google Groups "d3-js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to d3-js+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Regards,

(Voice  +91-7849069144)
( Shreeram Kushwaha )
Software Engineer

Samsung R&D Institute India - Bangalore

[Nate Williams]

Thanks Shreeram I really appreciate the link

My question is how much should I process the data before I import it into D3js

For example 

say I want to create a graph similar to a  word cloud where I need to know the word count of each word 

I could get this number by array functions in PHP and then import the data into d3 but that would require a server it for each view

I thought it might be more dynamic and efficient if D3js could handle this type of stuff?

How would you go about solving this problem?

[nick]

On Tuesday, September 17, 2013 12:16:43 PM UTC-4, Nate Williams wrote:

Thanks Shreeram I really appreciate the link

My question is how much should I process the data before I import it into D3js

For example 

say I want to create a graph similar to a  word cloud where I need to know the word count of each word 

I could get this number by array functions in PHP and then import the data into d3 but that would require a server it for each view

I thought it might be more dynamic and efficient if D3js could handle this type of stuff?

How would you go about solving this problem?

Like anything else where you are making a decision between front- and back-end rendering, you have to consider the pros and cons, and what the marginal effect on the end user would be.

foo foo bar

If the corpus of words is small, just do it front-end if your backend doesn't care one way or the other.

Actually, the d3.nest operator is great for that kind of thing, if you did want to do it on the frontend:

• docs: https://github.com/mbostock/d3/wiki/Arrays#-nest
• examples: http://bl.ocks.org/phoebebright/raw/3176159/

As mentioned in the docs, this is kind of like SQL GROUP BY.

If the corpus of words you are counting is huge and static (several books worth)...

Midway upon the journey of our life
... the rest of dante's _inferno_ ...
Thence we came forth to rebehold the stars.
... and some more books
 

 and you wouldn't have the raw data around on the client for another reason (like showing it, searching it, or allowing links to it, or something), you'll be almost always better off doing that on the backend... instead of passing megabytes of data, you'd be passing only a few kilobytes. Under most conditions, this will be preferable. If they need the text anyway, d3 (or other libraries) can do a lot of the heavy lifting for you, leaving you with some data to put into d3.

As to how you represent the data: d3 really likes arrays. In fact, if you have something you want to do for each of the properties in an object, you'll have to turn it into an array, first with, for example, d3.entries. So, in the case of your example above, instead of passing:

{"data": {foo: 2, bar: 1}}

you may want to pass:

{"data": [{word: "foo", "count": 2}, {word: "bar", "count": 1}]}

But this it's so much longer! This is true, but will give you more options down the road: once you want to store more than a count, you'll need a different structure than a straight object/dictionary/hash/map.

However, if the overhead of the JSON "schema" is really big, and if your data really is rows of one table coming off of a database, sometimes CSV or TSV can be a great way to go, for which d3.csv works great... but make sure you are using a library to generate your CSV server-side, or you could get some surprises with newlines, quotes, etc.!

word,count
foo,2
bar,1

Probably the best way to get a feel for how people structure their data is to just go start poking around on bl.ocks.org, the d3 gallery (on the wiki), etc. Good luck!

[Nate Williams]

Nick thanks so much for helping me with this. 

Your links and examples have pointed me in the right direction and I really appreciate it.