Noob: CSV import with d3.csv

[David Brown]

This below is a mess but it is simply wanting data.csv put to array dataset.

data.csv ==

X,Y
5,20
480,90
250,50
100,33
330,95
410,12
475,44
25,67
85,21
220,88
600,150

I found an example from 2013 but it's unclear why this doesn't just work: https://stackoverflow.com/questions/14986435/d3-csv-data-loading

I've tried alsorts of suggested alternates.. including parse and parseRows; forEach and map. I've looked at the documentation too but it's not making sense for something so simple.

The code works, if it's hardcoded and not from .csv.. but why!?

            var dataset = [];                                            //Initialize empty array

// Hardcoded option works
//          var dataset = [
//                          [5, 20], [480, 90], [250, 50], [100, 33], [330, 95],
//                          [410, 12], [475, 44], [25, 67], [85, 21], [220, 88],
//                          [600, 150]
//                        ];
 

// Import from csv fails with either Attempt1 or Attempt2
d3.csv("./data.csv", function(data) {

alert(data); // alert gives: [object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

//ATTEMPT1
//  data.forEach(function(d) {  
// d.X = +d.X;
// d.Y = +d.Y;
//};

//ATTEMPT2
//function(data) {
//  dataset = data.map(function(d) { return [ +d["X"], +d["Y"] ]; });
//};
 
   dataset = data;
 
alert(dataset);
   console.log(dataset);
 
});

???

[Dominikus Baur]

Hi David,

since your first alert actually displays something it looks like the file is loaded by d3 - often the major obstacle with csv, so congrats on that ;)

As for the parsing: Attempt1 probably loops through the data array even though a closing parenthesis is missing but doesn't return anything. So no wonder that dataset is still empty afterwards. Using something like

data.forEach(function(d) { 

   dataset.push([ +d.X, +d.Y ];

});

should do it.

Alternatively: Attempt2 is even closer to something that works. The problem is - I guess - that you're not calling the function you're defining, so nothing ever happens to the variables. Lose the function and brackets around the dataset = ... line and it should also work.

Hope it helps! JS is a real pain in the beginning (and in the middle, and at the end ;))

D

[David Brown]

Thanks.. I've learned that Firebug is useful for pointing out more obvious syntax errors.

So, I got now it working but with another oddity.. It only works if I have the alert which is a nuisance.
The console.log(dataset), which has been suggested as required in other examples, has no effect!?

d3.csv("./datatest.csv", function(data) {
data.forEach(function(d) { dataset.push([ +d.x, +d.y ])})
//console.log(dataset)
});
alert(dataset)

If d3 wasn't so pretty, I'd leave javascript well alone!..

[Curran]

Hi David,

I believe the crux of your issue is that d3.csv creates an array of objects, but what you are expecting is an array of arrays. Here is a code example that demonstrates d3.csv.parse.

var csvString = [

  "X,Y",

  "5,20",

  "480,90",

  "250,50"

].join("\n");

var data = d3.csv.parse(csvString);

Here, the contents of "data" is an array of objects. This is what your Alert was showing you with "[object Object],[object Object],..." To see the structure of the objects, you can serialize the array to JSON and print that out to the console, like this:

console.log(JSON.stringify(data, null, 2));

This will print the following to the JavaScript debugging console (which is part of Firebug and also is in the Chrome dev tools):

[

  {

    "X": "5",

    "Y": "20"

  },

  {

    "X": "480",

    "Y": "90"

  },

  {

    "X": "250",

    "Y": "50"

  }

]

This is the typical way that data tables are represented when using D3. Using arrays of arrays is not as common.

If you really do want to use arrays of arrays rather than arrays of objects, you'll need the following code:

var dataset = [];

data.forEach(function(d) { dataset.push([ +d.X, +d.Y ]); });

Notice that "X" and "Y" are used, not "x" and "y". Probably your previous code with lower case "x" and "y" was breaking, because those lower case properties did not exist on your objects. I would recommend the Chrome development tools for catching these kinds of errors.

Here's a video tutorial that does in depth on parsing CSV.

Best regards,

Curran

[David Brown]

So, I'm tired and I'll take another look tomorrow but this is not making a lot of sense..

Below are three console logs.
The first is as expected = [[5, 20], [480, 90], [250, 50]]
The second is blank [] but why?
The third is again correct = [[5, 20], [480, 90], [250, 50]] but only if the alert actions and I want to remove that alert.

Do Javascript variable have lifetimes like Rust?.. I can't see that is the answer as the alert is just odd.

var dataset = [];

d3.csv("./datatest.csv", function(data) {
data.forEach(function(d) { dataset.push([ +d.x, +d.y ]) })
console.log(dataset)
});
console.log(dataset)
alert(dataset)
console.log(dataset)

The full code is below if you want to take a closer look:
csv == datatest.csv

x,y
5,20
480,90
250,50

The principal code is a hack of a d3 scatter example .. so only real change is the attempt to put csv data in there..

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8">
        <title>D3: Transitioning points to randomized values, plus rescaled axes!</title>
        <script type="text/javascript" src="../d3/d3.js"></script>
        <style type="text/css">
             
            .axis path,
            .axis line {
                fill: none;
                stroke: black;
                shape-rendering: crispEdges;
            }
             
            .axis text {
                font-family: sans-serif;
                font-size: 11px;
            }
 
        </style>
    </head>
    <body>
   
        <p>Click on this text to update the chart with new data values as many times as you like!</p>
   
        <script type="text/javascript">
 
            //Width and height
            var w = 600;
            var h = 300;
            var padding = 50;
             
            //Dynamic, random dataset

// Hardcoded option works
//        var dataset = [
//                          [5, 20], [480, 90], [250, 50], [100, 33], [330, 95],
//                          [410, 12], [475, 44], [25, 67], [85, 21], [220, 88],
//                          [600, 150]
//                        ];

// csv Import option not working as expected for why???
var dataset = [];

d3.csv("./datatest.csv", function(data) {
data.forEach(function(d) { dataset.push([ +d.x, +d.y ]) })
console.log(dataset)
});
console.log(dataset)
alert(dataset)
console.log(dataset)

// ### end of csv import code, expecting the same data to follow from whichever method is used.

            //Create scale functions
            var xScale = d3.scale.linear()
                                 .domain([0, d3.max(dataset, function(d) { return d[0]; })])
                                 .range([padding, w - padding * 2]);
 
            var yScale = d3.scale.linear()
                                 .domain([0, d3.max(dataset, function(d) { return d[1]; })])
                                 .range([h - padding, padding]);
 
            //Define X axis
            var xAxis = d3.svg.axis()
                              .scale(xScale)
                              .orient("bottom")
                              .ticks(5);
 
            //Define Y axis
            var yAxis = d3.svg.axis()
                              .scale(yScale)
                              .orient("left")
                              .ticks(5);
 
            //Create SVG element
            var svg = d3.select("body")
                        .append("svg")
                        .attr("width", w)
                        .attr("height", h);
 
            //Create circles
            svg.selectAll("circle")
               .data(dataset)
               .enter()
               .append("circle")
               .attr("cx", function(d) {
                       return xScale(d[0]);
               })
               .attr("cy", function(d) {
                       return yScale(d[1]);
               })
               .attr("r", 3);
             
            //Create X axis
            svg.append("g")
                .attr("class", "x axis")
                .attr("transform", "translate(0," + (h - padding) + ")")
                .call(xAxis);
             
            //Create Y axis
            svg.append("g")
                .attr("class", "y axis")
                .attr("transform", "translate(" + padding + ",0)")
                .call(yAxis);
 
 
 
            //On click, update with new data            
            d3.select("p")
                .on("click", function() {
 
                    //New values for dataset
                    var numValues = dataset.length;                                 //Count original length of dataset
                    var maxRange = Math.random() * 1000;                        //Max range of new values
                    dataset = [];                                                   //Initialize empty array
                    for (var i = 0; i < numValues; i++) {                         //Loop numValues times
                        var newNumber1 = Math.floor(Math.random() * maxRange);    //New random integer
                        var newNumber2 = Math.floor(Math.random() * maxRange);    //New random integer
                        dataset.push([newNumber1, newNumber2]);                    //Add new number to array
                    }
                     
                    //Update scale domains
                    xScale.domain([0, d3.max(dataset, function(d) { return d[0]; })]);
                    yScale.domain([0, d3.max(dataset, function(d) { return d[1]; })]);
 
                    //Update all circles
                    svg.selectAll("circle")
                       .data(dataset)
                       .transition()
                       .duration(1000)        
                       .attr("cx", function(d) {
                               return xScale(d[0]);
                       })
                       .attr("cy", function(d) {
                               return yScale(d[1]);
                       });
 
                    //Update X axis
                    svg.select(".x.axis")
                        .transition()
                        .duration(1000)
                        .call(xAxis);
                     
                    //Update Y axis
                    svg.select(".y.axis")
                        .transition()
                        .duration(1000)
                        .call(yAxis);
 
                });
 
 
        </script>
    </body>
</html>

[Tristan Reid]

Hi David.  Here's what's going on:  Whenever you do a call to something like d3.csv, think of it as 'scheduling' the code to run instead of running it immediately.  The function within is getting called only after the file is finished loading.  Your code creates the dataset, schedules the call, prints the still-empty dataset, and the reason the dataset is available after the alert is that the alert gives it time to load.

What I believe you'll want to do is trigger all the rest of your code from within that d3.csv call.  

var dataset = []; 

d3.csv("./datatest.csv", function(data) { 
    data.forEach(function(d) { dataset.push([ +d.x, +d.y ]) }) 
    console.log(dataset) 

    main();
});

function main() {

   // do what you want with dataset, it will be populated now

}

Apologies in advance if the formatting of this post doesn't work out - first time using Google Groups.

-t.

...

[David Brown]

Many thanks, Tristan..

That's worked well but I've had to use a http refresh to get the effect of it picking up on live dynamic data.

I expect there must be a sensible fix from within javascript but then expected I couldn't do the same call to main() again without getting tied in a knot.

Rust has     try!(f.sync_data());
..so, I expect javascript has similar.

Anyhow, I'm happy :)

Thanks again

[Drew Winget]

I understand the urge to thrust forward and get something done without spending valuable time on abstract reading or fundamentals (we've all been through it!), but javascript really is very different from other languages in several key ways, which is confusing because it looks very similar to something like C or Java. Believe me, you will save time in the mid-run, and definitely in the long-run, if you take a week off to read a good JS book cover to cover, and maybe a d3 book too. I really wish I had taken a week or two off and read Speaking Javascript and Interactive Data Visualisation for the Web cover to cover early on. It would have saved me months of frustration and ignorance. Both are free online and well worth the time investment.

For a deeper look at all the surrounding nonsense about the way Javascript interacts with the browser, I recommend Secrets of the Javascript Ninja by John Resig, creator of jQuery. For more opinionated exposition of how to use Javascript as a language, many will recommend Javascript: the Good Parts (though I didn't find it very helpful). Finally, for a very well-written introduction to functional programming (albeit with rather academic examples) and the upcoming ES6, I recommend Reginald Braithwaite's Javascript Alongé, and Javascript Spessore (both of which are free to read online).

Good Luck!

-Drew

--
You received this message because you are subscribed to the Google Groups "d3-js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to d3-js+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Drew Winget]

You can get live data by updating it when it changes. Call whatever function is using the data again when you have new data.

If you are talking about live data from the server, use websockets, or poll (not recommended). What is your backend?

[David Brown]

Thanks Drew...

That's good advice but my principal interest was simply to use d3 to display dynamic data thrown out from Rust; javascript just getting in the way.
It seems I've broken the back of this now; I've got the d3 making clean transitions off the back of data in a file and using setInterval to update that regularly.

Later, I'll look at websockets rather than just a local file.