Load Large Files gist bl.ocks

[truoger]

Hi,

I'm new to d3 and bl.ocks, and I'm trying to make a basic map visualization using the census tracts in Ohio. I'm having trouble loading my JSON file into the gist. When I paste in the JSON data, it does not load all the data; it only shows a portion, and cuts everything after that. Based on some troubleshooting and googling, I think it's because the data file is larger than 1MB, but I'm not sure. 

Could somebody help explain to me how to correctly reference this larger JSON file in my code? The gist can be found at the link below. You'll see that the JSON data is cut off in the middle of an array in line 860. 

https://gist.github.com/learntocode611/7ea3a388ff58db62f27f

Thanks in advance for your help!

[Ian Johnson]

Hi Peter,

Your data isn't too big.

I noticed 2 problems:

1) you weren't properly calling d3.json, your callback should expect an error and a json object.

2) You had malformed geojson (you were missing an extra } and ] at the end of your features list

I forked your code and fixed those 2 issues:

http://bl.ocks.org/enjalot/bd8b668e718e6f21e64c

it looks like your projection is off though, as nothing is rendering still.

--
You received this message because you are subscribed to the Google Groups "d3-js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to d3-js+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Ian Johnson - 周彦

http://enja.org

[truoger]

Ian,

Thank you for your speedy reply. I appreciate your help! 

1) Understand about the improperly calling d3.json; thanks for the correction on that!

2)  Regarding the JSON data; the last example I had was a smaller sample dataset. I've updated the gist with the full dataset, and am again having the issue of the data cutting off around line 860. Would you mind taking a look at the gist again and seeing if you could help out? I get an error "Uncaught TypeError: Cannot read property 'features' of undefined" when I load the bl.ock, which I this is because the 'features' list is getting cut off...

I updated the bounding lat/long variables, so hopefully the projection will work if we can figure this out!

Thanks again - I really appreciate your taking the time to look into this! 

[Curran]

Hello,

Just as a quick test, I ran these two commands to count the number of lines in the json file you are using:

curl -O https://gist.githubusercontent.com/learntocode611/7ea3a388ff58db62f27f/raw/c78c606405e1967b29ff8466e4ecc0edf2fc12a2/us-states.json

wc -l us-states.json 

This shows that there are only 844 lines in the JSON file, which corresponds roughly with the number of features. When I clone the gist and count the lines, I get the same result. 

Also, the GeoJSON viewer in GitHub shows a map that has regions in Ohio, but does not cover the whole state. 

It looks like the data is getting cut off somewhere in your process before it actually gets added to the Git repository for the Gist, not anywhere in the D3 code. How are you generating us-states.json?

Best regards,

Curran

[truoger]

Curran - thanks for taking a look here. Yes, I agree that is what is happening.

The way I am generating us-states.json is copying the json data from the file on my computer into the Gist field. The file on my computer has 2000+ lines (one per census tract) and is just over 4MB in size. After I copy-paste into Gist, the file gets cut down to 860 lines for some reason. This leaves me with the partially rendered map you referenced. 

Is there a better way to load the data to Gist and generate my JSON file so that it doesn't get cut off? 

Thank you!

[Ian Johnson]

Yeah, what you'll need to do is clone the gist to your computer and copy the file into the directory. You can then commit the change like a normal git repo and push it back up to gists.

I thought the API limit was 10mb but it might be 1mb when posting data via the http API

I can't find any specific documentation, but this is what happens when you try to read large files:

https://developer.github.com/v3/gists/#truncation

You wont be able to use blocks with anything over 10mb because the raw_url only returns files up to 10mb in size (Which is what blocks uses).

[Phuoc Do]

Another option is to upload your json file to Amazon S3 or dropbox. Then use d3.json to load it.

For example:

d3.json("https://s3-us-west-2.amazonaws.com/vida-public/geo/us.json", function(error, us) {

});

[truoger]

Great, thank you all!

[truoger]

Hi fellows - 

If I may ask one additional follow-up question:

I've updated the code using Phuoc Do's suggestion of loading files via dropbox link (https://gist.github.com/learntocode611/7ea3a388ff58db62f27f) but the image still doesn't render on bl.ocks. I get an SVG with a g container, but no elements inside.

The file renders fine when i open it in a browser from the local file on my computer, so I know it works - its just a matter of translating it onto gist and bl.ocks. 

Any suggestions would be very appreciated. 

Thank you!

On Friday, August 28, 2015 at 3:36:09 AM UTC-4, Phuoc Do wrote: