Re: [nodejs] What's different of requiring json or js?

86 views
Skip to first unread message

Joshua Gross

unread,
Aug 5, 2012, 4:31:03 PM8/5/12
to nod...@googlegroups.com
There may be a better answer for this, but JSON is actually more rigidly structured than standard JavaScript. So, it makes sense to me that a more specialized parser would need more memory than just sending JavaScript code to the VM. 

-- Joshua Gross
Christian / SpanDeX, Inc. / BA Candidate of Computer Science, UW-Madison 2013

On Aug 4, 2012, at 9:43 AM, Kei Son <hey.ca...@gmail.com> wrote:

I have a huge JSON file, right, it's 78MB.
When I just require it from node-cli it takes 450MB memory footprint. I can understand that is bloated 6x times because the JSON file is just a string  in the storage, but it needs spaces for indexing, making some padding, optimization, linking and whatever else when it comes on memory.

But the other hand, when I add some strings like "module.exports=" at the top of the JSON file and require it as a normal js file, it takes only 200MB. I thought requiring as JSON would be faster and smaller than as JS. Because there are no seeking time for code-optimization and no extra process at the Node.js side.

What's different between two and why?

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Kei Son

unread,
Aug 5, 2012, 8:51:46 PM8/5/12
to nod...@googlegroups.com
makes sense. different parser needs more extra spaces. however the spaces for the parser-only should be freed at gc time, isn't it?

2012년 8월 6일 월요일 오전 5시 31분 3초 UTC+9, Joshua Gross 님의 말:

Marak Squires

unread,
Aug 5, 2012, 8:58:22 PM8/5/12
to nod...@googlegroups.com
You might want to consider using a streaming JSON parser.

On Sat, Aug 4, 2012 at 7:43 AM, Kei Son <hey.ca...@gmail.com> wrote:
I have a huge JSON file, right, it's 78MB.
When I just require it from node-cli it takes 450MB memory footprint. I can understand that is bloated 6x times because the JSON file is just a string  in the storage, but it needs spaces for indexing, making some padding, optimization, linking and whatever else when it comes on memory.

But the other hand, when I add some strings like "module.exports=" at the top of the JSON file and require it as a normal js file, it takes only 200MB. I thought requiring as JSON would be faster and smaller than as JS. Because there are no seeking time for code-optimization and no extra process at the Node.js side.

What's different between two and why?

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

Kei Son

unread,
Aug 5, 2012, 9:44:05 PM8/5/12
to nod...@googlegroups.com
Thanks for the idea.
But I just want to know why.

2012년 8월 6일 월요일 오전 9시 58분 22초 UTC+9, Marak Squires 님의 말:

Tim Caswell

unread,
Aug 6, 2012, 11:05:01 AM8/6/12
to nod...@googlegroups.com
On Sun, Aug 5, 2012 at 3:31 PM, Joshua Gross <joshua...@gmail.com> wrote:
> There may be a better answer for this, but JSON is actually more rigidly
> structured than standard JavaScript. So, it makes sense to me that a more
> specialized parser would need more memory than just sending JavaScript code
> to the VM.

That doesn't make sense to me. A simpler and more strict language is
easier to parse, not harder.

However, I'm pretty sure more effort has been put into parsing large
JS files than has been put into parsing large JSON files. It's not
uncommon to run across poorly written websites with megabytes of JS
code that needs to be parsed at startup.

As far as what concretely causes this, I think this will involving
digging into the V8 source code or getting an answer from someone on
the V8 team. Also there is a possibility that node is doing something
dumb and buffering too much when loading the file.

Kei Son

unread,
Aug 6, 2012, 11:13:37 AM8/6/12
to nod...@googlegroups.com
As we can see at #465 and #472, I'm pretty sure that there is no dumb and buffering things on Node.js side.
Unless there are some magic in the module._compile() as the opposite.

And i totally agree on your opinion. I should ask V8 team about this.
I'll let you know later at this thread.

Thanks.


2012년 8월 7일 화요일 오전 12시 5분 1초 UTC+9, Tim Caswell 님의 말:
> nodejs+unsubscribe@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+unsubscribe@googlegroups.com
Reply all
Reply to author
Forward
0 new messages