XML to JSON and storing in MongoDB

[Amit Asthana]

Hi, 

We have created a application which converts XML files into JSON and then stores it into MongoDB. 

We have a huge dataset, we are using this application to convert XML Data in JSON Format and store it into MongoDB

The data that we have is so huge that converting one day data takes 3 days of JAVA Script running. 

What possible solution can be for this

Thanks in anticipation. 

[Kevin Adistambha]

Hi Amit,

The data that we have is so huge that converting one day data takes 3 days of JAVA Script running.
What possible solution can be for this

I’m not clear whether the conversion is slow, or MongoDB itself is slow. Could you post more details and evidence? Also, what is the actual size of “huge”?

If I can suggest one thing, have you tried parallelizing the conversion process? It may be that the conversion process itself is too much for a single machine to handle.

Best regards,
Kevin

​

[Marte Marte]

I'm interested in sharing approaches on this.

Mongodb does not have any kind of helper in storing xml, because it's json-based.

From my experience, if you use jackson (https://github.com/FasterXML/jackson)  with following constraint:

- don't load all file in memory

- use streaming version for parser

- bulkwrite into mongodb

You can achieve good performance, in my test 2GB for xml file in 2 minute inserted in mongodb. If you are interested I can share the code

Best regards