Re: I want to ask about OSM -> JSON convert

[Jeffrey Warren]

can you post the memory error? how big was the .osm file?

I'm assuming you're working from this page? 

https://github.com/jywarren/cartagen/issues/103

I just also updated some of the links, fixed the API docs link in the README and put the whole API docs in the client as flat files. I'm also trying to get wiki.cartagen.org to redirect directly to entries in the Github wiki, but it may take some time for the DNS changes to go through.

Jeff

On Mon, Apr 16, 2012 at 4:40 AM, <le....@nomovok.com> wrote:

Hello,
My name is Dai. I started to use cartagen, and I'm searching for a method
that I can convert .osm file from OpenStreetMap (this for example:
http://downloads.cloudmade.com/asia/eastern_asia/japan/kanagawa/kanagawa.osm.bz2).
I tried xml2json from https://github.com/hay/xml2json but no luck, I can
not get the output file due to some memory error. Could you help me out
with my problem? Thanks in advance :D
Br, Dai

[Jeffrey Warren]

if you upload all your files somewhere we can see them, it'll be easier to see what's going wrong.

jeff

On Tue, Apr 17, 2012 at 1:22 AM, <le....@nomovok.com> wrote:

Hi,
Yes, my file is about 220MB, and the memory error is:
======
Traceback (most recent call last):
 File "./xml2json.py", line 197, in <module>
   main()
 File "./xml2json.py", line 185, in main
   out = xml2json(input, strip = 0)
 File "./xml2json.py", line 147, in xml2json
   return elem2json(elem,strip=strip)
 File "./xml2json.py", line 127, in elem2json
   return simplejson.dumps(elem_to_internal(elem,strip=strip))
 File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line
261, in dumps
   return _default_encoder.encode(obj)
 File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 214,
in encode
   chunks = self.iterencode(o, _one_shot=True)
 File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 282,
in iterencode
   return _iterencode(o, 0)
MemoryError
======
This only happens with large file, I successfully converted small file,
but if I put it in cartagen and edit the setup following the guide, it
won't work, nothing is rendered in browser (My command like this:
./xml2json.py -t xml2json test.osm -o test.json). Do you know why?
Br, Dai

[trey]

I see that this is several months old, but I did a rewrite of xml2json using Python3 which uses tempfiles to parse very large files without using all your memory. 

 I've successfully parsed OSM files as large as 4GB with the tool, though be warned that you will still need about 1GB of RAM available to do something that large.  Still, I don't know what you would do with such a large file...  The new script with usage information in the readme is on Github, along with the forked version.  My version is called xml2json3 (clever!)  

The output is Cartagen friendly.  Be aware that I changed up the command line options a bit.  The readme should be accurate.  Let me know if it works for you!

[Jeffrey Warren]

Dai eventually posted his files:

---------- Forwarded message ----------
From: <le....@nomovok.com>
Date: Tue, Apr 17, 2012 at 11:58 AM
Subject: Re: I want to ask about OSM -> JSON convert
To: Jeffrey Warren <je...@publiclaboratory.org>

Hi,
This is two files, .osm and .json that is converted successfully using
xml2json:
http://www.mediafire.com/?yzdr135ulm5xl93
http://www.mediafire.com/?8foxfgcj2qu75hm
Please have a look. Thanks :)
Br, Dai