OsmandMapCreator fails creating an .obf from .pbf (geofabrik), pls help!
1,106 views
Skip to first unread message
BlueCharge
Jan 3, 2012, 6:40:23 PM1/3/12
to Osmand
Hello,
I would like to convert an .obf from a downloaded .pbf (110 MB,
country: lower saxony).
First I tried the latest nb: OsmAndMapCreator-HEAD-nb-2012-01-03.zip
but it failed with a java exception.
Then I used the OsmAndMapCreator-master-nb-2011-12-26.zip,
but it also failed with a message: java.lang.OutOfMemoryError: Java
heap space.
Has anybody been successful with self-converting an .obf file using
the map creator?
What should I do, what is wrong? Constantly running into errors (of
course near the end of a long process) is really not nice...
I would appreciate any help!
Thank you,
Peter
I would like to convert an .obf from a downloaded .pbf (110 MB,
country: lower saxony).
First I tried the latest nb: OsmAndMapCreator-HEAD-nb-2012-01-03.zip
but it failed with a java exception.
Then I used the OsmAndMapCreator-master-nb-2011-12-26.zip,
but it also failed with a message: java.lang.OutOfMemoryError: Java
heap space.
Has anybody been successful with self-converting an .obf file using
the map creator?
What should I do, what is wrong? Constantly running into errors (of
course near the end of a long process) is really not nice...
I would appreciate any help!
Thank you,
Peter
Max
Jan 3, 2012, 9:53:03 PM1/3/12
to Osmand
> but it also failed with a message: java.lang.OutOfMemoryError: Java
> heap space.
Increase the maximum size of the memory allocation pool.
> heap space.
Therefor change the "Xmx" value in OsmAndMapCreator.sh or
OsmAndMapCreator.bat.
For example:
-Xmx2048M
BlueCharge
Jan 4, 2012, 2:13:13 AM1/4/12
to Osmand
Hi Max,
thank you for your advice. I regret that this is not possible,
although there are about 4300MB of free memory in this 6GB pc.
The maximum value the mapcreator would start with is xmx1448M (hey,
this feels like in the old times of DOS :-)
MapCreator now finishes with a .obf file of 20MB, which doesn´t work
in Osmand.
Of course that file can´t be correct regarding the input file of about
110MB.
Any ideas how to get it working?
Thank you very much,
Peter
thank you for your advice. I regret that this is not possible,
although there are about 4300MB of free memory in this 6GB pc.
The maximum value the mapcreator would start with is xmx1448M (hey,
this feels like in the old times of DOS :-)
MapCreator now finishes with a .obf file of 20MB, which doesn´t work
in Osmand.
Of course that file can´t be correct regarding the input file of about
110MB.
Any ideas how to get it working?
Thank you very much,
Peter
Jindřich Makovička
Jan 4, 2012, 2:35:25 AM1/4/12
to osm...@googlegroups.com
On Wed, Jan 4, 2012 at 08:13, BlueCharge <pmaen...@googlemail.com> wrote:
> Hi Max,
>
> thank you for your advice. I regret that this is not possible,
> although there are about 4300MB of free memory in this 6GB pc.
> The maximum value the mapcreator would start with is xmx1448M (hey,
> this feels like in the old times of DOS :-)
> Hi Max,
>
> thank you for your advice. I regret that this is not possible,
> although there are about 4300MB of free memory in this 6GB pc.
> The maximum value the mapcreator would start with is xmx1448M (hey,
> this feels like in the old times of DOS :-)
You probably need 64-bit Java for that.
--
Jindřich Makovička
Pavol Zibrita
Jan 4, 2012, 2:48:11 AM1/4/12
to osm...@googlegroups.com
Probably you have to clean up the directory where the files where computed, it is probable that
map creator wanted to start where it finished and this had a bad result because of the
previous crash. Just a suggestion.
Regards,
Pavol
PS: And yes, for having more memory, you need 64 bit system. But, I don't map creator can currently
use more than around 1gb memory. Not for a 100mb pbf file. Most of the data are stored in database
on filesystem. Currently the map creator is not very efficient memory consumer.
map creator wanted to start where it finished and this had a bad result because of the
previous crash. Just a suggestion.
Regards,
Pavol
PS: And yes, for having more memory, you need 64 bit system. But, I don't map creator can currently
use more than around 1gb memory. Not for a 100mb pbf file. Most of the data are stored in database
on filesystem. Currently the map creator is not very efficient memory consumer.
Jindřich Makovička
Jan 4, 2012, 2:55:53 AM1/4/12
to osm...@googlegroups.com
On Wed, Jan 4, 2012 at 08:48, Pavol Zibrita <pavol....@gmail.com> wrote:
> Probably you have to clean up the directory where the files where computed,
> it is probable that
> map creator wanted to start where it finished and this had a bad result
> because of the
> previous crash. Just a suggestion.
>
> Regards,
> Pavol
>
> PS: And yes, for having more memory, you need 64 bit system. But, I don't
> map creator can currently
> use more than around 1gb memory. Not for a 100mb pbf file. Most of the data
> are stored in database
> on filesystem. Currently the map creator is not very efficient memory
> consumer.
> Probably you have to clean up the directory where the files where computed,
> it is probable that
> map creator wanted to start where it finished and this had a bad result
> because of the
> previous crash. Just a suggestion.
>
> Regards,
> Pavol
>
> PS: And yes, for having more memory, you need 64 bit system. But, I don't
> map creator can currently
> use more than around 1gb memory. Not for a 100mb pbf file. Most of the data
> are stored in database
> on filesystem. Currently the map creator is not very efficient memory
> consumer.
Maybe, but with the default setting of 720M, I was getting out of
memory exceptions when processing Czech Republic. So I am using
-Xmx2048 now.
--
Jindřich Makovička
BlueCharge
Jan 4, 2012, 1:57:03 PM1/4/12
to Osmand
Yeah, it works for the 110 MB file now!
Thank you all for your input. I installed the 64bit version of java
and then increased the xmx value to 3448MB, that was helpful.
How much memory is needed for a 1.1GB file (Germany)? Can that be
estimated?
Greetings from Wolfsburg,
Peter
Thank you all for your input. I installed the 64bit version of java
and then increased the xmx value to 3448MB, that was helpful.
How much memory is needed for a 1.1GB file (Germany)? Can that be
estimated?
Greetings from Wolfsburg,
Peter
Jindřich Makovička
Jan 4, 2012, 2:50:37 PM1/4/12
to osm...@googlegroups.com
On Wed, Jan 4, 2012 at 19:57, BlueCharge <pmaen...@googlemail.com> wrote:
> Yeah, it works for the 110 MB file now!
>
> Thank you all for your input. I installed the 64bit version of java
> and then increased the xmx value to 3448MB, that was helpful.
>
> How much memory is needed for a 1.1GB file (Germany)? Can that be
> estimated?
> Yeah, it works for the 110 MB file now!
>
> Thank you all for your input. I installed the 64bit version of java
> and then increased the xmx value to 3448MB, that was helpful.
>
> How much memory is needed for a 1.1GB file (Germany)? Can that be
> estimated?
I'd suppose 3.5G heap will suffice, but you need either an SSD or at
least 16-32GB RAM for a disk cache to get the job done in a reasonable
time. I once did that on 2.2GHz opteron w/ 32GB RAM, and it took eight
hours. I suppose it will take much longer now, because Map Creator
became a bit more sophisticated since.
--
Jindřich Makovička
Horst Müller
Jan 4, 2012, 2:51:54 PM1/4/12
to osm...@googlegroups.com
I can provide you another result: A laptop with 768MB of RAM running XP was able to convert the map of Hessen (no changes were made to MapCreator).
From my own experience I hope you have a SSD or a very fast HDD. I once tried to process the file of Germany myself, but gave up after more than one day of process duration. For this experiment I used a PC with 4GB of RAM. At least it started, but as I said I didn't go all the way down...
Regards,
Horst
From my own experience I hope you have a SSD or a very fast HDD. I once tried to process the file of Germany myself, but gave up after more than one day of process duration. For this experiment I used a PC with 4GB of RAM. At least it started, but as I said I didn't go all the way down...
Regards,
Horst
2012/1/4 BlueCharge <pmaen...@googlemail.com>
Victor Shcherb
Jan 4, 2012, 2:56:27 PM1/4/12
to osm...@googlegroups.com
Actually the size of target obf is too close to maximum. Maximum obf part can be 2GB (because in Java int - 32bit and can be negative).
Also take care POI indexing is very consumable memory last step.
Stephan75
Jan 4, 2012, 3:14:45 PM1/4/12
to Osmand
Hey Peter,
If you want to process a source OSM file for whole Germany, I would
recommend a filtering of that before via osmfilter.
You can drop all unnecessary version info and author info, and even
more elements that are not so important, for example trees, hydrants,
lit, all about power lines ... or even some landuse objects.
Has anyone tried this before? what results / differences to unfiltered
osm source file?
Greets, Stephan
If you want to process a source OSM file for whole Germany, I would
recommend a filtering of that before via osmfilter.
You can drop all unnecessary version info and author info, and even
more elements that are not so important, for example trees, hydrants,
lit, all about power lines ... or even some landuse objects.
Has anyone tried this before? what results / differences to unfiltered
osm source file?
Greets, Stephan
BlueCharge
Jan 4, 2012, 5:01:53 PM1/4/12
to Osmand
This gets more and more interesting.
C: is an SSD with 128GB. I´ll give it a try. Would be great to have a
new and complete map of Germany in Osmand for my next trip :-)
C: is an SSD with 128GB. I´ll give it a try. Would be great to have a
new and complete map of Germany in Osmand for my next trip :-)
Victor Shcherb
Jan 4, 2012, 6:08:22 PM1/4/12
to osm...@googlegroups.com
I will also try to create France/Germany/Russia on the server without SSD but with 8 GB operating memory ;) At least for now it eats everything.
Best Regards,
BlueCharge
Jan 12, 2012, 10:53:44 AM1/12/12
to Osmand
Unfortunately it didn´t work... "java.lang.IndexOutOfBoundsExeption:
Index:0, Size: 0"
Machine: 6GB RAM, 128GB SSD.
hm....no way?
On 4 Jan., 20:56, Victor Shcherb <victor.shch...@gmail.com> wrote:
> Actually the size of target obf is too close to maximum. Maximum obf part
> can be 2GB (because in Java int - 32bit and can be negative).
>
> Also take care POI indexing is very consumable memory last step.
>
> Victor
>
> 2012/1/4 Horst Müller <lungenfil...@googlemail.com>
>
> - Zitierten Text anzeigen -
Index:0, Size: 0"
Machine: 6GB RAM, 128GB SSD.
hm....no way?
On 4 Jan., 20:56, Victor Shcherb <victor.shch...@gmail.com> wrote:
> Actually the size of target obf is too close to maximum. Maximum obf part
> can be 2GB (because in Java int - 32bit and can be negative).
>
> Also take care POI indexing is very consumable memory last step.
>
> Victor
>
>
>
>
> > I can provide you another result: A laptop with 768MB of RAM running XP
> > was able to convert the map of Hessen (no changes were made to MapCreator).
>
> > From my own experience I hope you have a SSD or a very fast HDD. I once
> > tried to process the file of Germany myself, but gave up after more than
> > one day of process duration. For this experiment I used a PC with 4GB of
> > RAM. At least it started, but as I said I didn't go all the way down...
>
> > Regards,
> > Horst
>
> > 2012/1/4 BlueCharge <pmaenne...@googlemail.com>
>
>
> > I can provide you another result: A laptop with 768MB of RAM running XP
> > was able to convert the map of Hessen (no changes were made to MapCreator).
>
> > From my own experience I hope you have a SSD or a very fast HDD. I once
> > tried to process the file of Germany myself, but gave up after more than
> > one day of process duration. For this experiment I used a PC with 4GB of
> > RAM. At least it started, but as I said I didn't go all the way down...
>
> > Regards,
> > Horst
>
>
> >> Yeah, it works for the 110 MB file now!
>
> >> Thank you all for your input. I installed the 64bit version of java
> >> and then increased the xmx value to 3448MB, that was helpful.
>
> >> How much memory is needed for a 1.1GB file (Germany)? Can that be
> >> estimated?
>
> >> Greetings from Wolfsburg,
> >> Peter- Zitierten Text ausblenden -
> >> Yeah, it works for the 110 MB file now!
>
> >> Thank you all for your input. I installed the 64bit version of java
> >> and then increased the xmx value to 3448MB, that was helpful.
>
> >> How much memory is needed for a 1.1GB file (Germany)? Can that be
> >> estimated?
>
> >> Greetings from Wolfsburg,
>
> - Zitierten Text anzeigen -
Victor Shcherb
Jan 12, 2012, 11:28:29 AM1/12/12
to osm...@googlegroups.com
I also tried to generate Germany but it looks ridiculous :) Even France/Russia/Canada was processed but not Germany.
I think that's not effective at all. The best way would be introducing "clever" algorithm of combining map indexes. I guess it should not be hard. Current indexes combining algorithm has only issue with Address combining (it doesn't combined as desired).
Horst Müller
Jan 12, 2012, 12:09:52 PM1/12/12
to osm...@googlegroups.com
Yeah, the Germans always make trouble ;)
Maybe that's the downside of having one of the most detailed maps. Even comparable to companies like Tele Atlas. Nevertheless I would like to have a "All-in-one"-solution, too. For the beginning I would be glad if there is a single file containing all necessary 16 files to download for Germany. Is this possible or would the size be too large to upload?
Horst
2012/1/12 Victor Shcherb <victor....@gmail.com>
Reply all
Reply to author
Forward
0 new messages