Load tile maps in chunks?

[Art Aquino]

I have a procedurally generated tilemap, like in Terraria. Is there a way to load my map in chunks? My current map size is 5056x5056. I would like the size of my world to be much much bigger. Unfortunately it seems Python chokes when the tile map size goes beyond a certain size. 

I tried adding a second tilemap and offset its position so that it is directly to the right of my main tile map. I added both maps to the scrolling manager. The main map works fine, but I cannot see the second tile map. So my next question is, is it possible to load multiple scrolling layers (or tile maps) in one scene and have them offset so they seamlessly connect at the edges and my player object can scroll from one layer to the next? The idea is to have layers load when the player object approaches them, and then they become visible. If the player is far enough from a layer then that layer gets hidden or some other method is used to free up system resources. 

[claudio canepa]

On Mon, Aug 5, 2013 at 5:35 PM, Art Aquino <art...@gmail.com> wrote:

I have a procedurally generated tilemap, like in Terraria. Is there a way to load my map in chunks? My current map size is 5056x5056.

Pixels or Tiles ? Which tile dimensions ? How many different tiles ?

 

I would like the size of my world to be much much bigger. Unfortunately it seems Python chokes when the tile map size goes beyond a certain size. 

Are you talking about load time or frame rate ? Or 'choke' means some other defect ? 

Are there other entities besides the tilemap ? If you have a lot of actors in the map you will need some way to only update actors near the player.

The tilemap itself is drawing only visible tiles, so it shouldn't pose a fps problem ... Wait, are you zooming out to make visible all the world ? 

 

I would try to identify the real bottlenecks in the app before changing the code. 

I tried adding a second tilemap and offset its position so that it is directly to the right of my main tile map. I added both maps to the scrolling manager. The main map works fine, but I cannot see the second tile map. So my next question is, is it possible to load multiple scrolling layers (or tile maps) in one scene and have them offset so they seamlessly connect at the edges and my player object can scroll from one layer to the next? The idea is to have layers load when the player object approaches them, and then they become visible. If the player is far enough from a layer then that layer gets hidden or some other method is used to free up system resources. 

See above.

[Arturo Aquino]

The tile dimensions are 32x32, the map size is 5056 pixels (not tiles), this works great. So I have 158 rows and columns of tiles. When I try to generate a tile map of, 50,560 pixels, the IDE that I use (PyCharm) gives a memory error. I also tried running with IDLE and receive a memory error again. The error occurs in cocos tiles.py, line 989, self.i = self.j = i, j

I do have zooming implemented, but in this case the error occurs right after the game window opens and it just locks up. The window is white, it does not get to the point of rendering the map itself.

Because of this, I was thinking of using multiple tile maps, that are themselves tiled, and are loaded offscreen dynamically as the player approaches that layer. Since my tile map is generated procedurally, I can easily change its size and make them very big. So in order to avoid the memory error, I am looking for a way to have multiple tile maps in the same scene, side by side, but only load the maps that are close to my player object.

I should also mention that I am using Pymunk physics, and my code generates pymunk segments around "solid" tiles of my map. The only other physics object is my player, which is a simple cube for now. But the memory error comes from cocos not from pymunk.

--
You received this message because you are subscribed to a topic in the Google Groups "cocos2d discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cocos-discuss/mPP7DS-JapY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cocos-discus...@googlegroups.com.
To post to this group, send email to cocos-...@googlegroups.com.
Visit this group at http://groups.google.com/group/cocos-discuss.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
"There is a peace that is not worthy to have, frankly, if it is at the cost of truth."

[claudio canepa]

On Mon, Aug 5, 2013 at 8:28 PM, Arturo Aquino <art...@gmail.com> wrote:

The tile dimensions are 32x32, the map size is 5056 pixels (not tiles), this works great. So I have 158 rows and columns of tiles. When I try to generate a tile map of, 50,560 pixels, the IDE that I use (PyCharm) gives a memory error. I also tried running with IDLE and receive a memory error again. The error occurs in cocos tiles.py, line 989, self.i = self.j = i, j

2.5 millions of cells. Not sure if a memory error should be expected, but is understandable the idea of divide in smaller chunks.

 

I do have zooming implemented, but in this case the error occurs right after the game window opens and it just locks up. The window is white, it does not get to the point of rendering the map itself.

Because of this, I was thinking of using multiple tile maps, that are themselves tiled, and are loaded offscreen dynamically as the player approaches that layer.

I think that is not directly supported in current cocos:  The way that MapLayer.set_view , MapLayer.get_visible_cells,  ScrollingManager.set_focus use the tilemap view_x, view_y seems to assume only one tilemap per ScrollableLayer.

 

Since my tile map is generated procedurally, I can easily change its size and make them very big. So in order to avoid the memory error, I am looking for a way to have multiple tile maps in the same scene, side by side, but only load the maps that are close to my player object.

Another option is to create / delete the cells that are visible

[Art Aquino]

Thanks for your comments Claudio. I have been increasing the size of the terrain little by little. I did a test with the terrain 15,000x15,000 pixels and it works fine. Im still not sure why it crashes when I bump up the size to 50,000 pixels. For now a smaller map should be fine so I will continue with my project. Thanks for your help. Cocos is a great library.

[Art Aquino]

I think I have found the problem. Python runs out of memory while parsing the XML file that has the map information when I generate 2.5 million cells. It looks like the cElementTree library I am using for XML parsing is not very good for large files. I will have to find another way to parse the XML. I found an article and I will try using that method instead.

http://www.ibm.com/developerworks/xml/library/x-hiperfparse/

[claudio canepa]

On Tue, Aug 13, 2013 at 10:37 PM, Art Aquino <art...@gmail.com> wrote:

I think I have found the problem. Python runs out of memory while parsing the XML file that has the map information when I generate 2.5 million cells. It looks like the cElementTree library I am using for XML parsing is not very good for large files. I will have to find another way to parse the XML. I found an article and I will try using that method instead.

http://www.ibm.com/developerworks/xml/library/x-hiperfparse/

  

Why not build some gameplay in a map size that the current code supports and deffer the problem of big maps to a moment you are more familiarized with cocos ?

This way you can verify if the game idea is sound before investing in infrastructure.

Besides, when you are more familiarized with cocos you could see how to support loading in chunks (while this is not directly supported in cocos, you can write your own classes to support this. )

[Art Aquino]

Very good advice Claudio. Yes I have been testing with small map size to test the procedural generation and everything works fine. Last night I learned how to generate Rectmaps without writing to xml or tmx format. The map is now generated in memory. I did a test with 2.5 million tiles and it loads and scrolls just fine. Now I will test saving the data to xml so that I can "save" and "load" my map using the lxml library instead of cElementTree. 

I have been reading the cocos documentation every day and learn something new. I love it. I would love to contribute to the project but I am self taught Python programmer and I am afraid my skills or coding practices are not up to standard yet. I did look through the cocos tiles.py code and noticed you also use cElementTree to parse XML information so I was very excited because I understand that code now.

Do you have advice on how I would load in chunks? Just a general idea of where to start? I will study cocos more and try to figure it out also. Thanks for your advice. 

[Art Aquino]

By the way, here is a zoomed out screen shot of what the procedural map looks like so far. bad quality because tiles look so small when zoomed out. 

https://www.dropbox.com/s/fjg0sddsrbvdcg9/map.PNG

I am very proud of this so far. Thanks for all your help. :)

[Art Aquino]

So I am doing more tests loading a large XML tilemap into cocos. The XML file is about 50 MB. Here is the error I get:

Traceback (most recent call last):

  File "C:/Users/Art/Dropbox/TerrainApp/main.py", line 19, in <module>

    main()

  File "C:/Users/Art/Dropbox/TerrainApp/main.py", line 12, in main

    my_scene = scenes.GameScene(director)

  File "C:\Users\Art\Dropbox\TerrainApp\scenes.py", line 16, in __init__

    self.scroller = layers.TerrainScroller(director, self.clock)

  File "C:\Users\Art\Dropbox\TerrainApp\layers.py", line 15, in __init__

    self.maptemplate = cocos.tiles.load('terrain.xml')['maptemplate']

  File "C:\Python27\lib\site-packages\cocos2d-0.5.5-py2.7.egg\cocos\tiles.py", line 213, in load

    obj = load_tiles(filename)

  File "C:\Python27\lib\site-packages\cocos2d-0.5.5-py2.7.egg\cocos\tiles.py", line 226, in load_tiles

    resource.handle(root)

  File "C:\Python27\lib\site-packages\cocos2d-0.5.5-py2.7.egg\cocos\tiles.py", line 119, in handle

    return self.factories[tag.tag](self, tag)

  File "C:\Python27\lib\site-packages\cocos2d-0.5.5-py2.7.egg\cocos\tiles.py", line 181, in resource_factory

    self.handle(child)

  File "C:\Python27\lib\site-packages\cocos2d-0.5.5-py2.7.egg\cocos\tiles.py", line 119, in handle

    return self.factories[tag.tag](self, tag)

  File "C:\Python27\lib\site-packages\cocos2d-0.5.5-py2.7.egg\cocos\tiles.py", line 530, in rectmap_factory

    properties = _handle_properties(cell)

  File "C:\Python27\lib\site-packages\cocos2d-0.5.5-py2.7.egg\cocos\tiles.py", line 349, in _handle_properties

    properties = {}

MemoryError

Claudio, I noticed that you use ElementTree and not cElementTree for parsing XML. Would it be possible to change this so it uses cElementTree instead since it is faster and uses less memory? 

[Art Aquino]

So I changed all references for ElementTree to cElementTree in cocos tiles.py and it is a lot faster, but I still get a memory error. I think I will try to write my own class to parse the xml.

Basically, if I generate the 2.5 million tiles using the methods cocos provides for RectCell, Tile, and RectMapLayer, it works. I can scroll and everything works fine. But if I write the data to xml using my own class, and then load that file using cocos.tiles.load('terrain.xml')['maptemplate'], then I receive a memory error. 

[Art Aquino]

I did further research and more tests and it looks like it is a problem with 32-bit Python. I will try using 64-bit Python and see if this prevents the memory error.