Memory error

[naab...@gmail.com]

Hi,

I use Batchrunner and Collector to collect a lot of data, and it may reach to 1GB if the number of runs is increased. I got a memory error. Any suggestion?

Thanks

[Tom Pike]

You need to save the data and clearit. Depending on how big each model run is you can either run a whole model and then save it as part of each model run (as you are generating the data yourself, pickle works well to save the data collector objects or you can save them as a .csv with pandas). You should then clear out the data collector to finish the model run, otherwise I pretty sure batchrunner will continue to store the data for each run (effectively bath runner will return an empyt dataframe). Or if each run is too large a certain number of steps you can follow the same process and then clear out the datacollector

Hope this helps.  

[naab...@gmail.com]

Thanks a lot. Shall I change the batchrunner.py or there is another alternative?

Regards

[Tom Pike]

I would think the easiest alternative is at the end of your model run save datacollector then clear datacollector.

So at the end of a model run something like:

```

data = datacollector.get_model_vars_dataframe()

data.to_csv("some_name"+str(iter)+".csv")

datacollector.model_vars = {}

```

The same would work for the agent records to.

The specific will matter here and what you want to do with the data, but hopefully this gives an example

Tom   

[nada gh]

Thanks.

If I use batch_run.run_all() it will run all the iterations at once!

--
Project repos: github.com/projectmesa
---
You received this message because you are subscribed to a topic in the Google Groups "ProjectMesa" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/projectmesa/5mNpkAGGn4o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to projectmesa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/projectmesa/d76794ac-fcc4-4efa-b64c-cdf0fb8ecf8bn%40googlegroups.com.