PHP Fatal error: Allowed memory size. In populate

[Magí Carbonell Coll]

Hello,

Use Elastica with fosBundle Symfony2 and when I run "fos: Elastic: populate - index = test" php memory consumption.

I can not do "populate" without doctrine? to use less memory

My test table is more than 100 thousand records.

Thank you.

[Cassiano Tartari]

I think this is php configuration of memory in php.ini or in elasticsearch.html

I've passed that once and solved adding more memory, I think that default is 128MB

--
You received this message because you are subscribed to the Google Groups "Elastica - Elasticsearch PHP Client" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elastica-php-cl...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Cassiano Tartari]

Html = yml (cel corrector)

[Magí Carbonell Coll]

If you're right that is the memory of php.ini, but I can not go higher. And I have to 512M.

I sobresctito the method populateIndex and I can see how up memory for each block .... 

Start 2013-06-28 09:18:13

Mem 0: 38.98551940918

Mem 500: 44.33634185791

Mem 1000: 49.153396606445

Mem 1500: 54.007141113281

Mem 2000: 58.817642211914

Mem 2500: 60.98388671875

Mem 3000: 64.510932922363

Mem 3500: 71.133201599121

Mem 4000: 76.95792388916

Mem 4500: 81.73885345459

Mem 5000: 86.800567626953

Mem 5500: 89.852645874023

Mem 6000: 95.936080932617

Mem 6500: 100.71150970459

Mem 7000: 105.17926025391

Mem 7500: 108.40924835205

Mem 8000: 111.92178344727

Mem 8500: 116.95142364502

Mem 9000: 121.47180175781

Mem 9500: 125.1831741333

Mem 10000: 130.42961883545

Mem 10500: 134.66538238525

Mem 11000: 139.061378479

Mem 11500: 144.11206817627

Mem 12000: 147.96585083008

Mem 12500: 153.04677581787

Mem 13000: 158.07273864746

....

From what I've seen is the client elastic that accumulates memory.

Thank you.

El jueves, 27 de junio de 2013 23:26:04 UTC+2, Cassiano Tartari escribió:

Html = yml (cel corrector)

On Jun 27, 2013 6:25 PM, "Cassiano Tartari" <cassian...@gmail.com> wrote:

I think this is php configuration of memory in php.ini or in elasticsearch.html

I've passed that once and solved adding more memory, I think that default is 128MB

On Jun 27, 2013 1:25 PM, "Magí Carbonell Coll" <mag...@gmail.com> wrote:

Hello,

Use Elastica with fosBundle Symfony2 and when I run "fos: Elastic: populate - index = test" php memory consumption.

I can not do "populate" without doctrine? to use less memory

My test table is more than 100 thousand records.

Thank you.

--
You received this message because you are subscribed to the Google Groups "Elastica - Elasticsearch PHP Client" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elastica-php-client+unsub...@googlegroups.com.

[ruflin]

Hi Magi

There were some memory leaks in Elastica in the past. I thought those were fixed or very minimal. The main issue here is, that php is not really made for long running scripts so it is quite bad in clearing the memory. There are multiple solutions to this problem:

  * Find out which objects are not cleaned and try to clean them directly yourself.

  * Restart the script for every bulk request. This is probably the easiest one. Instead of having a script that runs for a very long time. Start the script multiple times. This is not too nice, but it works.

Still I would really like to figure out what causes the memory leak. Can you try to dig deeper into this to see what takes up memory?

[Magí Carbonell Coll]

Hi Ruflin

I've been doing tests and there are two points that consument memory:

1. - It's FOS\ElasticaBundle\Command in the populate.

2. - Is a client when the request is made.

Everything else is perfect.

test1: Without touching anything

Start 2013-07-01 08:20:23

Mem 0: 38.993087768555

Mem 500: 50.757934570312

Mem 1000: 62.787483215332

Mem 1500: 73.953758239746

Mem 2000: 87.036750793457

Mem 2500: 95.59366607666

Mem 3000: 104.4871673584

Mem 3500: 116.40343475342

Mem 4000: 128.45097351074

test 2: freeing memory at "DOCTRINE" in ElasticaBundle

....

$this->objectPersister->insertMany($objects);

$this->em->clear();

....

Start 2013-07-01 08:23:42

Mem 0: 38.99365234375

Mem 500: 44.350799560547

Mem 1000: 49.156135559082

Mem 1500: 54.013641357422

Mem 2000: 58.824699401855

Mem 2500: 60.993171691895

Mem 3000: 64.514678955078

Mem 3500: 71.139297485352

Mem 4000: 76.962303161621

test 3: canceling sending in class Bulk

    public function send()

    {

        $path = $this->getPath();

        $data = $this->toString();

//        $response = $this->_client->request($path, Request::PUT, $data);

//

//        return $this->_processResponse($response);

    }

Start 2013-07-01 08:26:54

Mem 0: 30.418685913086

Mem 500: 30.352066040039

Mem 1000: 30.547119140625

Mem 1500: 30.558395385742

Mem 2000: 30.774528503418

Mem 2500: 29.773902893066

Mem 3000: 29.833023071289

Mem 3500: 30.969757080078

Mem 4000: 30.692306518555

This clearly is on the client, but do not see how free the memory.

Thank you.

[ruflin]

In the past there were some issues with curl leaking / not freeing memory. We should probably create a test case for this to first identify the exact line memory is leaking and to make sure, it does not happen in the future. I assume some of the bulk request or response data is not free. Do you have some tim to dig deeper here in the Elastica code to find the "problematic" lines?