s_push: parser stack overflow

[rixder]

Hi everyone,

I try eap in one of my testing project...
Eap seems to me be excelent job :-)
But today i've got several times:

s_push: parser stack overflow
Traceback (most recent call last):
File "GPTestMarket-04-adf.py", line 737, in <module>
main()
File "GPTestMarket-04-adf.py", line 658, in main
f_open = tools.lambdify(expr=[invalids[i][0]])
File "C:\Python26\lib\site-packages\eap\gp.py", line 85, in
lambdifyList
return lambdify(expr[0].pset, expr[0])
File "C:\Python26\lib\site-packages\eap\gp.py", line 75, in lambdify
return eval(lstr, dict(pset.func_dict))
MemoryError

I'm rather hobbyist in python... How can I solve this error?
Temporarily I changed function lambdify in module gp:

....
try:
return eval(lstr, dict(pset.func_dict))
except MemoryError:
return False

Is this way correct?
Best regards,
riXder

[fmder1]

Hi riXder,

First of all, thanks for using EAP.

The problem you mention is rather a Python limitation than a problem
with EAP. As MAG (a developer of EAP) mentionned in a now closed bug
report on EAP :

The Python parser has a depth limit somewhere between 92
and 99 when parsing an expression. For instance,
eval("("*100+"3"+")"*100) will fail with a MemoryError
exception thrown.

This means that you cannot open more that 92 (or 99) parenthesis
without closing a single one.

I assume that this means that some of your GP trees have a maximum
depth larger than 92 (or 99) nodes. We are currently working on a
different evaluation method for trees that are larger than the Python
parser limit.

An issue about this parser stack limit has already been posted at
http://bugs.python.org/issue1881.

Just as a matter of facts about GPs and really not knowing what you
are trying to do with them, There is alot of work about trying to
control bloat in GP Trees that could help you to limit the size of
your trees and increase the efficiency of your built programs.

I hope I answered your question,
Have fun with EAP,
FM

[Paweł Szurpita]

2010/11/12 fmder1 <f.dera...@gmail.com>

Hi riXder,

First of all, thanks for using EAP.

The problem you mention is rather a Python limitation than a problem
with EAP. As MAG (a developer of EAP) mentionned in a now closed bug
report on EAP :

    The Python parser has a depth limit somewhere between 92
    and 99 when parsing an expression. For instance,
    eval("("*100+"3"+")"*100) will fail with a MemoryError
    exception thrown.

This means that you cannot open more that 92 (or 99) parenthesis
without closing a single one.

I assume that this means that some of your GP trees have a maximum
depth larger than 92 (or 99) nodes. We are currently working on a
different evaluation method for trees that are larger than the Python
parser limit.

An issue about this parser stack limit has already been posted at
http://bugs.python.org/issue1881.

Just as a matter of facts about GPs and really not knowing what you
are trying to do with them, There is alot of work about trying to
control bloat in GP Trees that could help you to limit the size of
your trees and increase the efficiency of your built programs.

I hope I answered your question,
Have fun with EAP,
FM

Hi fmder1...
Thank You for Your response...
I thought so that was Python limitation... Currently I will be happy if I will get no crash my script...
Temporarily I get ride this error ( I hope :-) ) overriding lambdify function:

   .....

   try:
        return eval(lstr, dict(pset.func_dict))
    except MemoryError:

        lstr = "lambda %s: %s" % (args, False)
        return eval(lstr, dict(pset.func_dict))

With Your eap I try to find best rules to enter and exit in forex market... There is a lot of inputs and
a lot of EphemeralConstants and Terminals... Evaluating one generation takes about 10 minutes...
I'm still learning eap and I'm sure that I'll find better solution...  :-)

Best Regards,
riXder

P.S:
Hot news: I've got "parser stack overflow" but my script don't crash :-)

[Marc-André Gardner]

Hi riXder,

I strongly believe you should not use your "lambdify overriding" for
other uses than basic testing purposes : it bypass the normal
evolution process by creating some individuals with a bad fitness.

For instance, if you try to minimize the fitness, all your too big
individuals will have 0 as fitness, and a higher (if not the highest)
probability to be picked for the next generation, even if their real
fitness is actually really bad. This may disturb a lot the evolution,
and select many not-so-good individuals.

The point is that a normal GP evolution should never reach this level
of complexity. Reaching this limit should tell you that the evolution
was anyway unproductive : the trees were so bloated that nothing
really happens anymore (except for really special cases).

Bloat is a recurrent problem in GP; maybe we will include some bloat
controllers in a future release of EAP, but at the moment, you may add
one of the simpler one, proposed by Koza. On standard GP problems, it
works generally well, although more efficient way exists (double
tournament, dynamic equalisation, etc.).
It simply puts a hard limit on the trees depth (usually 17, but you
may change it). You can add it by modifying the evolution algorithms
or the toolbox operators.

If your evolution reachs the maximum level really fast without fitness
improvement, even with the bloat control, well, maybe it's because the
primitives you use do not allow the evolution to create good
individuals : choosing the primitives set is always an headache in
GP...


To speed up the evolution, you may consider to use the Python
multiprocessing library to parallelize the evaluations : an example is
provided with EAP (mpga_onemax.py). It's easy to use and very
efficient on multicore computers.


Hope it'll be useful,

MAG

On 12 nov, 10:15, Paweł Szurpita <pszurp...@gmail.com> wrote:
> 2010/11/12 fmder1 <f.derainvi...@gmail.com>

[Paweł Szurpita]

2010/11/12 Marc-André Gardner <marc.andr...@gmail.com>

Hi Marc-André,
First of all I thank You for Your explanation... You are absolutely right .  I will have to consider params in my GP...
Script crashes when my number of generations > 100 .... So, more not always means better :-)
Thank You again...
Example mpga_onemax.py not working on my computer... :

INFO:eap.algorithms:Start of evolution
Process PoolWorker-2:

Traceback (most recent call last):

  File "C:\PYTHON26\LIB\multiprocessing\process.py", line 231, in _bootstrap
    self.run()
  File "C:\PYTHON26\LIB\multiprocessing\process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "C:\PYTHON26\LIB\multiprocessing\pool.py", line 57, in worker
    task = get()
  File "C:\PYTHON26\LIB\multiprocessing\queues.py", line 339, in get
    return recv()
TypeError: type 'partial' takes at least one argument
Process PoolWorker-4:

Traceback (most recent call last):

  File "C:\PYTHON26\LIB\multiprocessing\process.py", line 231, in _bootstrap
    self.run()
  File "C:\PYTHON26\LIB\multiprocessing\process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "C:\PYTHON26\LIB\multiprocessing\pool.py", line 57, in worker
    task = get()
  File "C:\PYTHON26\LIB\multiprocessing\queues.py", line 339, in get
    return recv()
TypeError: type 'partial' takes at least one argument

Best Regards,
riXder

[Félix-Antoine Fortin]

Hi riXder,

The mpga example requires Python 2.7 to work correctly, since it needs
lambda function to be pickable. If you replace lambda functions by
normally defined, it should works. That said, I do encourage you to go
with Pyhton 2.7, except if you are dealing with non standard module
compatibility issues.

I'm glad you enjoy working with GP in EAP. Don't hesitate to question
us,
Felix-Antoine - live from New-Zealand



On Nov 13, 8:52 pm, Paweł Szurpita <pszurp...@gmail.com> wrote:
> 2010/11/12 Marc-André Gardner <marc.andre.gard...@gmail.com>

[Paweł Szurpita]

2010/11/13 Félix-Antoine Fortin <felix.anto...@gmail.com>

Hi riXder,

The mpga example requires Python 2.7 to work correctly, since it needs
lambda function to be pickable. If you replace lambda functions by
normally defined, it should works. That said, I do encourage you to go
with Pyhton 2.7, except if you are dealing with non standard module
compatibility issues.

I'm glad you enjoy working with GP in EAP. Don't hesitate to question
us,
Felix-Antoine - live from New-Zealand

Hi Félix-Antoine,
Thank You for Your answers and suggestion... I have over a dozen several modules for Python 2.6, which I played with
Artificial Intelligence, but now I see it's time to python 2.7 ... :-)

Best Regards,
riXder