Limits in the number of choice options

[Mahdi Ben Jelloul]

Dear liam2 users,

I hope you are all well in this covid period.

I use choice with a large number (> 180) of options and I got a MemoryError form numexpr (I am using Liam2 0.12a version) with python 2.7.

choice(
[
-60,
-59,-58,-57,-56,-55,-54,-53,-52,-51,-50,
-49,-48,-47,-46,-45,-44,-43,-42,-41,-40,
-39,-38,-37,-36,-35,-34,-33,-32,-31,-30,
-29,-28,-27,-26,-25,-24,-23,-22,-21,-20,
-19,-18,-17,-16,-15,-14,-13,-12,-11,-10,
-9,-8,-7,-6,-5,-4,-3,-2,-1,
0,1,2,3,4,5,6,7,8,9,
10,11,12,13,14,15,16,17,18,19,
20,21,22,23,24,25,26,27,28,29,
30,31,32,33,34,35,36,37,38,39,
40,41,42,43,44,45,46,47,48,49,
50,51,52,53,54,55,56,57,58,59,
60,61,62,63,64,65,66,67,68,69,
70,71,72,73,74,75,76,77,78,79,
80,81,82,83,84,85,86,87,88,89,
90,91,92,93,94,95,96,97,98,99,
100,101,102,103,104,105,106,107,108,109,
110,111,112,113,117,119,
120,121,124,125,126,127
],
array(depart_taux_plein_mois_decote_surcote[1 * sexe, decile_trimestres_manquants]).transpose(),
)

Is there a clear limit on the number of options ?

How could I achieve such a draw in liam2 in a different manner ?

I can code a recursive bisect like draw in liam2 but I am trying to avoid the hassle of coding that. 

Thank you !

All the best,

Mahdi Ben Jelloul

[Gaëtan de Menten]

Hello Mahdi,

 

A MemoryError is surprising to me, but some kind of error is to be expected in that case (when the probabilities are not constants but vary depending on the individual) because, in that case, the choice is implemented as something like:

 

u: uniform()

if(u < proba1, outcome1,
   if(u < proba1+proba2, outcome2,

     …

     outcomeN))

 

This means you will hit numexpr's limitations (31 different unique variables and 255 registers – variables or temporaries).

See https://github.com/pydata/numexpr/issues/167 for a bit more details.

 

One thing you could try is to transform your choice to the above expression manually, so you can split it as needed.

I admit this would be awfully long and ugly but I don't have another idea which does not involve modifying LIAM2's code at this point. That said, you could also try to modify liam2 code to automatically split these expression when they become too large (doing this in all generality might be hard, but doing just for these choice expressions should be relatively easy).

 

Sadly this whole translation is necessary in the first place because np.digitize does not support arrays for bins boundaries (i.e. have different bins per individuals). The best solution would of course be to implement (in cython or numba) a digitize-like function supporting arrays as bounds but that might require more work than you are ready to put into this 😉.

 

Hope it helps,

Gaëtan

 

--
You received this message because you are subscribed to the Google Groups "liam2-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liam2-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/liam2-users/CA%2BAkwSff%2B8brNrCRs%3DJV96e9tGZmYqHPTk66URNQfzyKHn0ecw%40mail.gmail.com.

Disclaimer: This e-mail may contain confidential information which is intended only for the use of the recipient(s) named above.
If you have received this communication in error, please notify the sender immediately and delete this e-mail from your system.
Please note that e-mail messages cannot be considered as official information from the Federal Planning Bureau.