ValueError: object of too small depth for desired array

[matus.s...@privatdemail.net]

Hi,

since I upgraded PyStan from 2.0 to 2.1 after the sampling is done the following sampling message sometimes occurs:

Enter code here...

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-10-cd3da076b43f> in <module>()
----> 1 fit=sm.sampling(data=dat,iter=1000,chains=4,warmup=200,thin=2,n_jobs=2)

/usr/local/lib/python2.7/dist-packages/pystan/model.pyc in sampling(self, data, pars, chains, iter, warmup, thin, seed, init, sample_file, diagnostic_file, verbose, algorithm, control, n_jobs, **kwargs)
    688                                                  m_pars, p_dims)
    689 
--> 690         random_state = np.random.RandomState(args_list[0]['seed'])
    691         perm_lst = [random_state.permutation(int(n_kept)) for _ in range(chains)]
    692         fnames_oi = fit._get_param_fnames_oi()

/usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so in mtrand.RandomState.__init__ (numpy/random/mtrand/mtrand.c:5217)()

/usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so in mtrand.RandomState.seed (numpy/random/mtrand/mtrand.c:5468)()

ValueError: object of too small depth for desired array

It doesn't happen always, I'd say it happens on half of the runs.

I upgraded stan and numpy to the most recent version but the problem still persists.

Any suggestions what might help?

Thanks,

Matus

[Allen Riddell]

Hi Matus,

This is the first time I've seen this error. Thanks for reporting it. Are you
using Linux?

I'm not sure how one can debug this error but I do have a suggestion that might
fix it. If you just set a random seed each time you call `sampling` it's
possible everything will work (it's also not a bad idea in general). For
example,

fit = sm.sampling(data=dat,iter=1000,chains=4,warmup=200,thin=2,n_jobs=2, seed=5)

If the error still occurs, please post the seed you are using.

Best wishes,

-a

> --
> You received this message because you are subscribed to the Google Groups "stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[matus.s...@privatdemail.net]

Yes, Linux.

If I set the seed from python the error doesn't appear. At least I haven't got any error by running

np.random.seed(42)
N=np.random.randint(2**16,size=100)
for n in N:
    print n
    fit=sm.sampling(iter=1000,chains=4,warmup=200,thin=2,n_jobs=2,seed=n)

I will take care to set the seed. That resolves my problem.

Thanks,

Matus

[Bob Carpenter]

Allen --- why would there be an error if a seed is set
and not if one isn't set?

Is there a way to set a default value for a parameter, say
based on the time like we do in RStan? It seems awkward to
have to have users alwyas set seeds.

It's also a good idea to save the seed that gets used, even
if it's generated randomly, so that users can reproduce their
results and we can help them debug issues.

- Bob

On 2/10/14, 1:17 PM, matus.s...@privatdemail.net wrote:
> Yes, Linux.
>
> If I set the seed from python the error doesn't appear. At least I haven't got any error by running
>
> |
> np.random.seed(42)
> N=np.random.randint(2**16,size=100)
> for n in N:
> print n
> fit=sm.sampling(iter=1000,chains=4,warmup=200,thin=2,n_jobs=2,seed=n)
> |
>
> I will take care to set the seed. That resolves my problem.
>
> Thanks,
>
> Matus
>
> On Monday, February 10, 2014 2:48:58 AM UTC+1, Allen Riddell wrote:
>
> Hi Matus,
>
> This is the first time I've seen this error. Thanks for reporting it. Are you
> using Linux?
>
> I'm not sure how one can debug this error but I do have a suggestion that might
> fix it. If you just set a random seed each time you call `sampling` it's
> possible everything will work (it's also not a bad idea in general). For
> example,
>
> fit = sm.sampling(data=dat,iter=1000,chains=4,warmup=200,thin=2,n_jobs=2, seed=5)
>
> If the error still occurs, please post the seed you are using.
>
> Best wishes,
>
> -a
>

> stan-users+...@googlegroups.com <javascript:>.
> > For more options, visit https://groups.google.com/groups/opt_out <https://groups.google.com/groups/opt_out>.

[Allen Riddell]

Hi Bob,

I'm not sure about this error. I wonder if it might have arisen via a mismatch
between some compiled thing (libstan.a or a Cython extension). Naturally I'll
revisit this judgement if someone else has the problem. Fixing the seed was my
band-aid solution.

Just like RStan, PyStan does save the seed that is used -- but we can only get
access to this seed if the sampler completes and returns a fit object.

Best,

-a

[Ken Arnold]

A very strange error. It seems to happen when RandomState is passed strange datatypes; I got it to say that for passing it a string. Could you possibly add `print args_list[0]['seed']` before the offending line?

-Ken

[Bob Carpenter]

I understand. The term "band-aid" just raises a big red
flag for me --- seems like it might be indicative of deeper
problems.

Out of curiosity, what gave you reason to believe that adding a seed
explicitly would fix the problem?

Forgive me if this is a naive question --- I'm not used to running in interpreted
environments like Python or R.

- Bob

[Allen Riddell]

Hi Bob,

re: the seed -- if you glance at the reported error, Python is telling us it got
passed something wacky as a seed and it is giving up. My thinking was that if
the user provides an integer as a seed then Python (or whatever) will not
provide a wacky one.

I still think this is likely a mismatch between compiled objects -- i.e.,
there's a compiled Cython extension that's maybe version 2.0 and libstan is 2.1
or something like that. In this particular case, I'm OK with a band-aid solution
since it seems next to impossible to debug without access to the affected
filesystem.

[matus.s...@privatdemail.net]

I added

print type(args_list[0]['seed']),args_list[0]['seed']

and got 

<type 'int'> 1372763590

when the command returned succesfully, but I got

<type 'long'> 3853325772

when the error occured. I looped the sampling command within try-except block and the error comes up exclusively when np.random.RandomState is passed type 'long' argument. 

Hope this helps,

Matus