Python and IDL, read PVAR files from specific processor

[Xiangyu LI]

Dear all,

When I tried to read PVAR files from a specific processor, I got the following error,

$IDL

IDL> pc_read_pvar,obj=pvar,ivar=10,proc=2

% CREATE_STRUCT: Variable is undefined: IIPAR.

% PC_READ_PVAR: ERROR Evaluating variables: object =

                 create_struct(name=objectname,['t','x','y','z','dx','dy','dz

                ','npar_found','ipar','xx','vv','a','npswarm'],t,x,y,z,dx,dy,

                dz,npar_found,iipar,xx,vv,a,npswarm)

Then I convert "pc_read_pvar.pro" to the revision 73989, it works.

Any idea about this?

For python,

In [2]: pvar=pc.read_pvar('PVAR10', proc=0)

ValueError: too many values to unpack (expected 2)

Python users, any about this?

Best regards,

Xiangyu

 

[Philippe-A. Bourdin]

Hello Xiangyu,

regarding the IDL problem: could you try a binary search in the revision numbers and find the revision that causes the error?

Thanks and best greetings,
Philippe.

[Xiangyu LI]

Hi Phillippe,

Yes, I found that this version 74448 of "pc_read_pvar" causes the problem.

Xiangyu

[SimonC (Iomsn)]

Hi Xiangyu,

considering that both, IDL and Python give you some errors, it could be that the binary format for the PVAR files changed without the routines being updated. If you use a new version of the read routines on old that the same could have happened.

Can you confirm that new reading routines on new data does not work?

The Python error is typical for reading in larger files than the routines expects.

Cheers

Simon

[Xiangyu LI]

Hi Simon,

Many thanks for your reply!

I tried to use the new Python routine,

import pencilnew as pcn

pvar=pcn.read.pvar('PVAR10', proc=0)

! ERROR: no idl<->python bridge found. Try whats written in pstalk-comment to fix that issue.

! 

! Use something like: (enshure you have IDL 8.5.1 or larger)

! export PYTHONPATH=$PYTHONPATH:$IDL_HOME/lib/bridges:$IDL_HOME/bin/bin.linux.x86_64

! export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64:$IDL_HOME/bin/bin.linux.x86_64

! in your .bashrc

! 

It seems that Python need "idl<->python bridge". But I do not have IDL 8.5.1 or higher.

I guess Python should be capable of reading binary files without any bridge.

Could you please kindly remind me why the bridge is needed?

For the longstanding point of view, Python routines should be completely independent from IDL

for post-processing. 

Cheers,

Xiangyu

[Andreas Schreiber]

Hi Xiangyu,

I tried to use the new Python routine,

import pencilnew as pcn

pvar=pcn.read.pvar('PVAR10', proc=0)

! ERROR: no idl<->python bridge found. Try whats written in pstalk-comment to fix that issue.

! 

! Use something like: (enshure you have IDL 8.5.1 or larger)

! export PYTHONPATH=$PYTHONPATH:$IDL_HOME/lib/bridges:$IDL_HOME/bin/bin.linux.x86_64

! export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64:$IDL_HOME/bin/bin.linux.x86_64

! in your .bashrc

! 

It seems that Python need "idl<->python bridge". But I do not have IDL 8.5.1 or higher.

I guess Python should be capable of reading binary files without any bridge.

Could you please kindly remind me why the bridge is needed?

IDL comes along with an Python module that lets you call and fully access an IDL session, i.e. you can use all existing IDL functionality and directly access the IDL variables insitu in memory. 

This is very powerfull and helps us to use old IDL scripts from Python. One of them is the pvar read routine. Reasons are many

a) IDL routines still tend to be more up to date

b) writing own reading routines that cover all possible pencil code extentions and special cases is very hard work and needs a very broad understanding of the output format, since the output is not self-explaining at all. Reading binary is not the problem, understanding binary is!

c) one does not want to wait until an routine has been ported to Python, but instead use the existing functionallity. One wants to work...

For the longstanding point of view, Python routines should be completely independent from IDL

for post-processing. 

Yes, please go for it!

Also check out the script from Colin: pencil-code/utils/readstalker_parallel.py

Additionally, I remember to have implemented an backup solution via pidly. Check out of this is existing, i.e. you have the right code version, or use that module yourself. I.e., worst case: use pidly to read whatevery you need into idl an store it in an .sav file. There is an sav file importer in the scipy module. That is not nice, but works....  To speed up, check out how Colin uses h5py for the best data in file storage one can have!

best wishes

Andreas

------------------------------

Andreas Schreiber

Max-Planck-Institute for Astronomy

Koenigstuhl 17

D-69117 Heidelberg, Germany

Email: aschr...@mpia.de

Phone: +49-6221-528-392

Room: E 114 (Elsässer Lab)

http://www.mpia.de

--
You received this message because you are subscribed to the Google Groups "pencil-code-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pencil-code-dis...@googlegroups.com.
To post to this group, send email to pencil-co...@googlegroups.com.
Visit this group at https://groups.google.com/group/pencil-code-discuss.
For more options, visit https://groups.google.com/d/optout.

[Xiangyu LI]

Hi Andreas,

Many thanks for your help.

I am now trying to find a higher version of IDL(higher than 8.5) to have the bridge.

Best regards,

Xiangu 

To unsubscribe from this group and stop receiving emails from it, send an email to pencil-code-discuss+unsub...@googlegroups.com.

[Andreas Schreiber]

Hi Xiangu,

good, tell us your experience and I hope you soon start to contribute to the pencilnew module!

best

Andreas

------------------------------

Andreas Schreiber

Max-Planck-Institute for Astronomy

Koenigstuhl 17

D-69117 Heidelberg, Germany

Email: aschr...@mpia.de

Phone: +49-6221-528-392

Room: E 114 (Elsässer Lab)

To unsubscribe from this group and stop receiving emails from it, send an email to pencil-code-dis...@googlegroups.com.