Attempting to use the Python tools of the pencil-code.

[Christer Sandin]

Dear All,

I am trying to analyze the calculated pencil models and I would like

to use Python to do this.

I am following the manual, which appears to contain the same information as the

web site https://github.com/pencil-code/pencil-code/wiki/PythonForPencil.

So, I've set up ipython to make use of Python 2.7 on our cluster (SNIC / PDC / Tegnér).

That is, at first:

===

# Python:
export PENCIL_HOME=/cfs/klemming/nobackup/c/csandin/pencil-code
export PYTHONPATH=$PENCIL_HOME/python

===

I am following the instructions that appear on the command line, and it appears

I should use "pencil_old". I have set the following files:

~/.ipython/ipythonrc ===

import_all pencil_old # Or should this always be just "pencil"?

============

~/.ipython/profile_default/startup/init.py ===

import numpy as np
import pylab as plt
import pencil_old as pc

import matplotlib
from matplotlib import rc

plt.ion()

matplotlib.rcParams['savefig.directory'] = ''
=================================

Doing this, I get the following when I launch ipython:

===

[csandin@tegner-login-1 isothermal_expwave_512_f4.0_dtf1.0]$ ipython
Python 2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jul  2 2016, 17:42:40)
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
Warning: pencilnew has moved to pencil.
         pencil has moved to pencil_old.
To change your scripts accordingly:
import pencilnew as pc -> import pencil as pc
import pencil as pc -> import pencil_old as pc
Warning: pencilnew has moved to pencil.
         pencil has moved to pencil_old.
To change your scripts accordingly:
import pencilnew as pc -> import pencil as pc
import pencil as pc -> import pencil_old as pc
[TerminalIPythonApp] WARNING | Unknown error in handling startup files:
  File "/cfs/klemming/nobackup/c/csandin/pencil-code/python/pencil/io/snapshot.py", line 690
    print('rank', rank, 'saving '+filename, flush=True)
                                                 ^
SyntaxError: invalid syntax


In [1]:
====

So, while I explicitly am attempting to use pencil_old, the tool uses files from "pencil"; this

does not work. What can I do?

(I'd like to use Python 2.7 as Lars M. here at Nordita has some tools setup to

work with Python 2.7).

Kind Regards,

 Christer

[Frederick Gent]

Hi Christer,

Please pull again. I have amended the print statements to be backwards compatable with Python2. I recommend using import pencil as pc, rather than pencil_old as pc. The new Python3 compatable modules are also better designed and integrated than the old stand alone scripts and will be supported going forward. They should also work with Python2, but I missed that flush=True is not available for Python2. I also recommend using Python3, it might not take many changes to make Lars' scripts Python3 friendly - print() being the most common culprit.

With pencil vs pencil_old the main changes in read scripts are that they will take the form pc.read.var() rather than pc.read_var(), etc., and the structures for the averages will now be more consistent with av.t, av.xy.__, av.xz.__, or for 2D averages av.z.__. VAR objects are the same and time_series objects, but video slices are a single object like the averages, etc.

Best wishes,

Fred

--
You received this message because you are subscribed to the Google Groups "pencil-code-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pencil-code-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pencil-code-discuss/d228fefb-dcbd-495a-9809-abe3bff010a3%40googlegroups.com.

[Christer Sandin]

Hi Frederick,

Thank You. I shall likely switch to Python 3 as well.

I think it would be worthwhile to update the Python

section in the manual as well, to reflect the use of

the new variable structures (in oarticular for newbies

like me).

Best Wishes,

 Christer

[Christer Sandin]

Hi Frederick,

so it seems I have two new issues regarding the use of Python + pencil-code:

o using "import pencil_old" (Python 2.7):

 There seem to be issues here as well:

 ===

In [1]: run ../../pyscripts/pc_plot_particles_PVARX_2.7
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/cfs/klemming/nobackup/c/csandin/pencil-code/christer/pyscripts/pc_plot_particles_PVARX_2.7.py in <module>()
     14
     15 # Particle positions:
---> 16 pdata=pc.read_pvar('PVAR41')
     17 #pdata=pc.read_pvar()
     18 hmax = 0

/cfs/klemming/nobackup/c/csandin/pencil-code/python/pencil_old/files/read_pvar.pyc in read_pvar(*args, **kwargs)
     27                                                 using pc_copyvar to copy and subsequently rename the reduced particle files for new runs
     28     """
---> 29         return pcpvar(*args, **kwargs)
     30
     31 class pcpvar(object):

/cfs/klemming/nobackup/c/csandin/pencil-code/python/pencil_old/files/read_pvar.pyc in __init__(self, varfile, datadir, proc, verbose, reduce_to)
     46                                 nprocs=len(procdirs)
     47                 ipars,pvars = collect_class_pdata(pfile=varfile,datadir=datadir,nprocs=nprocs,
---> 48                     verbose=verbose,reduce_to=reduce_to)
     49                         else:
     50                 ipars,pvars = read_class_npvar_red(datadir=datadir,proc=proc,

/cfs/klemming/nobackup/c/csandin/pencil-code/python/pencil_old/files/read_pvar.pyc in collect_class_pdata(pfile, datadir, nprocs, verbose, reduce_to)
    208         for i in procs:
    209                 if (not i==procs[-1]):
--> 210                         dom_ipar,dom_pvar, written_parts = read_class_npvar_red(pfile=pfile,datadir=datadir,proc=i,reduce_to=reduce_to,verbose=verbose)
    211                         if (reduce_to>0):
    212                                 parts_to_write -= written_parts

/cfs/klemming/nobackup/c/csandin/pencil-code/python/pencil_old/files/read_pvar.pyc in read_class_npvar_red(datadir, pfile, proc, verbose, reduce_to, set_reduce)
     94     set_reduce=-1):
     95
---> 96         dims = pc.read_dim(datadir,proc)
     97         pdims = pc.read_pdim(datadir)
     98         npar_loc = read_npar_loc(datadir=datadir,pfile=pfile,proc=proc)

AttributeError: 'module' object has no attribute 'read_dim'
===

o using "import pencil" (Python 3.6):

  It appears that some routines rely on the IDL Python bridge to read the data.

  What is good to know is which version of IDL one needs to look at these data.

  Because, currently used Python versions are only available in recent versions

  of IDL. For example, IDL version 8.5.1 contains files for Python versions 2.7

  (pencil_old) and 3.4 (which is obsolete).

  Here is some of the output I get when trying to read particle data using

  "import pencil":

===

In [5]: ?pdata
Type:           ParticleData
String form:    <pencil.read.pvar.ParticleData object at 0x7fe0a57dd0f0>
File:           /cfs/klemming/nobackup/c/csandin/pencil-code/python/pencil/read/pvar.py
Docstring:    
Read PVAR files from Pencil Code using IDL.
Uses IDL<->Python Bridge, this must be activated manually!

!! WARNING: SHAPE IS AS IN IDL: (X, Y, Z) !!

Args:
    - datadir      specify datadir, default False
    - sim           specify simulation from which you want to read
    - varfile       put 'PVARXYZ' or just number here, 'VAR' will be replaced by 'PVAR' autom.
    - npar_max      maximal number of particles to be read in

    - proc          read from single proc, set number here
    - swap_endian   change if needed to True, default False
    - quiet         verbosity, default False

If needed add manually to this script:
    - rmv, irmv, trmv, oldrmv are used for ???
    - solid_object is used for ???
    - theta_arr is used for ???
    - savefile is used for ???
Init docstring:
Read PVAR files from Pencil Code using IDL.
Uses IDL<->Python Bridge, this must be activated manually!
===

Hmm...where do I start?

Best Wishes and thanks for any help,

 Christer

Den onsdag 5 februari 2020 kl. 22:15:40 UTC+1 skrev Frederick Gent:

[SimonC (Iomsn)]

Hi Christer,

The particle reading routines were implemented using an IDL bridge, which makes use if the IDL reading routines. Currently there is no native implementation of this. However, if you have IDL installed on the system it should work.

Cheers

Simon

[Christer Sandin]

Hi Simon,

yes, you're of course right, but there are some restrictions.

I think it could be worthwhile to add to the documentation

what the exact requirements are. For simplicity, here they

are:

o "pencil_old" uses Python 2.7 and can be used with IDL >= 8.5.

 However, "pencil_old" appears to be broken and is unmaintained.

o pencil[_new] uses Python 3 and can be used with:

 IDL <= 8.4 ::

  Nope. Unavailable functionality.

 IDL 8.5: Python 3.4 ::

  This version of Python must be compiled from source

  or installed using virtualenv (or similar), unless one has

  an old installation on the computer.

 IDL 8.6: Python 3.4 or Python 3.5 ::

  Python 3.5 is still used.

 IDL 8.7: Python 3.5 or Python 3.6 ::

  Most convenient, currently, if one has access to this version of IDL.

Best,

 Christer

[Christer Sandin]

My attempt to install Python 3.4 and then Virtualenv on my Ubuntu 18.04 has failed

(as Python is unable to install the SSL module, which is required by Virtualenv).

I could perhaps spend time on this to make it work.

However, as I see it I have two options:

 o Wait for IDL 8.7.2 or even IDL 8.7.3 to become available at my workstation.

 o Use IDL, without Python. I think I will have to go for this option.

C.

[Christer Sandin]

As Xiang-Yu has insisted that it works for him using "pencil_old",

I determined that it must work for me as well. And, after replacing

all "import pencil" with "import pencil_old" in all files the directory

"python/pencil_old/files/", it works!

I have committed these changes to have this work for anybody else

who's interested in using the Python particle tools.

C.

[Frederick Gent]

Dear all,

I am starting to use the particles functionality of the Pencil Code. In order to work with this using Python, I have revised the reading routines, particularly pc.read.pvar(), to work without the need of the IDL function calls. If you have been working with particles and Python, please check that the new module provides the necessary functionality for the particles and the old IDL wrapping version. If there are any errors or missing functionality please let me know and I shall try to address it. The old module has been committed as pvarfile.py.bak, so can still be used if the new module causes difficulties.

Best regards,

Fred