About of PyRods and iRODS 4.0 and python-irods client

[Bernardo Augusto García Loaiza]

Hi Jerome

Receive you a warm greeting.

I want ask to you the help and orientation to the following  answers to these questions:

• The iRODS Python project [ https://code.google.com/p/irodspython/ ], the PyRODS API, the last version is the 3.3.6 ?  [ such as describe here --- https://code.google.com/p/irodspython/wiki/Downloads?tm=2 ]
  

In affirmative case, currently not exist a PyRODS version for iRODS 4.0? 

In a previous opportunnity , you told me that you are working in a new PyRods version for iRODS 4.0 ... 

Do you know in that state stay this idea or project that you did have in mind for the community?

Currently, there is a PyRODS version for iRODS 4.0 or upper?

Or I should use PyRODS 3.3.6 as a last version for iRODS 3.3.1

• What are the differences between iRODS Python project [ https://code.google.com/p/irodspython/ ],  which include the PyRODS API  and the python-irods client hosted on github [ https://github.com/irods/python-irodsclient ] and linked from wiki.irods.org ? Are these API Clients the same? 

I see that the  iRODS Python project [ https://code.google.com/p/irodspython/ ], include PyRODS, EmbedPython and PyRODS Admin ... and the python-irods client is more specific such as we can see in this link https://github.com/irods/python-irodsclient/blob/master/README.md 

Is python-irods client similar or related to PyRODS of iRODS Python project ?

I have some doubts about of what of them use for my purpose of build a application on iRODS 3.3.6 deployment datagrid that I have.

I appreciate your help and everyone.

Greetings

[de Torcy, Antoine]

Greetings Bernardo,

Regarding https://github.com/irods/python-irodsclient, this is a fork of a native Python API to iRODS originally developed at iPlant and currently supported by the iRODS consortium.
For testing and coverage info see: http://ci-dev.renci.org/hudson/view/iRODS/job/irods-python-irodsclient-buildandtest-all/
Feel free to ask if you have any specific questions about it.
Thanks,

Antoine de Torcy
RENCI

---

From: irod...@googlegroups.com [irod...@googlegroups.com] on behalf of Bernardo Augusto García Loaiza [boti...@gmail.com]
Sent: Wednesday, April 15, 2015 4:26 PM
To: Jérôme Fuselier; irod-chat
Subject: [iROD-Chat:13549] About of PyRods and iRODS 4.0 and python-irods client

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bernardo Augusto García Loaiza]

Thanks for the information Antoine. 

I want know, in comparision with PyRODS, python-irodsclient support more functions on iRODS data grid? 

python-irodsclient is iseful for one iRODS version specifically or any version?

I have an iRODS 3.3.2 deployment data grid, and I have been thinking in migrate to iRODS 4.0.3, althought I will use the API in python, then I am not sure about it.

PyRODS 3.3.6 works with iRODS 3.3.2 and so..

In the readme file of python-irodsclient repository I see many functions about of how to works the api

https://github.com/irods/python-irodsclient/blob/master/README.md 

I guess that the unchecked box are the functions that python-irodsclient does not support .. really?

What relation have python-irodsclient and PyRODS? 

I guess that together are two differentes clients native python api's for access and work with iRODS data grid ... 

Is this right?

Thanks for the orientation

[de Torcy, Antoine]

You are correct, they are two separate and independent efforts. I have not used PyRODS recently so I don't know how their feature sets compare, but what you see in the README is an up to date list of what is currently supported in python-irodsclient.
Do you have a particular use case in mind or list of features that you need?

Antoine

---

From: irod...@googlegroups.com [irod...@googlegroups.com] on behalf of Bernardo Augusto García Loaiza [boti...@gmail.com]

Sent: Wednesday, April 15, 2015 6:37 PM
To: irod...@googlegroups.com
Subject: Re: [iROD-Chat:13551] About of PyRods and iRODS 4.0 and python-irods client

[Jérôme Fuselier]

Hi Bernardo,

I've been working for a while on a PyRods version for iRODS 4.x but the modifications that have been done to the code are not making things really easy for PyRods approach. It's quite unlikely that I manage to provide a "production-ready" version in a near future as I'm not involved anymore in iRODS related projects.

From what I've read the python-irodsclient uses iRODS messages to communicate with the server. That's probably the way I would have chosen too if I had the API documentation at the time I started working on PyRods. It's probably your best bet if you plan to migrate to iRODS 4.x anytime soon.

Best
Jerome

[Bernardo Augusto García Loaiza]

Jerome, thanks for your attention. I will see more about of python-irodsclient and follow with PyRods with the I already made.
Greetings :)

[Bernardo Augusto García Loaiza]

Hi Antoine thanks for the responses and orientation

I want to know then. ..

python-irodsclient [ https://github.com/irods/python-irodsclient/blob/master/README.md ] workfor iRODS 4.0.3 currently?

If I have an iRODS 4.03 deployment, can I work eith python-irodsclient native API?

I want build a web application using iRODS infraestructure (currently I have the iRODS 3.3.2 deployment in mys servers)

Some of the requirements that I want build in my web application are:

• Access to the iRODS infraestructure with the native iRODS challenge/response (password) enabled
  

(Is possible with other authetication methods?)

• The users can view the dataresources created and create news dataresources
• The users can view the iRODS collections and create new
  
• Perform searchs of files in the iRODS infraestructure.
  
• Add metadata to the iRODS files that upload via web application 
  

Is possible run python-irodsclient sucessfuly in Django web framework? 

I'll think that it's affirmative due to install with pip, python-irodsclient will be one application more in a DjangoProject or even I can import their libraries   .. really?

Thanks for the feedback and orientation.

Best Regards

[de Torcy, Antoine]

Based on what you listed you should be good to go. Python-irodsclient supports manipulation of files and collections, resources, metadata, and exposes the general query API.
For now only iRODS' native authentication is supported.
To get started with it you can install it somewhere (I like virtualenv personally) and point it at your grid and see if you can list your files and collections, and run queries (you can look at the code under https://github.com/irods/python-irodsclient/tree/master/irods/test for examples). Or if you have a grid for development/testing you can also try running the test suite against it, and go from there.

Antoine

---

From: irod...@googlegroups.com [irod...@googlegroups.com] on behalf of Bernardo Augusto García Loaiza [boti...@gmail.com]

Sent: Thursday, April 16, 2015 2:35 PM
To: irod...@googlegroups.com
Subject: Re: [iROD-Chat:13555] About of PyRods and iRODS 4.0 and python-irods client

[Bernardo García]

Hi Antoine

python-irodsclient only work with python 2.7 ? --- https://github.com/irods/python-irodsclient/blob/master/README.md 

The test samples too? https://github.com/irods/python-irodsclient/tree/master/irods/test

Best Regards

[de Torcy, Antoine]

Hi Bernardo,

The test suite currently passes on CentOS 6.3 with python 2.6.6, and Ubuntu 12 with python 2.7.3

Antoine

---

From: irod...@googlegroups.com [irod...@googlegroups.com] on behalf of Bernardo García [boti...@gmail.com]
Sent: Thursday, May 07, 2015 3:58 PM
To: irod...@googlegroups.com
Subject: Re: [iROD-Chat:13741] About of PyRods and iRODS 4.0 and python-irods client

[Bernardo García]

 I have Ubuntu 14.04 with python 2.7.6 .... it's works really? 

How to test the samples originals, not the unit tests?

I send another mail with other questions 

  Best Regards

[de Torcy, Antoine]

Hi Bernardo,

I will try to answer your questions about how to use the python client with an example. This is on an Ubuntu 14.04 VM with Python 2.7.6 and virtualenv. I'll copy-paste my terminal session.
I am using an existing account on a remote server. This account has test data in it (iRODS source files).
In the sequence below I install python-irodsclient from git and try a few commands, as in README.md.
In particular I:
* Create an iRODS session object, using a valid account
* List the collections in the user's home collection
* List the objects (files) in another collection
* Run a query and display its results


(testenv)antoine@ub14:~/testenv$
(testenv)antoine@ub14:~/testenv$
(testenv)antoine@ub14:~/testenv$ pip install git+git://github.com/irods/python-irodsclient.git
Collecting git+git://github.com/irods/python-irodsclient.git
  Cloning git://github.com/irods/python-irodsclient.git to /tmp/pip-9sUGSi-build
Collecting PrettyTable>=0.7.2 (from python-irodsclient==0.4.0)
  Downloading prettytable-0.7.2.tar.bz2
Installing collected packages: PrettyTable, python-irodsclient
  Running setup.py install for PrettyTable
  Running setup.py install for python-irodsclient
Successfully installed PrettyTable-0.7.2 python-irodsclient-0.4.0
(testenv)antoine@ub14:~/testenv$
(testenv)antoine@ub14:~/testenv$
(testenv)antoine@ub14:~/testenv$
(testenv)antoine@ub14:~/testenv$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from irods.session import iRODSSession
>>> sess  = iRODSSession(host='XXXXXXXX.renci.org', port=1247, user='rods', password='XXXXXXXXX', zone='dev')
>>>
>>>
>>> coll = sess.collections.get('/dev/home/rods')
>>> coll.id
9027
>>> coll.path
'/dev/home/rods'
>>>
>>> for subcoll in coll.subcollections:
...     print subcoll
...
<iRODSCollection 12640 tests>
>>>
>>>
>>>
>>>
>>>
>>> coll = sess.collections.get('/dev/home/rods/tests/pydevtest')
>>>
>>>
>>> for object in coll.data_objects:
...     print object.name
...
auth_suite.py
catalog_suite.py
cleanup_resource_tree.py
compatibility_suite.py
configuration.py
configuration.pyc
iadmin_suite.py
irods_session.py
irods_session.pyc
lib.py
lib.pyc
make_resource_tree.py
metaclass_unittest_test_case_generator.py
README
resource_suite.py
rule1_2242.r
rule2_2242.r
rulebase_fastswap_test_2276.sh
rulebase_suite.py
run_tests.py
test_allrules.py
test_chunkydevtest.py
test_control_plane.py
test_federation.py
test_federation.pyc
test_fuse_suite.py
test_icommands_file_operations.py
test_imeta_set.py
test_ireg_suite.py
test_iscan.py
test_load_balanced_suite.py
test_mso_suite.py
test_resource_tree.py
test_resource_types.py
test_xmsg.py
ticket_suite.py
>>>
>>>
>>>
>>> from irods.models import Collection, DataObject
>>>
>>> query_results = sess.query(Collection.name, DataObject.name, DataObject.size, DataObject.owner_name).all()
>>>
>>> print query_results
+--------------------------------+-------------------------------------------+-----------+--------------+
| COLL_NAME                      | DATA_NAME                                 | DATA_SIZE | D_OWNER_NAME |
+--------------------------------+-------------------------------------------+-----------+--------------+
| /dev/home/rods#community       | bigfile                                   | 67108864  | rods         |
| /dev/home/rods/tests           | localhost_setup_postgres.input            | 45        | rods         |
| /dev/home/rods/tests           | zombiereaper.sh                           | 570       | rods         |
| /dev/home/rods/tests/pydevtest | auth_suite.py                             | 7253      | rods         |
| /dev/home/rods/tests/pydevtest | catalog_suite.py                          | 9866      | rods         |
| /dev/home/rods/tests/pydevtest | cleanup_resource_tree.py                  | 1022      | rods         |
| /dev/home/rods/tests/pydevtest | compatibility_suite.py                    | 2307      | rods         |
| /dev/home/rods/tests/pydevtest | configuration.py                          | 226       | rods         |
| /dev/home/rods/tests/pydevtest | configuration.pyc                         | 425       | rods         |
| /dev/home/rods/tests/pydevtest | iadmin_suite.py                           | 56099     | rods         |
| /dev/home/rods/tests/pydevtest | irods_session.py                          | 6384      | rods         |
| /dev/home/rods/tests/pydevtest | irods_session.pyc                         | 7857      | rods         |
| /dev/home/rods/tests/pydevtest | lib.py                                    | 14240     | rods         |
| /dev/home/rods/tests/pydevtest | lib.pyc                                   | 16891     | rods         |
| /dev/home/rods/tests/pydevtest | make_resource_tree.py                     | 3417      | rods         |
| /dev/home/rods/tests/pydevtest | metaclass_unittest_test_case_generator.py | 439       | rods         |
| /dev/home/rods/tests/pydevtest | README                                    | 3325      | rods         |
| /dev/home/rods/tests/pydevtest | resource_suite.py                         | 50955     | rods         |
| /dev/home/rods/tests/pydevtest | rule1_2242.r                              | 80        | rods         |
| /dev/home/rods/tests/pydevtest | rule2_2242.r                              | 160       | rods         |
| /dev/home/rods/tests/pydevtest | rulebase_fastswap_test_2276.sh            | 1165      | rods         |
| /dev/home/rods/tests/pydevtest | rulebase_suite.py                         | 4475      | rods         |
| /dev/home/rods/tests/pydevtest | run_tests.py                              | 4016      | rods         |
| /dev/home/rods/tests/pydevtest | test_allrules.py                          | 18210     | rods         |
| /dev/home/rods/tests/pydevtest | test_chunkydevtest.py                     | 44811     | rods         |
| /dev/home/rods/tests/pydevtest | test_control_plane.py                     | 1231      | rods         |
| /dev/home/rods/tests/pydevtest | test_federation.py                        | 19758     | rods         |
| /dev/home/rods/tests/pydevtest | test_federation.pyc                       | 13863     | rods         |
| /dev/home/rods/tests/pydevtest | test_fuse_suite.py                        | 5161      | rods         |
| /dev/home/rods/tests/pydevtest | test_icommands_file_operations.py         | 12482     | rods         |
| /dev/home/rods/tests/pydevtest | test_imeta_set.py                         | 6443      | rods         |
| /dev/home/rods/tests/pydevtest | test_ireg_suite.py                        | 2947      | rods         |
| /dev/home/rods/tests/pydevtest | test_iscan.py                             | 3864      | rods         |
| /dev/home/rods/tests/pydevtest | test_load_balanced_suite.py               | 5087      | rods         |
| /dev/home/rods/tests/pydevtest | test_mso_suite.py                         | 4345      | rods         |
| /dev/home/rods/tests/pydevtest | test_resource_tree.py                     | 838       | rods         |
| /dev/home/rods/tests/pydevtest | test_resource_types.py                    | 204544    | rods         |
| /dev/home/rods/tests/pydevtest | test_xmsg.py                              | 2068      | rods         |
| /dev/home/rods/tests/pydevtest | ticket_suite.py                           | 565       | rods         |
+--------------------------------+-------------------------------------------+-----------+--------------+
>>> quit()
(testenv)antoine@ub14:~/testenv$


Possible prerequisites: If you get an SSL error ("SSLContext object not available...") when making pip install requests to git you may fix it by upgrading your Python to 2.7.9 or installing extra packages as follows:

antoine@ub14 ~> sudo apt-get install libffi-dev libssl-dev

(testenv)antoine@ub14:~/testenv$
(testenv)antoine@ub14:~/testenv$ pip install pyopenssl ndg-httpsclient pyasn1
Collecting pyopenssl
  Using cached pyOpenSSL-0.15.1-py2.py3-none-any.whl
Collecting ndg-httpsclient
  Using cached ndg_httpsclient-0.4.0.tar.gz
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in ./lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in ./lib/python2.7/site-packages (from pyopenssl)
Collecting cryptography>=0.7 (from pyopenssl)
  Using cached cryptography-0.8.2.tar.gz
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python2.7/site-packages (from cryptography>=0.7->pyopenssl)
Requirement already satisfied (use --upgrade to upgrade): enum34 in ./lib/python2.7/site-packages (from cryptography>=0.7->pyopenssl)
Collecting cffi>=0.8 (from cryptography>=0.7->pyopenssl)
  Using cached cffi-0.9.2.tar.gz
Requirement already satisfied (use --upgrade to upgrade): pycparser in ./lib/python2.7/site-packages (from cffi>=0.8->cryptography>=0.7->pyopenssl)
Installing collected packages: cffi, cryptography, pyopenssl, ndg-httpsclient
  Running setup.py install for cffi
  Running setup.py install for cryptography
  Running setup.py install for ndg-httpsclient
Successfully installed cffi-0.9.2 cryptography-0.8.2 ndg-httpsclient-0.4.0 pyopenssl-0.15.1
(testenv)antoine@ub14:~/testenv$
(testenv)antoine@ub14:~/testenv$

I hope this helps you get started. Beyond that you can look at the test suite code for example of how to use metadata, administrative features, queries, et...

Antoine

---

From: irod...@googlegroups.com [irod...@googlegroups.com] on behalf of Bernardo García [boti...@gmail.com]

Sent: Thursday, May 07, 2015 11:54 PM
To: irod...@googlegroups.com
Subject: Re: [iROD-Chat:13748] About of PyRods and iRODS 4.0 and python-irods client

[Bernardo García]

Hi ANtoine, 

Thanks a lot to you. 

The python-irodsclient already works for me, I had some problems with my virtualenv and for this reason the interpreter don't take the instructions when I try connect to iRODS Server.

Now, this works form me

I just had a problem executing a query for their results:

>>> from irods.models import Collection, DataObject 

>>> query_results = session.query(Collection.name, DataObject.name, DataObject.size, DataObject.owner_name).all()

>>> print query_results

Traceback (most recent call last):

  File "<input>", line 1, in <module>

UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 2017: ordinal not in range(128)

>>> 

I am not sure if this is related with the format type that arrive the response from my iCAT and my interpreter don't know the format

this is utf-8 ..? 

I am not sure about of what is happen here ..

Thanks.

...

[de Torcy, Antoine]

Do you see this in your interpreter?

>>> import sys
>>> print sys.stdout.encoding
UTF-8
>>>

---

From: irod...@googlegroups.com [irod...@googlegroups.com] on behalf of Bernardo García [boti...@gmail.com]

Sent: Friday, May 08, 2015 5:09 PM
To: irod...@googlegroups.com
Subject: Re: [iROD-Chat:13754] About of PyRods and iRODS 4.0 and python-irods client

[de Torcy, Antoine]

Never mind, I was able to reproduce your error:

iRODS side:

irods@osiris ~> touch "åäö"
irods@osiris ~>
irods@osiris ~>
irods@osiris ~>
irods@osiris ~> iput åäö
irods@osiris ~>
irods@osiris ~>
irods@osiris ~>
irods@osiris ~> ils
/dev/home/rods:
  åäö
  C- /dev/home/rods/tests
irods@osiris ~>
irods@osiris ~>

Python client side:

>>> print "åäö"
åäö
>>>
>>>
>>> for obj in coll.data_objects:
...     print obj
...
Exception irods.exception.NetworkException: NetworkException('Unable to send message',) in <bound method Connection.__del__ of <irods.connection.Connection object at 0x7f6c6fe9fdd0>> ignored

Traceback (most recent call last):

  File "<stdin>", line 2, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 23-25: ordinal not in range(128)
>>>
 
I have made a new github issue to keep track of Unicode support in python-irodsclient (https://github.com/irods/python-irodsclient/issues/22)
Thanks,

Antoine

---

From: irod...@googlegroups.com [irod...@googlegroups.com] on behalf of de Torcy, Antoine [adet...@email.unc.edu]
Sent: Friday, May 08, 2015 6:24 PM
To: irod...@googlegroups.com
Subject: RE: [iROD-Chat:13754] About of PyRods and iRODS 4.0 and python-irods client

[Bernardo Augusto García Loaiza]

Antoine,

In my python interpreter I can see the UTF-8 encoding

(iRODS)➜  ~  python 

Python 2.7.6 (default, Mar 22 2014, 22:59:56) 

[GCC 4.8.2] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import sys

>>> print sys.stdout.encoding

UTF-8

>>>

But, when I use bpython or IPython (installed with pip) , is here when I get the error

(iRODS)➜  ~  bpython

bpython version 0.14.1 on top of Python 2.7.6 /home/bgarcial/.virtualenvs/iRODS/bin/python

>>> import sys

>>> print sys.stdout.encoding

Traceback (most recent call last):

  File "<input>", line 1, in <module>

AttributeError: 'FakeOutput' object has no attribute 'encoding'

>>> 

In IPython (and IPython 2) works

(iRODS)➜  ~  ipython 

WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.

Python 2.7.6 (default, Mar 22 2014, 22:59:56) 

Type "copyright", "credits" or "license" for more information.

IPython 2.4.0 -- An enhanced Interactive Python.

?         -> Introduction and overview of IPython's features.

%quickref -> Quick reference.

help      -> Python's own help system.

object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import sys

In [2]: print sys.stdout.encoding

UTF-8

In [3]: 

But, the error of encoding denoted previously appear with python, bpython and IPython

WIth place to doubts I have a problem with some of my interpreter or unless with aome of these utilities associated to it.

I will review. 

Thanks. :)

[Bernardo Augusto García Loaiza]

Did you mean ...

Is possible that there is some problem with these API in relation with Unicode support? 

[de Torcy, Antoine]

Most likely. Please check back next week for updates.

Antoine

[Bernardo Augusto García Loaiza]

You refer to python-irodsclient API.
I will have install it again the next week, just in case?

Thanks :)