How to get a value in irods_types.char_array

[mustafa dikmen]

Hello,

Before and after atomic metadata operations are applied on an object, I would like to get the path of that object. For this aim, I am trying to catch the value of rule_args[2].buf in pep_api_atomic_apply_metadata_operations_pre/post. To succeed this, I have:

def pep_api_atomic_apply_metadata_operations_post(rule_args, callback, rei):

    var_map = session_vars.get_map(rei)
    user = var_map["client_user"]["user_name"]
    obj = str(rule_args[2].buf)
    obj_dict = json.loads(obj)
    #collName = obj_dict["entity_name"]
    #refColl = str(collName).split(os.sep)[:5]
    #refColl = "/".join(refColl)

However this line obj_dict = json.loads(obj) throws the following on the client side:

Traceback (most recent call last):
  File "test.py", line 12, in <module>
    collection.metadata.apply_atomic_operations(AVUOperation(operation='add',avu=iRODSMeta('user.git.hooks.project_name', 'numa')))
  File "/usr/lib/python2.7/site-packages/irods/meta.py", line 103, in apply_atomic_operations
    self._manager.apply_atomic_operations(self._model_cls, self._path, *avu_ops)
  File "/usr/lib/python2.7/site-packages/irods/manager/metadata_manager.py", line 157, in apply_atomic_operations
    self._call_atomic_metadata_api(request)
  File "/usr/lib/python2.7/site-packages/irods/manager/metadata_manager.py", line 164, in _call_atomic_metadata_api
    response = conn.recv()
  File "/usr/lib/python2.7/site-packages/irods/connection.py", line 121, in recv
    raise get_exception_by_code(msg.int_info, err_msg)
irods.exception.UnknowniRODSError: None

I dont se anything in the server log.

The type of rule_args[2].buf is <class 'irods_types.char_array'>.

And in my example it is:

 {"operations": [{"attribute": "user.git.hooks.project_name", "operation": "add", "value": "numa"}], "entity_name": "/u0137480/home/set_pilot003/repositories/test", "entity_type": "collection"}

I would like to get the value of "entity_name". Do you know how I can catch this?

I hope I am not missing something here.

Thank you.

Mustafa Dikmen

Regards.

[mustafa dikmen]

I have solved it by using string manipulation. Thanks.

[mustafa dikmen]

Hello,

Could someone explain me how I can get/catch correctly the path (value of "entity_name":) after I trigger pep_api_atomic_apply_metadata_operations_post(rule_args, callback, rei)?

A part of my use case is; I will need to apply atomic metadata on, for instance, collection_ABC by a prc script and finally I need to get the path of the collection_ABC in the rule engine/server.

As mentioned in the first post, I cannot succeed to get the path of an object on which metadata attached by atomic operations as key/val.

The string manipulation fails if the client environment is python3. As given in the example below, the coll variable doesnt represents the collection path once this pep is triggered by the second script. The order of characters in str(rule_args[2].buf) comes differently depending on the client python version (prc is the same version 1.1.1). Is there a healthy way to catch the path of the object that is added metadata by atomic operations via prc?

script 1 (executed in my development environment that is Python 2.7.5):

collection = session.collections.get(' /u0137480/home/set_pilot003/repositories/test  ')

collection.metadata.apply_atomic_operations(AVUOperation(operation='add',avu=iRODSMeta('atr', 'val')))

script 2 (executed in my development environment that is Python 3.9.12):

collection = session.collections.get(' /icts_icts/home/set_pilot003/repositories/test  ')

collection.metadata.apply_atomic_operations(AVUOperation(operation='add',avu=iRODSMeta('atr', 'val')))

This solution 

def pep_api_atomic_apply_metadata_operations_post(rule_args, callback, rei):

    obj = str(rule_args[2].buf)

    obj = [item for item in obj.split(',')]

    coll = [item.strip('"') for item in obj[3].split()]

    print coll[1]

    refColl = coll[1].split(os.sep)[:5]

    refColl = "/".join(refColl)

    if refColl.endswith("something/to/ref"):

    ...

['{"operations": [{"attribute": "atribute"', ' "operation": "add"', ' "value": "value"}]', ' "entity_name": "/u0137480/home/set_pilot003/repositories/test"', ' "entity_type": "collection"}']

/u0137480/home/set_pilot003/repositories/test

['{"entity_name": "/icts_icts/home/set_pilot003/repositories/test"', ' "entity_type": "collection"', ' "operations": [{"operation": "add"', ' "attribute": "attribute"', ' "value": "value"}]}']
attribute

Thank you.

[dmoore.renci]

hi Mustafa,

I will say the PEP rule code looks fine .

I've tried your scenario by putting that exact /etc/irods/core.py , and I do not see the error on the client end.

In your instance though, the signs are that  indeed the rule is failing for some reason.

Check the following:

  * In  your do you have an import json statement? ( Admittedly, if you didn't, you should have seen  NameError  global name 'json' is not defined thrown in the server logs. )

If that isn't the problem, let's maybe try changing the line

obj_dict = json.loads(obj)

to:

print ( "DEBUG: obj type = {}, value = {!r}".format(type(obj),obj))

and then : tail -0f  on the server log to see what prints out.

obj's type should be str and the value should contain a valid JSON string.  If that is true, there's no reason json.loads( ) should fail.

-Daniel Moore - Applications Engineer - iRODS Consortium

[dmoore.renci]

Sorry! : ) should have said I placed your exact PEP rule code into my  /etc/irods/core.py , preceded by the imports:

import session_vars
import json
import os

[mustafa dikmen]

Hi Daniel,

Thanks for your reply. Yes, the json was imported - used the import json statement at the beginning of the rule file and all rules were loaded to /etc/irods/core.py. The obj type is indeed str.

I am not sure what is happening but I am getting the same error (in my first mail) on the client end as I add some extra lines (codes) inside the pep. Moreover, with the same code it sometimes throws the mentioned error, some time not.

I will look into it a bit more. If I cannot find a reason for this, can I maybe share with you the rules?

Thanks. Regards

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to a topic in the Google Groups "iRODS-Chat" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/irod-chat/FpYwVH2JVo8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/2f423e48-6178-49ee-9bcb-784aded068e7n%40googlegroups.com.

[dmoore.renci]

Mustafa,

yes, if nothing else works, share what you can including all rules called and the client that uses the atomic_metadata call, and how you are invoking the client.

thanks! - Daniel

[mustafa dikmen]

Hello,

I am having trouble with the same issue. It seems to me that pep_api_atomic_apply_metadata_operations_post is not consistent in terms of serialized objects (parameters - referring the earlier discussion as well).

Pep loaded into core.py

def pep_api_atomic_apply_metadata_operations_post(rule_args, callback, rei):

    var_map = session_vars.get_map(rei)

    username = var_map["client_user"]["user_name"]

    obj = str(rule_args[2].buf)

    print obj

Logs:

{"operations": [{"attribute": "atr1", "operation": "add", "value": "val1"}, {"attribute": "atr2", "operation": "add", "value": "val2"}, {"attribute": "atr3", "operation": "add", "value": "val3"}], "entity_name": "/u0137480_instance1/home/redcap/destination/records_20220906_1038.csv", "entity_type": "data_object"} 

I would like to somehow catch the object path which is the value of "entity_name". But I couldnt succeed this in a reliable way. 

Client script:

import os, os.path
from irods.session import iRODSSession
from irods.meta import iRODSMeta, AVUOperation

avus = [('atr1', 'val1'), ('atr2', 'val2'), ('atr3', 'val3')]
env_file = os.getenv('IRODS_ENVIRONMENT_FILE', os.path.expanduser('~/.irods/irods_environment.json'))
with iRODSSession(irods_env_file=env_file) as session:

    obj = session.data_objects.get("/u0137480_instance1/home/redcap/destination/records_20220906_1038.csv")
    obj.metadata.apply_atomic_operations(*[AVUOperation(operation='add', \
                                  avu=iRODSMeta("{}".format(str(meta[0])), "{}".format(str(meta[1])))) for meta in avus])

The type of rule_args[2].buf is <class 'irods_types.char_array'>. And if I want to convert it to json object to be able to catch the path easily, then I am having the errors below:

Pep loaded into core.py

import json

def pep_api_atomic_apply_metadata_operations_post(rule_args, callback, rei):

    var_map = session_vars.get_map(rei)

    username = var_map["client_user"]["user_name"]

    obj = str(rule_args[2].buf)

    obj_dict = json.loads(obj)

    print obj_dict

Errors in the client side:

Traceback (most recent call last):

  File "test.py", line 11, in <module>
    avu=iRODSMeta("{}".format(str(meta[0])), "{}".format(str(meta[1])))) for meta in avus])
  File "/usr/lib/python2.7/site-packages/irods/meta.py", line 118, in apply_atomic_operations
    self._manager.apply_atomic_operations(self._model_cls, self._path, *avu_ops)
  File "/usr/lib/python2.7/site-packages/irods/manager/metadata_manager.py", line 176, in apply_atomic_operations
    self._call_atomic_metadata_api(request)
  File "/usr/lib/python2.7/site-packages/irods/manager/metadata_manager.py", line 183, in _call_atomic_metadata_api
    response = conn.recv()
  File "/usr/lib/python2.7/site-packages/irods/connection.py", line 132, in recv

    raise get_exception_by_code(msg.int_info, err_msg)
irods.exception.UnknowniRODSError: None

Errors in the server:

rsyslogd stdout | 2022-09-07T15:52:58.596142+02:00 dev-u0137480-instance1 rodsServer[715476]: u0137480_instance1 - remote addresses: 10.0.2.100, 2a02:2c40:0:51:177:0:9:2ca7 ERROR: caught python exception: Traceback (most recent call last):#012  File "/etc/irods/rules_numa.py", line 64, in pep_api_atomic_apply_metadata_operations_post#012    obj_dict = json.loads(obj)#012  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads#012    return _default_decoder.decode(s)#012  File "/usr/lib64/python2.7/json/decoder.py", line 369, in decode#012    raise ValueError(errmsg("Extra data", s, end, len(s)))#012ValueError: Extra data: line 1 column 315 - line 1 column 330 (char 314 - 329)

Could you try this? And please let me know if you think I am missing something.

Thanks.

Regards,

Mustafa

[Terrell Russell]

with the following core.py...

import json
import session_vars

def pep_api_atomic_apply_metadata_operations_post(rule_args, callback, rei):
    var_map = session_vars.get_map(rei)
    username = var_map["client_user"]["user_name"]
    obj = str(rule_args[2].buf)

    print('obj', obj)
    obj_dict = json.loads(obj)
    print('dict', obj_dict)

I see the error... and some extra data... notice the 'ypea' garbage at the end of the printout of 'obj'...

('obj', '{"entity_name": "/tempZone/home/alice/mustafa.txt", "operations": [{"attribute": "atr1", "operation": "add", "value": "val1"}, {"attribute": "atr2", "operation": "add", "value": "val2"}, {"attribute": "atr3", "operation": "add", "value": "val3"}], "entity_type": "data_object"}ypea')

seems we have a real bug...

Terrell

You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/1061f7ed-8ac3-4e55-aca9-3c263cb2844bn%40googlegroups.com.

[Terrell Russell]

I've created an issue for this here:

   https://github.com/irods/irods/issues/6892

Terrell