msiCollRsync - optional inputParameter

[mustafa dikmen]

Hello,

I have some questions regarding with the msiCollRsync:

msi link at https://docs.irods.org/4.2.8/doxygen/reDataObjOpr_8cpp.html#ab45c69071a587e63fbb9e107a656b907

example at https://github.com/irods/irods_rule_engine_plugin_python/blob/master/python_rules/rulemsiCollRsync.r

- As you see at the links, the target resource and the synchronization mode parameters of this msi are optional. So I don't have to specify my resource, right? And for my use case I don't want to specify. For this aim how can I use it? I couldn't manage it. Once I try with an empty string I am receiving "SYS_INVALID_INPUT_PARAM". I think either the documentations are misleading or I misunderstood. If the documentations imply that you can specify/use one of the resource, meaning multiple-choose, then I think that is misleading. But I assume optional is optional. Meaning you are not forced to use, it is not mandatory. For example, I don't have specified -R with iCommand "irsync -r foo1 i:foo2".

- After I put the resource argument like in the example, it does work. However it behaves slightly different than iCommand and writes errors below onto the logs. Why does this happen, should I care?

Apr 30 11:43:16 pid:2861263 remote addresses: 10.0.2.100, ::1 ERROR: [-]        /irods_git_repo/server/api/src/rsDataObjChksum.cpp:95:int rsDataObjChksum(rsComm_t *, dataObjInp_t *, char **) :  status [CAT_NO_ROWS_FOUND]  errno [] -- message [failed in irods::resolve_resource_hierarchy for [/tempZone/home/rods/sub2/test1.txt]]

        [-]     /irods_git_repo/server/core/src/irods_resource_redirect.cpp:610:irods::error irods::resolve_resource_hierarchy(const std::string &, rsComm_t *, dataObjInp_t *, std::string &, dataObjInfo_t **) :  status [CAT_NO_ROWS_FOUND]  errno [] -- message [resolve_resource_hierarchy :: failed in file_object_factory]

                [-]     /irods_git_repo/server/core/src/irods_file_object.cpp:334:irods::error irods::file_object_factory(rsComm_t *, dataObjInp_t *, irods::file_object_ptr, dataObjInfo_t **) :  status [CAT_NO_ROWS_FOUND]  errno [] -- message [failed in call to getDataObjInfoIncSpecColl for [/tempZone/home/rods/sub2/test1.txt] CAT_NO_ROWS_FOUND ]

- Maybe it has been discussed before. I just wonder wouldn't it be nice to have a metadata copy option inside this msi? 

Thank you.

Mustafa

[Alan King]

Hi Mustafa,

Sorry for the late reply!

I believe resolve_resource_hierarchy requires at least the presence of DEF_RESC_NAME_KW, which specifies the default resource of the user. The iCommands tend to provide this keyword in the inputs as part of detecting the client environment.

So in this case, I would say that the documentation is misleading or this could be considered a bug in the server. In theory the server should be able to detect its own default resource (server_config.json has this configuration: https://docs.irods.org/4.2.8/system_overview/configuration/).

As for the error message in the log, it's not clear to me what happened. Could you please share the rule code which triggered it and any other related messages from the log? I'm assuming you confirmed that the data object /tempZone/home/rods/sub2/test1.txt exists? Sorry for potentially dumb questions :)

Finally, could you please provide more details about your request for a metadata copy microservice?

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/cf8891d5-aebc-4429-b293-66a92b697214n%40googlegroups.com.

--

Alan King

Software Developer | iRODS Consortium

[mustafa dikmen]

Hi Alan,

Thanks for replying.

So basically what I understand is that the iComomand "irsync -r i:foo1 i:foo2" and "msiCollRsync" msi should behave same in this sense.

One of the old microservice manual has this remark, "note that optional parameters take the value 'null' when not being used". But I cannot see the same remark in the "doxygen". There could be then some issue after the version upgrade? I can open an issue for this if you see as bug. 

Note: I tested by providing with "null". It didn't work. Even for the native rule language I got "NO_MICROSERVICE_FOUND_ERR". 

With regards to the error message in the log;

I have two collections, sub1 and sub2. I wanted to test few things like what I have done with "irsync". Assume "test1.txt" exists in "sub1" and doesn't exist in "sub2", then if I execute "irsync -r i:sub1 i:sub2" I don't have any issue in the logs and I can see the "test1.txt" in "sub2", meaning copied to "sub2". If I do the same with the following test rule I am getting the error I shared earlier (nothing more in the log). But it also copies test1.txt in spite of errors in the log. So no, the data object "test1.txt" didn't exist in "/tempZone/home/rods/sub2/" if that was the the question. However, if I am not mixing things, like how iCommand does this msi first will look at the source and compares the dataobjs there with target-source. If that is not in the target, it will write/copy it. Then later it will check the size/checksum to update the files in the target based on the source.

def main(rule_args, callback, rei):

    src_coll = global_vars['*srcColl'][1:-1]

    dest_coll = global_vars['*destColl'][1:-1]

    resource = global_vars['*Resource'][1:-1]

    callback.msiCollRsync(src_coll, dest_coll, resource, 'IRODS_TO_IRODS', 0)

    callback.writeLine('stdout', 'Synchronized collection ' + src_coll + ' with collection ' + dest_coll)

INPUT *srcColl="/tempZone/home/rods/sub1", *destColl="/tempZone/home/rods/sub2", *Resource="demoResc"

OUTPUT ruleExecOut

As for the metadata copy microservice:

When there is a need to copy the content (dataObjs together with their attached metadata) of "collection1" to "collection2", we are using our own function that works with "msiString2KeyValPair", "msiAssociateKeyValuePairsToObj" and "Query" object.

callback.msiDataObjCopy(collection1/file_name, collection2/file_name, "forceFlag=", 0)

copyMetadataToDataObj(callback, collection1, file_name, collection2/file_name)

Instead of doing like this, I would prefer to use either a single msi or want it part of the one another msi as an option. But that is okay for now.

One extra question;

In older manuals, I see "msiIsColl" but not exists anymore. Currently I am using the python subprocess module to get the content of a collection. I guess there isnt any counterpart of "msiIsColl" for 4.2.8. Is this correct?

Regards

Mustafa