migrate data from old to new irods-system

[Ger Strikwerda]

We (university of Groningen/NL) happen to have an old irods system (irods0) and a new irods system (rdms). What would be most useful automated/scripted method to
mass migrate all the user-data from the old system to the new system?

irods0: 200+ users, pnumbers-as-accounts, 70 TB not-replicated storage, irods version 4.2.6
rdms: email-address-as-accounts, 90 TB replicated storage, irods version 4.2.6

using irsync with environment bits like

{
    "irods_default_resource": "pt_top",
    "irods_host": "rdms-prod-icat.data.rug.nl",
    "irods_user_name": "irods",                                                                                                                     
    "irods_zone_name": "rug",
}

ger@irods0:~$ irsync -r /Vault5/home/p236378/ i:/rug/home/j.p.mc....@rug.nl/
remote addresses: 129.125.77.40 ERROR: rsyncDirToCollUtil: put /Vault5/home/p236378/test9_k6Jt1V.dat failed. status = -818000 status = -818000 CAT_NO_ACCESS_PERMISSION
remote addresses: 129.125.77.40 ERROR: rsyncDirToCollUtil: put /Vault5/home/p236378/test2_DjwAWe.dat failed. status = -818000 status = -818000 CAT_NO_ACCESS_PERMISSION
remote addresses: 129.125.77.40 ERROR: rsyncDirToCollUtil: put /Vault5/home/p236378/test8_Oftfax.dat failed. status = -818000 status = -818000 CAT_NO_ACCESS_PERMISSION
remote addresses: 129.125.77.40 ERROR: rsyncUtil: rsync error for /rug/home/j.p.mc....@rug.nl status = -818000 CAT_NO_ACCESS_PERMISSION
Level 0: no permission to update collection '/rug/home/j.p.mc....@rug.nl'

cleary doesn't fly.

How (as an admin) can i mass-migrate irods data from a 'old' system to a 'new' system? Any of you guys out there with migration experience?

Thanks in advance,

Ger Strikwerda
Chef Special/Team HPC
University of Groningen

[Tony Edgin]

Does the new iRODS share an ICAT database with the old one, i.e., is there a separate DBMS hosting the ICAT? If so, you can federate the two iRODS zones and then use irsync.  If not, use irepl followed by itrim instead.  Use irepl to replicate the data onto the new resource, then use itrim to remove the data from the old resource.

Cheers,

Tony

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/b296f639-f6f9-40b4-932c-b86f974def23%40googlegroups.com.

[Ger Strikwerda]

Hi Tony (and others),

The 2 iRODS systems do not share the same ICAT, so 2 different zones and it's not so easy to copy data from the old to new. Federate the 2 zones and do a irsync is probably the way to go. But we have come up with this Plan A:

- create migrate-user on irods0

- make a migrate group on irods0 with read rights on all data

- mount irods0 as webdav-fuse-mount (expose all irods0 data via webdav)

- do a irsync -r /mnt/irods0/* i:rug

(with i being the new environment)

Let's see if this is possible.. 

vr.gr. Ger Strikwerda/University Groningen

On Wednesday, September 4, 2019 at 5:53:50 PM UTC+2, Tony Edgin wrote:

Does the new iRODS share an ICAT database with the old one, i.e., is there a separate DBMS hosting the ICAT? If so, you can federate the two iRODS zones and then use irsync.  If not, use irepl followed by itrim instead.  Use irepl to replicate the data onto the new resource, then use itrim to remove the data from the old resource.

Cheers,

Tony

On Wed, Sep 4, 2019 at 5:33 AM Ger Strikwerda <g.j.c....@rug.nl> wrote:

We (university of Groningen/NL) happen to have an old irods system (irods0) and a new irods system (rdms). What would be most useful automated/scripted method to
mass migrate all the user-data from the old system to the new system?

irods0: 200+ users, pnumbers-as-accounts, 70 TB not-replicated storage, irods version 4.2.6
rdms: email-address-as-accounts, 90 TB replicated storage, irods version 4.2.6

using irsync with environment bits like

{
    "irods_default_resource": "pt_top",
    "irods_host": "rdms-prod-icat.data.rug.nl",
    "irods_user_name": "irods",                                                                                                                     
    "irods_zone_name": "rug",
}

ger@irods0:~$ irsync -r /Vault5/home/p236378/ i:/rug/home/j.p.mc.farland@rug.nl/

remote addresses: 129.125.77.40 ERROR: rsyncDirToCollUtil: put /Vault5/home/p236378/test9_k6Jt1V.dat failed. status = -818000 status = -818000 CAT_NO_ACCESS_PERMISSION
remote addresses: 129.125.77.40 ERROR: rsyncDirToCollUtil: put /Vault5/home/p236378/test2_DjwAWe.dat failed. status = -818000 status = -818000 CAT_NO_ACCESS_PERMISSION
remote addresses: 129.125.77.40 ERROR: rsyncDirToCollUtil: put /Vault5/home/p236378/test8_Oftfax.dat failed. status = -818000 status = -818000 CAT_NO_ACCESS_PERMISSION

remote addresses: 129.125.77.40 ERROR: rsyncUtil: rsync error for /rug/home/j.p.m...@rug.nl status = -818000 CAT_NO_ACCESS_PERMISSION
Level 0: no permission to update collection '/rug/home/j.p.m...@rug.nl'

cleary doesn't fly.

How (as an admin) can i mass-migrate irods data from a 'old' system to a 'new' system? Any of you guys out there with migration experience?

Thanks in advance,

Ger Strikwerda
Chef Special/Team HPC
University of Groningen

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.

To unsubscribe from this group and stop receiving emails from it, send an email to irod...@googlegroups.com.