Storage tiering not replicating changed files downstream?

[Jan de Graaf]

Hi,

I'm experiencing the issue that the storage tiering plugin will replicate new files to the downstream storage tiers. But then when we change a file on the top tier the lower tier will not receive the changed file.

The only way to force this is by manually replicate with the irepl command.

For example by executing the following commands: 

irepl -r -P -B -S RES_IMG_Store_01  -R RES_IMG_Store_02 /nkiImaging/home/BioImaging/

irepl -r -P -B -S RES_IMG_Store_02  -R RES_IMG_Store_03 /nkiImaging/home/BioImaging/

So to keep the tiers in sync I now run these commands from time to time.

Is this intended/by design? Or is something not working correctly in the tiering plugin?

Best,

Jan de Graaf

[Alan King]

Hi Jan,

On initial reading, this sounds like things are working as expected.

Replicas only tier out from lower to higher tiers. So if a replica is modified in a higher tier, its access_time AVU will be updated, and it will tier out to an even higher tier once the recurring storage tiering rule finds it to be in violation of the configured tiering policy (for age-based policies). Replicas can move from higher tiers to lower tiers via restaging, but restaging is supposed to only occur when data is explicitly being read by a client (via open or get).

To get your desired results, you would need to update the replicas in the low(est) tier and then the data would tier out to the higher tiers, thereby updating the other replicas.

We are of course willing to hear arguments for the contrary. :)

Alan

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/2d3922a6-43c9-48c1-a6f0-f09cb06c0b1dn%40googlegroups.com.

--

Alan King

Senior Software Developer | iRODS Consortium

[Jan de Graaf]

Hi Alan,

I think we should align the higher and lower tiers definition. ;-)

This is our resource setup

RES_IMG_Store_01:passthru
└── RES_IMG_Store_PT01:passthru
    └── IMG_Store_01:unixfilesystem (Tier 0)
RES_IMG_Store_02:passthru
└── RES_IMG_Store_PT02:passthru 
    └── IMG_Store_02:unixfilesystem (Tier 1)
RES_IMG_Store_03:passthru
└── IMG_Store_PT03:passthru
    └── IMG_Store_03:unixfilesystem (Tier 2)
RES_IMG_Store_04:passthru
└── IMG_Store_PT04:passthru
    └── IMG_Store_04:unixfilesystem (Tier 3)

RES_IMG_Store_05:passthru
└── IMG_Store_PT05:passthru
    └── IMG_Store_05:unixfilesystem

So by your definition data should go from 0 down to 3 and restage from 3 up to 0? Correct? This is also the behavior we see in general. New files a stored on Tier 0 and then as expected go down to 1,2,3. When we trim tiers 0,1,2 and retrieve data, the restaging of file go into effect. So data is restaged to Tier 0. No problem so far.

But when we change a, existing file on Tier 0 the other tiers (1,2,3)  will not recieve the changed files.

With an ILS -L we see that replica's are out of sync.

For example:

 ils -L test.txt
  rods              0 RES_IMG_Store_01;RES_IMG_Store_PT01;IMG_Store_01           18 2024-05-14.14:37 & test.txt
        generic    /irods/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt
  rods              1 RES_IMG_Store_02;RES_IMG_Store_PT02;IMG_Store_02            4 2024-05-14.12:27 X test.txt
        generic    /data/irods/data/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt
  rods              2 RES_IMG_Store_03;IMG_Store_PT03;IMG_Store_03            4 2024-05-14.12:27 X test.txt
        generic    /mtn_NFS_IMG_Cold/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt

But if i don't issue a manual replication then the tiers stay out of sync.

For example:  

irepl -V -P -S RES_IMG_Store_01 -R RES_IMG_Store_02 /nkiImaging/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt

irepl -V -P -S RES_IMG_Store_02 -R RES_IMG_Store_03 /nkiImaging/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt

then the files are back in sync again.

ils -L test.txt
  rods              0 RES_IMG_Store_01;RES_IMG_Store_PT01;IMG_Store_01           18 2024-05-14.14:37 & test.txt
        generic    /irods/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt
  rods              1 RES_IMG_Store_02;RES_IMG_Store_PT02;IMG_Store_02           18 2024-05-14.14:45 & test.txt
        generic    /data/irods/data/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt
  rods              2 RES_IMG_Store_03;IMG_Store_PT03;IMG_Store_03           18 2024-05-14.14:45 & test.txt
        generic    /mtn_NFS_IMG_Cold/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt

If I read you response what i now do manual the storage tiering plugin should be doing for me? Based on the access time AVU?

Now the files are modifies through our ingestion plugin and we still have issues with that but working around that issue by manually adding the accestime avu through the post events.

    @staticmethod
    def post_data_obj_create(hdlr_mod, logger, session, meta, **options):
        epoch_time = str(int(time.time()))
        obj.metadata.add("irods::access_time", epoch_time, '')

    @staticmethod
    def post_data_obj_modify(hdlr_mod, logger, session, meta, **options):
        epoch_time = str(int(time.time()))
        obj.metadata.set("irods::access_time", epoch_time, '')

Op dinsdag 14 mei 2024 om 16:26:09 UTC+2 schreef alankin...@gmail.com:

[Alan King]

Ah, sorry for the confusion. Yes, you have correctly described how I'm thinking of the tiers.

Now that things have been cleared up, I think what you are seeing is a bug, yes. If the data in tier 0 is modified, the access_time should be updated (please confirm) and the time-based violating query should be enforced.

Are you modifying the data using the ingest tool? Is that via a PUT_SYNC, PUT_APPEND, or another operation? Have you tried other ways of modifying the data to see if that works (e.g. iput, istream --append)?

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/9a9464b1-b351-40b6-808c-05442a470a56n%40googlegroups.com.

[Jan de Graaf]

We use the PUT_SYC operation for the ingestion of data.

AVU before update:

imeta ls -d test.txt
AVUs defined for dataObj /nkiImaging/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt:
attribute: irods::access_time
value: 1715697953
units:

sync status before update:

ils -L test.txt
  rods              0 RES_IMG_Store_01;RES_IMG_Store_PT01;IMG_Store_01           18 2024-05-14.14:37 & test.txt
        generic    /irods/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt
  rods              1 RES_IMG_Store_02;RES_IMG_Store_PT02;IMG_Store_02           18 2024-05-14.14:45 & test.txt
        generic    /data/irods/data/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt
  rods              2 RES_IMG_Store_03;IMG_Store_PT03;IMG_Store_03           18 2024-05-14.14:45 & test.txt
        generic    /mtn_NFS_IMG_Cold/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt

Update of file done via auto ingest (PUT_SYC) 

(modification of AVU done via workaround in eventhandler)

    @staticmethod
    def post_data_obj_modify(hdlr_mod, logger, session, meta, **options):
        epoch_time = str(int(time.time()))
        obj.metadata.set("irods::access_time", epoch_time, '')

AVU after update:

imeta ls -d test.txt
AVUs defined for dataObj /nkiImaging/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt:
attribute: irods::access_time
value: 1715758634 
units:

sync after update:

 ils -L test.txt
  rods              0 RES_IMG_Store_01;RES_IMG_Store_PT01;IMG_Store_01           25 2024-05-15.07:37 & test.txt
        generic    /irods/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt
  rods              1 RES_IMG_Store_02;RES_IMG_Store_PT02;IMG_Store_02           18 2024-05-14.14:45 X test.txt
        generic    /data/irods/data/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt
  rods              2 RES_IMG_Store_03;IMG_Store_PT03;IMG_Store_03           18 2024-05-14.14:45 X test.txt
        generic    /mtn_NFS_IMG_Cold/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt

But replication will not happen.

Then manual replication:

 irepl -V -P -S RES_IMG_Store_01 -R RES_IMG_Store_02 /nkiImaging/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt

0/1 -  0.00% of files done   0.000/0.000 MB -  0.00% of file sizes done
Processing test.txt - 0.000 MB   2024-05-15.09:48:42
   test.txt                        0.000 MB | 2.363 sec | 0 thr |  0.000 MB/s

 irepl -V -P -S RES_IMG_Store_02 -R RES_IMG_Store_03 /nkiImaging/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt

0/1 -  0.00% of files done   0.000/0.000 MB -  0.00% of file sizes done
Processing test.txt - 0.000 MB   2024-05-15.09:48:45
   test.txt                        0.000 MB | 2.499 sec | 0 thr |  0.000 MB/s

Then the sync is ok again:

 ils -L test.txt
  rods              0 RES_IMG_Store_01;RES_IMG_Store_PT01;IMG_Store_01           25 2024-05-15.07:37 & test.txt
        generic    /irods/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt
  rods              1 RES_IMG_Store_02;RES_IMG_Store_PT02;IMG_Store_02           25 2024-05-15.07:48 & test.txt
        generic    /data/irods/data/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.txt
  rods              2 RES_IMG_Store_03;IMG_Store_PT03;IMG_Store_03           25 2024-05-15.07:48 & test.txt
        generic    /mtn_NFS_IMG_Cold/home/BioImaging/2024/Backup_CSLM/BIF/jg_bif/2024-05-14/test.tx

Op dinsdag 14 mei 2024 om 17:45:09 UTC+2 schreef alankin...@gmail.com:

[Alan King]

Okay, thanks for the information. I'm guessing that you are using the post-create method in your event handler because of the issue you ran into here, wherein access_time AVUs are not being annotated/updated via ingest: https://groups.google.com/g/irod-chat/c/4jGipTo0 Is that right?

Now, onto this new issue. I was able to reproduce this behavior, and my current thinking is that it has nothing to do with ingest. I tried updating the replica in tier 0 via istream write and the result was the same. So from now on in this discussion, let's take ingest off the table to keep things simpler.

This feels related to but not exactly the same as this issue: https://github.com/irods/irods_capability_storage_tiering/issues/235

I've opened a new issue to investigate this: https://github.com/irods/irods_capability_storage_tiering/issues/272

I was able to get the tier out to happen only if I trimmed the other two replicas. This is probably a clue as to how to fix this, but I think we also need to think about what this situation means.

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/3579bade-2921-44f7-b983-c6b39783b85bn%40googlegroups.com.