jobID in hdfs path

[Ahmet Uyar]

Hi Chathura,

The persisted data is saved in hdfs at the given directory in config files. However, we should add jobID to this directory I think. Otherwise, consecutive or concurrent runs may interfere. 

Currently, persisted data is saved in the following directory at hdfs: 

/twister2/persistent/data/__kgather4_0

/twister2/persistent/data/__kgather4_1

/twister2/persistent/data/__kgather4_2

/twister2/persistent/data/__kgather4_3

Ahmet

[Ahmet Uyar]

Hi Chathura,

CheckpointManager seems to save the checkpoints at hdfs at: 

/twister2/persistent/twister2-checkpoints/<jobID>/

Maybe the tset data can also be persisted/saved at the same location under the data directory. Just a suggestion. 

I also checked nfs persistent directory whether there is anything in there. There is logs directory and that is empty. Previously, it was saving the logs there as I remember. Not sure, whether logs should also go to hdfs. 

thanks,

Ahmet

[Chathura Widanage]

Hi Ahmet,

I am looking into your previous issue. Will get back to you soon.

Regards,

Chathura

On Tue, Aug 4, 2020 at 12:07 PM Ahmet Uyar <ahme...@gmail.com> wrote:

Hi Chathura,

CheckpointManager seems to save the checkpoints at hdfs at: 

/twister2/persistent/twister2-checkpoints/<jobID>/

Maybe the tset data can also be persisted/saved at the same location under the data directory. Just a suggestion. 

I also checked nfs persistent directory whether there is anything in there. There is logs directory and that is empty. Previously, it was saving the logs there as I remember. Not sure, whether logs should also go to hdfs. 

thanks,

Ahmet

On Tue, Aug 4, 2020 at 6:51 PM Ahmet Uyar <ahme...@gmail.com> wrote:

Hi Chathura,

The persisted data is saved in hdfs at the given directory in config files. However, we should add jobID to this directory I think. Otherwise, consecutive or concurrent runs may interfere. 

Currently, persisted data is saved in the following directory at hdfs: 

/twister2/persistent/data/__kgather4_0

/twister2/persistent/data/__kgather4_1

/twister2/persistent/data/__kgather4_2

/twister2/persistent/data/__kgather4_3

Ahmet

--
You received this message because you are subscribed to the Google Groups "Twister2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to twister2+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/twister2/CAPBRfYcA9hU6o36D%3D%2B3xeq1beP28nnz53Q5SJRvAzecuNyv1NQ%40mail.gmail.com.

[Chathura Widanage]

@Niranda Perera  Would it break anything if we change referencePrefix variable to include the job ID?

Regards,

Chathura

[Niranda Perera]

No it would not. It's okay to use any string there as long as it's deterministic and the same in the sink and source both (for disk persistent sink/ source) 

To view this discussion on the web visit https://groups.google.com/d/msgid/twister2/CAK8mM2HBmcKvXfiWfD2oD2JqpnHeD8Y%3Dy_RrRBbAkBvW9Jv2-g%40mail.gmail.com.