persisting to HDFS

2 views
Skip to first unread message

Ahmet Uyar

unread,
Jul 29, 2020, 12:54:50 PM7/29/20
to Twister2
Hi guys,

It seems that twister2 is persisting the data to NFS even though I set the following parameter in checkpoint.yaml file

twister2.checkpointing.store: edu.iu.dsc.tws.checkpointing.stores.HDFSFileStateStore

how can I persist it to HDFS? Is there any other parameter that I need to set?

thanks,

Ahmet

Ahmet Uyar

unread,
Jul 30, 2020, 5:48:28 AM7/30/20
to Twister2
Hi Niranda and Chathura,

I was wondering whether you have tested checkpointing on hdfs before. It seems that a directory with the jobID is created on hdfs but nothing is written in it. All checkpointing files are written to nfs. 

thanks,

Ahmet

Chathura Widanage

unread,
Jul 30, 2020, 10:44:56 AM7/30/20
to Ahmet Uyar, Twister2
Hi Ahmet,

Can you add some logs at line line 59 edu.iu.dsc.tws.checkpointing.stores.HDFSFileStateStore and Line 104 edu.iu.dsc.tws.checkpointing.stores.LocalFileStateStore.

Regards,
Chathura


--
You received this message because you are subscribed to the Google Groups "Twister2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to twister2+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/twister2/CAPBRfYe%3DsQxHKNZFGRhQn8H8Xfo5ux%2ByNecXBKRmgq53NqpJzw%40mail.gmail.com.

Ahmet Uyar

unread,
Aug 3, 2020, 8:02:04 AM8/3/20
to Chathura Widanage, Twister2
Hi Chathura,

I added log lines to both put methods as you suggested.  
I ran with 4 workers only. 
Each worker reads 10M tweetID-date pairs. 
Both deleteIDs and tweetID-date pairs are persisted. 
Strangely, neither of the log messages are ever printed in logs. 

I put some other log messages in MPIWorkerStarter class to see whether my modifications are taking effect. Those log message are printed. 

I checked the nfs drive for this job, it has checkpointed the files for persisting delete-keys. 
It throws an exception before starting to checkpoint tweetID-date pairs. 

thanks,

Ahmet



auyar-membership-finding-n1n2cl9.log

Chathura Widanage

unread,
Aug 3, 2020, 9:21:48 AM8/3/20
to Ahmet Uyar, Twister2
Hi Ahmet,

Could you please compress and attach your config folder, and MembershipFinder4 class?

Regards,
Chathura

Ahmet Uyar

unread,
Aug 3, 2020, 9:29:43 AM8/3/20
to Chathura Widanage, Twister2
hi Chathura,

my conf dir is attached. 

thanks,

Ahmet
conf.tar

Chathura Widanage

unread,
Aug 3, 2020, 3:15:48 PM8/3/20
to Ahmet Uyar, Twister2
Since UCX PR is getting delayed to merge, I sent the HDFS changes in a different PR.

https://github.com/DSC-SPIDAL/twister2/pull/939

Regards,
Chathura

Reply all
Reply to author
Forward
0 new messages