workers blocking when reading hdfs files

[Ahmet Uyar]

Hi guys,

I am running 240 twister2 workers, each of them is supposed to read a separate hdfs file with 1000 tweetIDs in them. Usually they all read the files without any issues. But, sometimes they just block and wait indefinitely.  

I am attaching jstack outputs from two such workers. 

thanks,

Ahmet

[Ahmet Uyar]

Hi guys,

I think this issue is not related to hdfs file reading. I think it is related to some communication issue. 

The Main worker thread is waiting on the method: mpi.Request.testStatus(Native Method).

It also happens after all workers have finished reading the input files for deleteIDs. 

Ahmet

[Ahmet Uyar]

Hi guys,

This issue also happens when partitioning tweedID-date pairs. 

Attached jstack output is from such a run. 

The job blocks when partitioning tweetID-date pairs. 

Ahmet

[Gregor von Laszewski]

Last year we talked about educational material for Twister2 that is not just a Manual.
The gaol was to have some material that would introduce the students to twister2 and have
practical assignments such as

a) installation
b) hello word
c) examples on to do xyz

Now that a lot of changes have been made is such material available?

Originally this was supposed to be developed in markdown so it could be integrated into bookmanager
And our lecture notes, but I do not think that was done. I think instead the manual pages have been developed.

Is this right?

Let me know if such educational material is or is not available.

Thanks

Gregor

[Supun Kamburugamuve]

The current document is in 

https://twister2.org/

There are few guides on compiling twister2, in mac and Linux.

https://twister2.org/docs/compiling/compile_overview

https://twister2.org/docs/compiling/linux

https://twister2.org/docs/compiling/macos

There is a tutorial on getting started. 

https://twister2.org/docs/introduction

The website is generated from markdown files found in the source. 

https://github.com/DSC-SPIDAL/twister2/tree/master/docs/docs

It uses docusaurus to generate the website. The website is hosted in Github.

https://docusaurus.io/

Let us know, what you think. We can create/rearrange material as needed.

Best,

Supun..

--
You received this message because you are subscribed to the Google Groups "Twister2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to twister2+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/twister2/F45B10D2-534C-49F1-B7DF-E9BE32C83107%40gmail.com.

--

Supun Kamburugamuve, PhD

Digital Science Center, Indiana University

Member, Apache Software Foundation; http://www.apache.org
E-mail: supun@apache.org;  Mobile: +1 812 219 2563