Twister2 - Google Groups

Groups

Contact owners and managers

1–30 of 648

Welcome to the Twister2 mailing list. Please use this email list to discuss and share information regarding the project. The project source code is listed at [1]

[1] https://github.com/DSC-SPIDAL/twister2

0 selected

Supun Kamburugamuve

12/1/20

Meeting tomorrow

Hi all, We are canceling the Twister2 meeting tomorrow. Let's meet next week. Best, -- Supun

unread,

Meeting tomorrow

Hi all, We are canceling the Twister2 meeting tomorrow. Let's meet next week. Best, -- Supun

12/1/20

Supun Kamburugamuve

10/14/20

Cylon presentation

Hi Everyone, Here is the Cylon presentation for ApacheCon. Best, Supun.. -- Supun Kamburugamuve, PhD

unread,

Cylon presentation

Hi Everyone, Here is the Cylon presentation for ApacheCon. Best, Supun.. -- Supun Kamburugamuve, PhD

10/14/20

Supun Kamburugamuve

10/14/20

Article about DL

Hi Everyone, Here is the DL article I talked about. https://techxplore.com/news/2020-10-deep-neurons-

unread,

Article about DL

Hi Everyone, Here is the DL article I talked about. https://techxplore.com/news/2020-10-deep-neurons-

10/14/20

Chathura Widanage, … Ahmet Uyar5

10/4/20

Twister2 v0.8.0 Released!

Thanks, Ahmet. Even the maven artefacts for 0.8.0 are now available in public repositories. https://

unread,

Twister2 v0.8.0 Released!

Thanks, Ahmet. Even the maven artefacts for 0.8.0 are now available in public repositories. https://

10/4/20

Pulasthi Supun Wickramasinghe, … Geoffrey Fox6

9/29/20

Twister2 listed on Apache Beam Website

I think they have not completed all the documentation work, the quickstarts, etc, I added all the

unread,

Twister2 listed on Apache Beam Website

I think they have not completed all the documentation work, the quickstarts, etc, I added all the

9/29/20

Ahmet Uyar, Chathura Widanage14

9/23/20

Fwd: Twister2 debug session

Hi guys, I tested without using JobMaster. I thought maybe somehow it is related to running the job

unread,

Fwd: Twister2 debug session

Hi guys, I tested without using JobMaster. I thought maybe somehow it is related to running the job

9/23/20

Pulasthi Supun Wickramasinghe, Ahmet Uyar3

9/17/20

Issue when running higher parallelism

Hi Ahmet Thanks Ahmet. I can review it and merge it, this seems like a small fix so no reason to

unread,

Issue when running higher parallelism

Hi Ahmet Thanks Ahmet. I can review it and merge it, this seems like a small fix so no reason to

9/17/20

Ahmet Uyar, … Chathura Widanage13

9/15/20

Seems they have that support. https://github.com/openucx/ucx/blob/

unread,

Seems they have that support. https://github.com/openucx/ucx/blob/

9/15/20

Ahmet Uyar, … Pulasthi Supun Wickramasinghe11

9/15/20

twister2 compile error at victor

HI Ahmet That worked, i did not make any changes in the google-protobuf/BUILD file just linked gcc to

unread,

twister2 compile error at victor

HI Ahmet That worked, i did not make any changes in the google-protobuf/BUILD file just linked gcc to

9/15/20

Ahmet Uyar, Chathura Widanage6

9/4/20

testing ucx with high number of workers

I am testing that. thanks. Ahmet On Fri, Sep 4, 2020 at 6:22 PM Chathura Widanage <

unread,

testing ucx with high number of workers

I am testing that. thanks. Ahmet On Fri, Sep 4, 2020 at 6:22 PM Chathura Widanage <

9/4/20

Pulasthi Supun Wickramasinghe, Ahmet Uyar2

8/28/20

Changes in BatchTSetIWorker

Hi Pulasthi, Yes we have that code in Twister2WorkerStarter.java But, I think we should remove this

unread,

Changes in BatchTSetIWorker

Hi Pulasthi, Yes we have that code in Twister2WorkerStarter.java But, I think we should remove this

8/28/20

Ahmet Uyar, Supun Kamburugamuve2

8/26/20

configuring storage in Twister2

+1 for the two configuration parameters. Best, Supun.. On Wed, Aug 26, 2020 at 8:52 AM Ahmet Uyar

unread,

configuring storage in Twister2

+1 for the two configuration parameters. Best, Supun.. On Wed, Aug 26, 2020 at 8:52 AM Ahmet Uyar

8/26/20

Niranda Perera, Ahmet Uyar3

8/24/20

new branch with tset api changes

Great! :-) thanks Ahmet On Mon, Aug 24, 2020, 15:37 Ahmet Uyar <ahme...@gmail.com> wrote: Hi

unread,

new branch with tset api changes

Great! :-) thanks Ahmet On Mon, Aug 24, 2020, 15:37 Ahmet Uyar <ahme...@gmail.com> wrote: Hi

8/24/20

Geoffrey Fox, Supun Kamburugamuve2

8/19/20

Good datasets and deep learning to look at

Thank you. Will have a look. Best, Supun.. On Wed, Aug 19, 2020 at 9:51 AM Geoffrey Fox <

unread,

Good datasets and deep learning to look at

Thank you. Will have a look. Best, Supun.. On Wed, Aug 19, 2020 at 9:51 AM Geoffrey Fox <

8/19/20

Ahmet Uyar, … Supun Kamburugamuve7

8/13/20

processing 50M tweetID-date pairs per worker

It must be because all workers are trying to use the same disk. When going through HDFS, it controls

unread,

processing 50M tweetID-date pairs per worker

It must be because all workers are trying to use the same disk. When going through HDFS, it controls

8/13/20

Ahmet Uyar, … Chathura Widanage6

8/13/20

java.lang.OutOfMemoryError

Super! On Thu, Aug 13, 2020, 13:49 Ahmet Uyar <ahme...@gmail.com> wrote: It ran successfully

unread,

java.lang.OutOfMemoryError

Super! On Thu, Aug 13, 2020, 13:49 Ahmet Uyar <ahme...@gmail.com> wrote: It ran successfully

8/13/20

Ahmet Uyar, Chathura Widanage25

8/13/20

exception on addInput to ComputeTSet

Great, thanks! Regards, Chathura On Thu, Aug 13, 2020 at 12:38 PM Ahmet Uyar <ahme...@gmail.com

unread,

exception on addInput to ComputeTSet

Great, thanks! Regards, Chathura On Thu, Aug 13, 2020 at 12:38 PM Ahmet Uyar <ahme...@gmail.com

8/13/20

8/10/20

Hi Supun, When I run twister2 with the local disk, it works with 10M tweetID-date pairs. It completed

unread,

Hi Supun, When I run twister2 with the local disk, it works with 10M tweetID-date pairs. It completed

8/10/20

Gregor von Laszewski, Allan Streib2

8/7/20

Install tensorflow with GPU support on Ubuntu 20.04 and anaconda perfomance observations

A couple of comments: The version of Anaconda on Romeo is at this point relatively old. Perhaps worth

unread,

Install tensorflow with GPU support on Ubuntu 20.04 and anaconda perfomance observations

A couple of comments: The version of Anaconda on Romeo is at this point relatively old. Perhaps worth

8/7/20

Ahmet Uyar, … Niranda Perera5

8/4/20

jobID in hdfs path

No it would not. It's okay to use any string there as long as it's deterministic and the same

unread,

jobID in hdfs path

No it would not. It's okay to use any string there as long as it's deterministic and the same

8/4/20

Ahmet Uyar, Chathura Widanage4

8/4/20

build failure with ucx

Great! Regards, Chathura On Tue, Aug 4, 2020 at 11:39 AM Ahmet Uyar <ahme...@gmail.com> wrote

unread,

build failure with ucx

Great! Regards, Chathura On Tue, Aug 4, 2020 at 11:39 AM Ahmet Uyar <ahme...@gmail.com> wrote

8/4/20

Chathura Widanage, … Supun Kamburugamuve4

8/3/20

UCX upgraded to 1.10.0

Restarted the build with caches cleared. On Mon, Aug 3, 2020 at 3:10 PM Chathura Widanage <

unread,

UCX upgraded to 1.10.0

Restarted the build with caches cleared. On Mon, Aug 3, 2020 at 3:10 PM Chathura Widanage <

8/3/20

Ahmet Uyar, Chathura Widanage7

8/3/20

persisting to HDFS

Since UCX PR is getting delayed to merge, I sent the HDFS changes in a different PR. https://github.

unread,

persisting to HDFS

Since UCX PR is getting delayed to merge, I sent the HDFS changes in a different PR. https://github.

8/3/20

Ahmet Uyar, … Supun Kamburugamuve5

8/3/20

workers blocking when reading hdfs files

The current document is in https://twister2.org/ There are few guides on compiling twister2, in mac

unread,

workers blocking when reading hdfs files

The current document is in https://twister2.org/ There are few guides on compiling twister2, in mac

8/3/20

Ahmet Uyar, Niranda Perera3

8/3/20

exception when adding input to ComputeTSet

Hi Niranda, I see that DeleteIDs are persisted successfully. I see checkpointed files. Content of the

unread,

exception when adding input to ComputeTSet

Hi Niranda, I see that DeleteIDs are persisted successfully. I see checkpointed files. Content of the

8/3/20

Ahmet Uyar, Supun Kamburugamuve3

7/29/20

exception when processing 200M tweets per worker

Hi Supun, I ran it with the heap dump turned on. I put the generated heap-dump files under: /

unread,

exception when processing 200M tweets per worker

Hi Supun, I ran it with the heap dump turned on. I put the generated heap-dump files under: /

7/29/20

Niranda Perera, Vibhatha Abeykoon3

7/28/20

Major API changes in TSet

I tried running this SReduceWindowExample with the following command. ./bin/twister2 submit

unread,

Major API changes in TSet

I tried running this SReduceWindowExample with the following command. ./bin/twister2 submit

7/28/20

Supun Kamburugamuve, Ahmet Uyar3

7/27/20

Hi Ahmet, Pipe is an in-memory streaming semantic. So it cannot have iterator afterwords. task1 ->

unread,

Hi Ahmet, Pipe is an in-memory streaming semantic. So it cannot have iterator afterwords. task1 ->

7/27/20

Ahmet Uyar, Chathura Widanage4

7/24/20

useDisk exception

Great, thanks! Regards, Chathura On Fri, Jul 24, 2020 at 12:21 PM Ahmet Uyar <ahme...@gmail.com

unread,

useDisk exception

Great, thanks! Regards, Chathura On Fri, Jul 24, 2020 at 12:21 PM Ahmet Uyar <ahme...@gmail.com

7/24/20

Ahmet Uyar, Supun Kamburugamuve2

7/24/20

tweet ID deletion

Hi Ahmet, I think we can assume the second input is small and keep it in a hash (in-memory). When we

unread,

tweet ID deletion

Hi Ahmet, I think we can assume the second input is small and keep it in a hash (in-memory). When we

7/24/20

Search

Clear search

Close search

Google apps

Main menu