testing ucx with high number of workers

[Ahmet Uyar]

Hi Chathura,

I tested ucx by running terasort with more than 200 workers at victor in standalone mode. 

Sometimes, it works, but most of the time, it throws an exception. 

Following exception is thrown: 

Caused by: org.openucx.jucx.UcxException: Destination is unreachable
        at org.openucx.jucx.ucp.UcpListener.createUcpListener(Native Method)
        at org.openucx.jucx.ucp.UcpListener.<init>(UcpListener.java:25)
        at org.openucx.jucx.ucp.UcpWorker.newListener(UcpWorker.java:49)
        at edu.iu.dsc.tws.comms.ucx.TWSUCXChannel.createUXCWorker(TWSUCXChannel.java:100)
        at edu.iu.dsc.tws.comms.ucx.TWSUCXChannel.<init>(TWSUCXChannel.java:86)
        ... 13 more

I attached the logs. 

Ahmet

[Chathura Widanage]

Hi Ahmet,

When it randomly works, do you still see below logs?

[1599227712.059761] [v-014:184865:0]       context.cc:48   UCX  WARN  JUCX: no such key UCX_SOCKADDR_TLS_PRIORITY, ignoring
[1599227712.187487] [v-011:159084:0]   ucp_listener.c:429  UCX  ERROR none of the available transports can listen for connections on 172.29.200.211:39732

Regards,

Chathura

--
You received this message because you are subscribed to the Google Groups "Twister2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to twister2+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/twister2/CAPBRfYfifECHt5Mno1MeK49R8efEHy%3Dk2a_cXFO-f66rAOF4pg%40mail.gmail.com.

[Ahmet Uyar]

WARN messages are always there, but the ERROR message is printed only on failures. 

Ahmet

[Chathura Widanage]

I raised this as an issue in the UCX mailing list. Let's wait for their response.

Regards,

Chathura

[Chathura Widanage]

Ahmet,

I have applied a small fix based on Peter's feedback. Could you please rerun with below change.

https://github.com/DSC-SPIDAL/twister2/pull/945

Regards,

Chathura

[Ahmet Uyar]

I am testing that. thanks. 

Ahmet