socket timeout errors

5 views

Skip to first unread message

Anne-Marie...@bull.com

unread,

Jun 7, 2006, 2:56:44 PM6/7/06

to slur...@lists.llnl.gov

Hello,

I would like some help please to figure out what problem we are experiencing.

We have 32 nodes, running an mpi "hellow" program that works with smaller number of processes but gets "socket timeout" errors with larger number of processes.

We are going to experiment with increasing the SLURM_MESSAGE_TIMEOUT_MSEC_STATIC value.

Any ideas on what possible reasons for the timeout errors?

Thanks for your help,
Anne Marie

---------------
slurm config:
Configuration data as of 06/07-20:47:22
AuthType = auth/none
BackupAddr = (null)
BackupController = (null)
CacheGroups = 1
CheckpointType = checkpoint/none
ControlAddr = 15.0.0.1
ControlMachine = lynx0
Epilog = (null)
FastSchedule = 0
FirstJobId = 1000
InactiveLimit = 0
JobAcctLogFile = /var/log/slurm/slurm_acct.log
JobAcctFrequency = 30
JobAcctType = jobacct/none
JobCompLoc = /var/log/slurm/slurm.job.log
JobCompType = jobcomp/filetxt
JobCredentialPrivateKey = /opt/slurm/etc/slurm.key
JobCredentialPublicCertificate = /opt/slurm/etc/slurm.cert
KillWait = 30
MaxJobCount = 2000
MinJobAge = 300
MpiDefault = mvapich
PluginDir = /opt/slurm/lib/slurm
PlugStackConfig = /opt/slurm/etc/plugstack.conf
ProctrackType = proctrack/pgid
Prolog = (null)
PropagatePrioProcess = 0
PropagateResourceLimits = ALL
PropagateResourceLimitsExcept = (null)
ReturnToService = 1
SchedulerAuth = (null)
SchedulerPort = 0
SchedulerRootFilter = 1
SchedulerType = sched/builtin
SelectType = select/linear
SlurmUser = root(0)
SlurmctldDebug = 4
SlurmctldLogFile = (null)
SlurmctldPidFile = /var/run/slurmctld.pid
SlurmctldPort = 6817
SlurmctldTimeout = 900
SlurmdDebug = 4
SlurmdLogFile = (null)
SlurmdPidFile = /var/run/slurmd.pid
SlurmdPort = 6818
SlurmdSpoolDir = /var/log/slurm/log_slurmd/
SlurmdTimeout = 900
SLURM_CONFIG_FILE = /opt/slurm/etc/slurm.conf
SLURM_VERSION = 1.1.1
SrunProlog = (null)
SrunEpilog = (null)
StateSaveLocation = /var/log/slurm/log_slurmctld
SwitchType = switch/none
TaskEpilog = (null)
TaskPlugin = task/affinity
TaskProlog = (null)
TmpFS = /tmp
TreeWidth = 2000
UsePam = 0
WaitTime = 0

Slurmctld(primary/backup) at lynx0/(NULL) are UP/DOWN
---------------

[ccrt@lynx0 HELLO]$ srun -n 256 -N 32 -o trace.log -pparallel ./a.out srun: error: lynx24: task151: Exited with exit code 13 srun: error: lynx25: task157: Exited with exit code 13 srun: error: lynx20: task118: Exited with exit code 13 srun: error: lynx27: task172: Exited with exit code 13 srun: error: lynx28: task178: Exited with exit code 13 srun: error: lynx35: task234: Exited with exit code 13 srun: error: lynx30: task192: Exited with exit code 13 srun: error: lynx31: task205: Exited with exit code 13 srun: error: lynx26: task163: Exited with exit code 13 srun: error: lynx33: task218: Exited with exit code 13 srun: error: lynx34: task225: Exited with exit code 13 srun: error: lynx23: task142: Exited with exit code 13 srun: error: lynx36: task245: Exited with exit code 13 srun: error: lynx37: task253: Exited with exit code 13 srun: error: lynx32: task209: Exited with exit code 13 srun: error: lynx27: task173: Exited with exit code 13 srun: error: lynx28: task182: Exited with exit code 13 srun: error: lynx29: task186: Exited with exit code 13 srun: error: lynx6: task6: Exited with exit code 13 srun: error: lynx25: task158: Exited with exit code 13 srun: error: lynx26: task160: Exited with exit code 13 srun: error: lynx33: task216: Exited with exit code 13 srun: error: lynx34: task226: Exited with exit code 13 srun: error: lynx35: task238: Exited with exit code 13 srun: error: lynx30: task193: Exited with exit code 13 srun: error: lynx31: task206: Exited with exit code 13 srun: error: lynx32: task211: Exited with exit code 13 srun: error: lynx33: task217: Exited with exit code 13 srun: error: lynx28: task177: Exited with exit code 13 srun: error: lynx29: task187: Exited with exit code 13 srun: error: lynx24: task149: Exited with exit code 13 srun: error: lynx37: task252: Exited with exit code 13 srun: error: lynx26: task161: Exited with exit code 13 srun: error: lynx27: task175: Exited with exit code 13 srun: error: lynx34: task227: Exited with exit code 13 srun: error: lynx23: task141: Exited with exit code 13 srun: error: lynx36: task247: Exited with exit code 13 srun: error: lynx25: task154: Exited with exit code 13 srun: error: lynx32: task210: Exited with exit code 13 srun: error: lynx27: task171: Exited with exit code 13 srun: error: lynx28: task183: Exited with exit code 13 srun: error: lynx29: task184: Exited with exit code 13 srun: error: lynx30: task198: Exited with exit code 13 srun: error: lynx31: task203: Exited with exit code 13 srun: error: lynx26: task167: Exited with exit code 13 srun: error: lynx33: task219: Exited with exit code 13 srun: error: lynx34: task231: Exited with exit code 13 srun: error: lynx35: task235: Exited with exit code 13 srun: error: lynx24: task145: Exited with exit code 13 srun: error: lynx37: task255: Exited with exit code 13 srun: error: lynx32: task214: Exited with exit code 13 srun: error: lynx27: task168: Exited with exit code 13 srun: error: lynx28: task179: Exited with exit code 13 srun: error: lynx35: task239: Exited with exit code 13 srun: error: lynx36: task244: Exited with exit code 13 srun: error: lynx31: task200: Exited with exit code 13 srun: error: lynx26: task165: Exited with exit code 13 srun: error: lynx33: task221: Exited with exit code 13 srun: error: lynx34: task224: Exited with exit code 13 srun: error: lynx29: task188: Exited with exit code 13 srun: error: lynx24: task144: Exited with exit code 13 srun: error: lynx37: task249: Exited with exit code 13 srun: error: lynx32: task212: Exited with exit code 13 srun: error: lynx33: task222: Exited with exit code 13 srun: error: lynx28: task181: Exited with exit code 13 srun: error: lynx35: task232: Exited with exit code 13 srun: error: lynx30: task196: Exited with exit code 13 srun: error: lynx37: task251: Exited with exit code 13 srun: error: lynx26: task164: Exited with exit code 13 srun: error: lynx27: task170: Exited with exit code 13 srun: error: lynx34: task228: Exited with exit code 13 srun: error: lynx29: task189: Exited with exit code 13 srun: error: lynx36: task246: Exited with exit code 13 srun: error: lynx31: task207: Exited with exit code 13 srun: error: lynx32: task208: Exited with exit code 13 srun: error: lynx33: task223: Exited with exit code 13 srun: error: lynx34: task229: Exited with exit code 13 srun: error: lynx35: task233: Exited with exit code 13 srun: error: lynx36: task242: Exited with exit code 13 srun: error: lynx37: task248: Exited with exit code 13 srun: error: lynx32: task215: Exited with exit code 13 srun: error: lynx27: task174: Exited with exit code 13 srun: error: lynx34: task230: Exited with exit code 13 srun: error: lynx29: task191: Exited with exit code 13 srun: error: lynx30: task197: Exited with exit code 13 srun: error: lynx37: task250: Exited with exit code 13 srun: error: lynx36: task240: Exited with exit code 13 srun: error: lynx30: task199: Exited with exit code 13 srun: error: lynx36: task241: Exited with exit code 13 srun: error: lynx30: task195: Exited with exit code 13 srun: error: lynx6: task2: Exited with exit code 13 srun: error: lynx7: task11: Exited with exit code 13 srun: error: lynx28: task176: Exited with exit code 13 srun: error: lynx23: task143: Exited with exit code 13 srun: error: lynx20: task119: Exited with exit code 13 srun: error: lynx19: task104: Exited with exit code 13 srun: error: lynx32: task213: Exited with exit code 13 srun: error: lynx18: task96: Exited with exit code 13 srun: error: slurm_receive_msg: Zero Bytes were transmitted or received srun: error: slurm_receive_msg[15.0.0.19]: Zero Bytes were transmitted or received srun: error: lynx22: task128: Exited with exit code 13 srun: error: lynx7: task9: Exited with exit code 13 srun: error: lynx7: task8: Exited with exit code 13 srun: error: slurm_receive_msg: Zero Bytes were transmitted or received srun: error: slurm_receive_msg[15.0.0.23]: Zero Bytes were transmitted or received srun: error: lynx11: task45: Exited with exit code 13 srun: error: slurm_receive_msg: Zero Bytes were transmitted or received srun: error: slurm_receive_msg[15.0.0.33]: Zero Bytes were transmitted or received srun: error: slurm_receive_msg: Zero Bytes were transmitted or received srun: error: slurm_receive_msg[15.0.0.12]: Zero Bytes were transmitted or received srun: error: lynx14: task68: Exited with exit code 13 srun: error: slurm_receive_msg: Zero Bytes were transmitted or received srun: error: slurm_receive_msg[15.0.0.15]: Zero Bytes were transmitted or received srun: error: lynx7: task13: Exited with exit code 13 srun: error: lynx6: task3: Exited with exit code 13 srun: error: lynx12: task49: Exited with exit code 13 srun: error: lynx7: task10: Exited with exit code 13 srun: error: lynx21: task122: Exited with exit code 13 srun: error: slurm_receive_msg: Zero Bytes were transmitted or received srun: error: slurm_receive_msg[15.0.0.8]: Zero Bytes were transmitted or received srun: error: lynx6: task7: Exited with exit code 13 srun: error: lynx6: task0: Exited with exit code 13 srun: error: slurm_receive_msg: Socket timed out on send/recv operation srun: error: slurm_receive_msg[15.0.0.38]: Socket timed out on send/recv operation srun: error: lynx24: task150: Exited with exit code 13 srun: error: lynx25: task152: Exited with exit code 13 srun: error: lynx8: task22: Exited with exit code 13 srun: error: lynx9: task27: Exited with exit code 13 srun: error: lynx10: task39: Exited with exit code 13 srun: error: lynx23: task137: Exited with exit code 13 srun: error: lynx6: task4: Exited with exit code 13 srun: error: lynx31: task202: Exited with exit code 13 srun: error: lynx8: task16: Exited with exit code 13 srun: error: lynx9: task26: Exited with exit code 13 srun: error: lynx22: task134: Exited with exit code 13 srun: error: lynx23: task139: Exited with exit code 13 srun: error: lynx24: task147: Exited with exit code 13 srun: error: lynx19: task111: Exited with exit code 13 srun: error: lynx8: task19: Exited with exit code 13 srun: error: lynx27: task169: Exited with exit code 13 srun: error: lynx22: task131: Exited with exit code 13 srun: error: lynx35: task236: Exited with exit code 13 srun: error: lynx12: task50: Exited with exit code 13 srun: error: lynx7: task14: Exited with exit code 13 srun: error: lynx20: task115: Exited with exit code 13 srun: error: lynx9: task28: Exited with exit code 13 srun: error: lynx22: task133: Exited with exit code 13 srun: error: lynx11: task40: Exited with exit code 13 srun: error: lynx12: task52: Exited with exit code 13 srun: error: lynx7: task15: Exited with exit code 13 srun: error: lynx26: task166: Exited with exit code 13 srun: error: lynx9: task29: Exited with exit code 13 srun: error: lynx10: task32: Exited with exit code 13 srun: error: lynx11: task47: Exited with exit code 13 srun: error: lynx12: task55: Exited with exit code 13 srun: error: lynx13: task62: Exited with exit code 13 srun: error: lynx26: task162: Exited with exit code 13 srun: error: lynx15: task77: Exited with exit code 13 srun: error: lynx10: task33: Exited with exit code 13 srun: error: lynx29: task190: Exited with exit code 13 srun: error: lynx24: task148: Exited with exit code 13 srun: error: lynx19: task109: Exited with exit code 13 srun: error: lynx8: task20: Exited with exit code 13 srun: error: lynx9: task30: Exited with exit code 13 srun: error: lynx10: task36: Exited with exit code 13 srun: error: lynx17: task88: Exited with exit code 13 srun: error: lynx12: task54: Exited with exit code 13 srun: error: lynx13: task56: Exited with exit code 13 srun: error: lynx14: task66: Exited with exit code 13 srun: error: lynx9: task31: Exited with exit code 13 srun: error: lynx10: task35: Exited with exit code 13 srun: error: lynx11: task42: Exited with exit code 13 srun: error: lynx12: task48: Exited with exit code 13 srun: error: lynx13: task63: Exited with exit code 13 srun: error: lynx8: task18: Exited with exit code 13 srun: error: lynx9: task24: Exited with exit code 13 srun: error: lynx10: task38: Exited with exit code 13 srun: error: lynx35: task237: Exited with exit code 13 srun: error: lynx36: task243: Exited with exit code 13 srun: error: lynx13: task60: Exited with exit code 13 srun: error: lynx20: task112: Exited with exit code 13 srun: error: lynx15: task73: Exited with exit code 13 srun: error: lynx16: task82: Exited with exit code 13 srun: error: lynx17: task91: Exited with exit code 13 srun: error: lynx6: task5: Exited with exit code 13 srun: error: lynx31: task201: Exited with exit code 13 srun: error: lynx14: task69: Exited with exit code 13 srun: error: lynx33: task220: Exited with exit code 13 srun: error: lynx10: task34: Exited with exit code 13 srun: error: lynx17: task89: Exited with exit code 13 srun: error: lynx18: task100: Exited with exit code 13 srun: error: lynx13: task61: Exited with exit code 13 srun: error: lynx8: task21: Exited with exit code 13 srun: error: lynx15: task78: Exited with exit code 13 srun: error: lynx28: task180: Exited with exit code 13 srun: error: lynx29: task185: Exited with exit code 13 srun: error: lynx18: task101: Exited with exit code 13 srun: error: lynx31: task204: Exited with exit code 13 srun: error: lynx14: task67: Exited with exit code 13 srun: error: lynx9: task25: Exited with exit code 13 srun: error: lynx16: task84: Exited with exit code 13 srun: error: lynx17: task93: Exited with exit code 13 srun: error: lynx18: task97: Exited with exit code 13 srun: error: lynx13: task57: Exited with exit code 13 srun: error: lynx8: task17: Exited with exit code 13 srun: error: lynx15: task79: Exited with exit code 13 srun: error: lynx16: task85: Exited with exit code 13 srun: error: lynx17: task90: Exited with exit code 13 srun: error: lynx6: task1: Exited with exit code 13 srun: error: lynx13: task59: Exited with exit code 13 srun: error: lynx14: task71: Exited with exit code 13 srun: error: lynx21: task124: Exited with exit code 13 srun: error: lynx16: task83: Exited with exit code 13 srun: error: lynx23: task136: Exited with exit code 13 srun: error: lynx18: task103: Exited with exit code 13 srun: error: lynx7: task12: Exited with exit code 13 srun: error: lynx8: task23: Exited with exit code 13 srun: error: lynx21: task123: Exited with exit code 13 srun: error: lynx16: task86: Exited with exit code 13 srun: error: lynx17: task94: Exited with exit code 13 srun: error: lynx30: task194: Exited with exit code 13 srun: error: lynx19: task105: Exited with exit code 13 srun: error: lynx14: task70: Exited with exit code 13 srun: error: lynx21: task126: Exited with exit code 13 srun: error: lynx16: task87: Exited with exit code 13 srun: error: lynx23: task138: Exited with exit code 13 srun: error: lynx18: task99: Exited with exit code 13 srun: error: lynx19: task107: Exited with exit code 13 srun: error: lynx14: task65: Exited with exit code 13 srun: error: lynx21: task120: Exited with exit code 13 srun: error: lynx16: task80: Exited with exit code 13 srun: error: lynx17: task95: Exited with exit code 13 srun: error: lynx18: task102: Exited with exit code 13 srun: error: lynx37: task254: Exited with exit code 13 srun: error: lynx20: task117: Exited with exit code 13 srun: error: lynx21: task125: Exited with exit code 13 srun: error: lynx22: task130: Exited with exit code 13 srun: error: lynx23: task140: Exited with exit code 13 srun: error: lynx12: task53: Exited with exit code 13 srun: error: lynx19: task108: Exited with exit code 13 srun: error: lynx20: task114: Exited with exit code 13 srun: error: lynx21: task127: Exited with exit code 13 srun: error: lynx22: task129: Exited with exit code 13 srun: error: lynx18: task98: Exited with exit code 13 srun: error: lynx25: task153: Exited with exit code 13 srun: error: lynx20: task113: Exited with exit code 13 srun: error: lynx19: task110: Exited with exit code 13 srun: error: lynx20: task116: Exited with exit code 13 srun: error: lynx25: task159: Exited with exit code 13 srun: error: lynx19: task106: Exited with exit code 13 srun: error: lynx25: task156: Exited with exit code 13 srun: error: lynx25: task155: Exited with exit code 13 srun: error: lynx10: task37: Exited with exit code 13 srun: error: slurm_receive_msg: Socket timed out on send/recv operation srun: error: slurm_receive_msg[15.0.0.32]: Socket timed out on send/recv operation srun: error: lynx13: task58: Exited with exit code 13 srun: error: slurm_receive_msg: Socket timed out on send/recv operation srun: error: slurm_receive_msg[15.0.0.28]: Socket timed out on send/recv operation srun: error: slurm_receive_msg: Zero Bytes were transmitted or received srun: error: slurm_receive_msg[15.0.0.11]: Zero Bytes were transmitted or received srun: error: slurm_receive_msg: Zero Bytes were transmitted or received srun: error: slurm_receive_msg[15.0.0.14]: Zero Bytes were transmitted or received srun: error: slurm_receive_msg: Zero Bytes were transmitted or received srun: error: slurm_receive_msg[15.0.0.26]: Zero Bytes were transmitted or received srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: slurm_receive_msg: Socket timed out on send/recv operation srun: error: slurm_receive_msg[15.0.0.8]: Socket timed out on send/recv operation srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: _server_write write failed: Broken pipe srun: error: slurm_receive_msg: Socket timed out on send/recv operation srun: error: slurm_receive_msg[15.0.0.22]: Socket timed out on send/recv operation srun: error: slurm_receive_msg: Socket timed out on send/recv operation srun: error: slurm_receive_msg[15.0.0.13]: Socket timed out on send/recv operation srun: error: slurm_receive_msg: Socket timed out on send/recv operation srun: error: slurm_receive_msg[15.0.0.7]: Socket timed out on send/recv operation srun: error: _server_write write failed: Connection reset by peer srun: error: _server_write write failed: Connection reset by peer srun: error: _server_write write failed: Connection reset by peer srun: error: _server_write write failed: Connection reset by peer srun: error: _server_write write failed: Connection reset by peer srun: error: _server_write write failed: Connection reset by peer srun: error: _server_write write failed: Connection reset by peer srun: error: _server_write write failed: Connection reset by peer srun: error: _server_write write failed: Connection reset by peer srun: error: lynx11: task43: Exited with exit code 13 srun: error: lynx12: task51: Exited with exit code 13 srun: error: lynx11: task41: Exited with exit code 13 srun: error: lynx11: task44: Exited with exit code 13 srun: error: lynx21: task121: Exited with exit code 13 srun: error: lynx22: task135: Exited with exit code 13 srun: error: lynx22: task132: Exited with exit code 13 srun: error: lynx15: task75: Exited with exit code 13 srun: error: lynx16: task81: Exited with exit code 13 srun: error: lynx17: task92: Exited with exit code 13 srun: error: lynx24: task146: Exited with exit code 13 srun: error: lynx15: task72: Exited with exit code 13 srun: error: lynx15: task74: Exited with exit code 13 srun: error: lynx15: task76: Exited with exit code 13 srun: error: lynx11: task46: Exited with exit code 13 srun: error: lynx14: task64: Exited with exit code 13 [ccrt@lynx0 HELLO]$ a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_msg_sendto: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_msg_sendto: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_msg_sendto: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed
a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_msg_sendto: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_msg_sendto: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_msg_sendto: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_msg_sendto: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_msg_sendto: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed
MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_receive_msg: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_msg_sendto: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_msg_sendto: Socket timed out on send/recv operation a.out: error: slurm_get_kvs_comm_set: Socket timed out on send/recv operation Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1a.out: error: slurm_get_kvs_comm_set: Connection timed out Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(225)...........: Initialization failed MPID_Init(127)..................: channel initialization failed MPIDI_CH3_Init(95)..............: Channel init failed MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1

trace.log

je...@llnl.gov

unread,

Jun 7, 2006, 9:12:21 PM6/7/06

to slur...@lists.llnl.gov, Anne-Marie...@bull.com

You need to relink a.out with the new library to get the new limits.

The default timeout on messages is 5 seconds, which seems sufficient

on even the largest systems here. Do you have any idea why communications

are so slow on your system?

At 3:48 PM -0700 6/7/06, Anne-Marie...@bull.com wrote:

Changed the following -- I did this before seeing your mail message ...
#define SLURM_MESSAGE_TIMEOUT_MSEC_STATIC 1000000
#define SLURM_MESSAGE_TIMEOUT_SEC_STATIC 1000

-- did not get the socket timed out this time, but saw the error below.

Thanks for your help,
Anne Marie

[ccrt@lynx0 HELLO]$ srun -n256 -N32 -pparallel -l -d 4 -o slurm-trace.log ./a.out

in the slurm-trace.log:
.....
011: In: PMI_KVS_Commit
049: a.out: error: slurm_get_kvs_comm_set: Connection timed out
049: Fatal error in MPI_Init: Other MPI error, error stack:
049: MPIR_Init_thread(225)...........: Initialization failed
049: MPID_Init(127)..................: channel initialization failed
049: MPIDI_CH3_Init(95)..............: Channel init failed
049: MPIDI_CH3I_SMP_pull_header(1092): PMI_KVS_Commit returned -1

Danny Auble <d...@llnl.gov>
Sent by: owner-s...@lists.llnl.gov

06/07/2006 12:07 PM

Please respond to
slur...@lists.llnl.gov

To

slur...@lists.llnl.gov

cc

Subject

RE: socket timeout errors

Anne Marie could you set the SLURM_MESSAGE_TIMEOUT_MSEC_STATIC to 15000 and SLURM_MESSAGE_TIMEOUT_SEC_STATIC to 5 and see if you get errors?

Thanks,
Danny

Attachment converted: Macintosh HD:slurm-trace.log ( / ) (0018D878)

--

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Morris "Moe" Jette       jet...@llnl.gov                 925-423-4856
Integrated Computational Resource Management Group   fax 925-423-6961
Livermore Computing            Lawrence Livermore National Laboratory
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Anne-Marie...@bull.com

unread,

Jun 7, 2006, 6:48:58 PM6/7/06

to slur...@lists.llnl.gov

slurm-trace.log

Reply all

Reply to author

Forward

0 new messages