Died on std::runtime_error: `packetshell': process exited with failure status 1

[johnso...@gmail.com]

Dear pantheon group members, I was using mahimahi for pantheon and passing the mahimahi cmd into the receiver. However, it shows the error below:

No handlers could be found for logger "visdom"

[Environment] cleanup success! Preparing to start episode 1, sleep for a while

[Environment] start sender...

[sender] Listening on port 1600

[Environment] start receiver...

$ mm-link --uplink-queue=droptail --uplink-queue-args=packets=30000 -- sh -c 'python /home/xia/CC/pantheon/third_party/glider/main/run_receiver.py $MAHIMAHI_BASE 1600'

Died on std::runtime_error: dnsmasq: did not start after 20 attempts

Died on std::runtime_error: `packetshell': process exited with failure status 1

And the code is:

self.receiver = subprocess.Popen(cmd, preexec_fn=os.setsid, shell=True,

                                             stdin=subprocess.PIPE, stdout=subprocess.PIPE)

cmd:

$ mm-link --uplink-queue=droptail --uplink-queue-args=packets=30000 -- sh -c 'python /home/xia/CC/pantheon/third_party/glider/main/run_receiver.py $MAHIMAHI_BASE 1600'

I would like to know how to solve this problem.

Thanks for your time.

Best regard,

Johnson

[Keith Winstein]

Hello Johnson,

This might be a problem with the underlying Mahimahi emulators, rather than anything having to do with Pantheon. What happens if you run "mm-delay 10" on this system? Do you get a shell (the prompt might look like "[delay 10 ms] user@machine:~$") and can you ping the outside world from within that shell? Or do you get the same error message?

If the latter, see https://github.com/ravinet/mahimahi/issues/143 -- this can be caused by a firewall configuration on the system.

Best regards,

Keith

--
You received this message because you are subscribed to the Google Groups "Pantheon" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pantheon-stanf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pantheon-stanford/402a99aa-535f-4eb0-9a1c-c1513844839ao%40googlegroups.com.

[johnso...@gmail.com]

Thanks for the reply.

After running mm-delay 10, the problem has been solved.

However, There's another issue:

(pth2.7tfagt) [tunnelserver] [delay 10 ms] xia@xia-virtual-machine:~/CC/pantheon$ python src/experiments/test.py local --schemes "glider"

/home/xia/CC/pantheon/src/helpers/utils.py:40: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

  return yaml.load(config)

$ /home/xia/CC/pantheon/src/experiments/git_summary.sh

Testing scheme glider for experiment run 1/1...

$ /home/xia/CC/pantheon/src/wrappers/glider.py run_first

[tunnel server manager (tsm)] $ python /home/xia/CC/pantheon/src/experiments/tunnel_manager.py

tunnel manager is running

[tunnel client manager (tcm)] prompt [tsm]

$ mm-link /home/xia/CC/pantheon/src/experiments/12mbps.trace /home/xia/CC/pantheon/src/experiments/12mbps.trace --uplink-log=/home/xia/CC/pantheon/src/experiments/data/glider_mm_datalink_run1.log --downlink-log=/home/xia/CC/pantheon/src/experiments/data/glider_mm_acklink_run1.log python /home/xia/CC/pantheon/src/experiments/tunnel_manager.py

tunnel manager is running

[tsm] tunnel 1 mm-tunnelserver --ingress-log=/home/xia/CC/pantheon/tmp/glider_datalink_run1_flow1_uid9a716f3c-b7b4-4cc8-83db-ceeb9dcd4881.log.ingress --egress-log=/home/xia/CC/pantheon/tmp/glider_acklink_run1_flow1_uid9a716f3c-b7b4-4cc8-83db-ceeb9dcd4881.log.egress

prompt [tcm]

[tsm] tunnel 1 readline

[tcm] tunnel 1 mm-tunnelclient $MAHIMAHI_BASE 41015 100.64.0.6 100.64.0.5 --ingress-log=/home/xia/CC/pantheon/tmp/glider_acklink_run1_flow1_uid9a716f3c-b7b4-4cc8-83db-ceeb9dcd4881.log.ingress --egress-log=/home/xia/CC/pantheon/tmp/glider_datalink_run1_flow1_uid9a716f3c-b7b4-4cc8-83db-ceeb9dcd4881.log.egress

[tcm] tunnel 1 readline

Tunnelclient listening for server on port 54690

Tunnelclient received no response from tunnelserver, retrying 1/5

Tunnelclient ignoring exception sending a syn: Died on unix_error: write: Connection refused

Tunnelclient received no response from tunnelserver, retrying 2/5

Tunnelclient received no response from tunnelserver, retrying 3/5

Tunnelclient ignoring exception sending a syn: Died on unix_error: write: Connection refused

Tunnelclient received no response from tunnelserver, retrying 4/5

Tunnelclient received no response from tunnelserver, retrying 5/5

Tunnelclient ignoring exception sending a syn: Died on unix_error: write: Connection refused

Failed to connect to tunnel server after 5 tries, exiting..

Tunnel connection timeout

[tcm] tunnel 1 mm-tunnelclient $MAHIMAHI_BASE 41015 100.64.0.6 100.64.0.5 --ingress-log=/home/xia/CC/pantheon/tmp/glider_acklink_run1_flow1_uid9a716f3c-b7b4-4cc8-83db-ceeb9dcd4881.log.ingress --egress-log=/home/xia/CC/pantheon/tmp/glider_datalink_run1_flow1_uid9a716f3c-b7b4-4cc8-83db-ceeb9dcd4881.log.egress

[tcm] tunnel 1 readline

Tunnelclient listening for server on port 51766

Tunnelclient received no response from tunnelserver, retrying 1/5

Tunnelclient ignoring exception sending a syn: Died on unix_error: write: Connection refused

Tunnelclient received no response from tunnelserver, retrying 2/5

Tunnelclient received no response from tunnelserver, retrying 3/5

Tunnelclient ignoring exception sending a syn: Died on unix_error: write: Connection refused

Tunnelclient received no response from tunnelserver, retrying 4/5

Tunnelclient received no response from tunnelserver, retrying 5/5

Tunnelclient ignoring exception sending a syn: Died on unix_error: write: Connection refused

Failed to connect to tunnel server after 5 tries, exiting..

Tunnel connection timeout

[tcm] tunnel 1 mm-tunnelclient $MAHIMAHI_BASE 41015 100.64.0.6 100.64.0.5 --ingress-log=/home/xia/CC/pantheon/tmp/glider_acklink_run1_flow1_uid9a716f3c-b7b4-4cc8-83db-ceeb9dcd4881.log.ingress --egress-log=/home/xia/CC/pantheon/tmp/glider_datalink_run1_flow1_uid9a716f3c-b7b4-4cc8-83db-ceeb9dcd4881.log.egress

[tcm] tunnel 1 readline

Tunnelclient listening for server on port 43527

Tunnelclient received no response from tunnelserver, retrying 1/5

Tunnelclient ignoring exception sending a syn: Died on unix_error: write: Connection refused

Tunnelclient received no response from tunnelserver, retrying 2/5

Tunnelclient received no response from tunnelserver, retrying 3/5

Tunnelclient ignoring exception sending a syn: Died on unix_error: write: Connection refused

Tunnelclient received no response from tunnelserver, retrying 4/5

Tunnelclient received no response from tunnelserver, retrying 5/5

Tunnelclient ignoring exception sending a syn: Died on unix_error: write: Connection refused

Failed to connect to tunnel server after 5 tries, exiting..

Tunnel connection timeout

Unable to establish tunnel

kill_proc_group: killed process group with pgid 54636

kill_proc_group: killed process group with pgid 54637

Error in testing scheme glider with run ID 1

All tests done!

kill_proc_group: killed process group with pgid 54644

tunnel_manager: caught signal 15 and cleaned up

kill_proc_group: killed process group with pgid 54976

tunnel_manager: caught signal 15 and cleaned up

Since the issue: https://groups.google.com/forum/#!topic/pantheon-stanford/ASAZdjq5-vA which is almost the same, I tried mm-tunnelserver/mm-client and failed.

I would like to know whether there's any way to solve this problem.

Thanks again for your time.

Best regards,

Johnson

Keith Winstein於 2020年8月5日星期三 UTC+8下午5時28分47秒寫道：

To unsubscribe from this group and stop receiving emails from it, send an email to pantheon...@googlegroups.com.

[Francis Y. Yan]

Hi Johnson,

What error message was output after running mm-tunnelserver in one terminal and its output in another?

Best,

Francis

To unsubscribe from this group and stop receiving emails from it, send an email to pantheon-stanf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pantheon-stanford/b79103b7-cc29-4633-b5ed-c1c29d4296d8o%40googlegroups.com.

[johnso...@gmail.com]

Hi Francis,

This is the output after running mm-tunnelserver:

(base) xia@xia-virtual-machine:~/CC/pantheon$ mm-tunnelserver

mm-tunnelclient localhost 47200 100.64.0.6 100.64.0.5

Tunnelserver got connection from tunnelclient

And this is the output after running tunnelclient:

(base) xia@xia-virtual-machine:~/CC/pantheon$ mm-tunnelclient localhost 47200 100.64.0.6 100.64.0.5

Tunnelclient listening for server on port 58135

Tunnelclient got connection from tunnelserver at 127.0.0.1

There's no error after running both of them. However, I discover that the establishment of the connection and the running of the test is under different environments(base / pth2.7tfagt(The env I create by conda)).

Does that required the same environment?

Best regard,

Johnson

Francis Y. Yan於 2020年8月11日星期二 UTC+8上午1時35分11秒寫道：

To view this discussion on the web visit https://groups.google.com/d/msgid/pantheon-stanford/b79103b7-cc29-4633-b5ed-c1c29d4296d8o%40googlegroups.com.

[Francis Y. Yan]

Hi Johnson,

It looks that the tunnel server is correctly brought up on 100.64.0.5 and the tunnel client is running on 100.64.0.6 on your machine. You can further verify that everything works by running a pair of iperf server and client inside the Pantheon tunnel.

I haven't tested different Python environments, which might be the reason why the test failed, so I would recommend running the test within the same virtual environment.

Best,

Francis 

To unsubscribe from this group and stop receiving emails from it, send an email to pantheon-stanf...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pantheon-stanford/f3ee7365-602e-45be-9905-b2e71c973df6o%40googlegroups.com.