Frequent mobility thread crash due to assertion in cmd()

[alexande...@gmail.com]

Hi

I've been encountering this error a lot while working with mobility code, and was wondering if there was a suggested solution to avoid it as it renders runs useless generally which has made debugging other issues frustrating.

Traceback (most recent call last):

  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/dist-packages/mininet_wifi-2.6-py3.8.egg/mn_wifi/mobility.py", line 171, in parameters
    self.config_links(mob_nodes)
  File "/usr/local/lib/python3.8/dist-packages/mininet_wifi-2.6-py3.8.egg/mn_wifi/mobility.py", line 195, in config_links
    ack = self.check_in_range(intf, ap_intf)
  File "/usr/local/lib/python3.8/dist-packages/mininet_wifi-2.6-py3.8.egg/mn_wifi/mobility.py", line 139, in check_in_range
    self.ap_out_of_range(intf, ap_intf)
  File "/usr/local/lib/python3.8/dist-packages/mininet_wifi-2.6-py3.8.egg/mn_wifi/mobility.py", line 105, in ap_out_of_range
    intf.disconnect(ap_intf)
  File "/usr/local/lib/python3.8/dist-packages/mininet_wifi-2.6-py3.8.egg/mn_wifi/link.py", line 515, in disconnect
    self.iwdev_cmd('{} disconnect'.format(self.name))
  File "/usr/local/lib/python3.8/dist-packages/mininet_wifi-2.6-py3.8.egg/mn_wifi/link.py", line 117, in iwdev_cmd
    return self.cmd('iw dev', *args)
  File "/usr/local/lib/python3.8/dist-packages/mininet/link.py", line 70, in cmd
    return self.node.cmd( *args, **kwargs )
  File "/usr/local/lib/python3.8/dist-packages/mininet/node.py", line 386, in cmd
    self.sendCmd( *args, **kwargs )
  File "/usr/local/lib/python3.8/dist-packages/mininet/node.py", line 303, in sendCmd
    assert self.shell and not self.waiting

AssertionError

Thanks,

AL

[Ramon Fontes]

Hello,

How can I reproduce the issue?

Sent from my android

--
You received this message because you are subscribed to the Google Groups "mininet-wifi-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mininet-wifi-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mininet-wifi-discuss/fbf454a8-f767-4caf-85a4-0432b3260600n%40googlegroups.com.

[alexande...@gmail.com]

Sorry on the delay, was looking into this and had some other issues arise. It's hard to send a test case because it's to some extent a race condition- essentially a main script that makes calls to node.cmd() heavily seems to have a chance of overlapping with calls to node.cmd() from the mobility thread (mostly related to association code), which leads to the assertion above in Mininet proper triggering and crashing mobility.
I decreased the chance of this occurring (but not removed it) by replacing some instances of cmd() with pexec() in our libraries; this would likely work on your end as well, though I understand there's behavior differences that would potentially need accounting for.

[Ramon Fontes]

Cannot you use makeTerm()? This will avoid that error.

Sent from my android

To view this discussion on the web visit https://groups.google.com/d/msgid/mininet-wifi-discuss/147b41c5-78ef-4c02-8bcf-655e736bb151n%40googlegroups.com.

[alexande...@gmail.com]

We're running our code headless, there are logistical and financial issues involved in our setup that make it difficult or impossible otherwise.

[alexande...@gmail.com]

We're able to stop this issue on our end by using Popen() rather than cmd(), potentially you could resolve it on yours with pexec() or Popen() as well? I am not in a position to write a patch for this at the moment as my team is under time constraints on multiple levels but I will make a Github issue for this.