Problem receiving UDP broadcast packets.
Grant Edwards
Linux.
Here's the receiving code:
------------------------------receive.py-------------------------------
#!/usr/bin/python
import socket
host = ''
port = 5010
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host, port))
while 1:
try:
message = s.recv(8192)
print "Got data: %s" % repr(message)
except KeyboardInterrupt:
break
----------------------------------------------------------------------
Here's the sending code:
--------------------------------send.py-------------------------------
#!/usr/bin/python
import sys,socket,time
host = sys.argv[1]
port = 5010
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host,port))
s.sendto(str(time.time()), ('255.255.255.255', port))
----------------------------------------------------------------------
On the receiving machine, I've used tcpdump to verify that broadcast
packets are being seen and have a destination IP of 255.255.255.255 and
destination MAC of ff:ff:ff:ff:ff:ff
03:05:09.187327 IP 10.0.0.1.5010 > 255.255.255.255.5010: UDP, length 13
0x0000: ffff ffff ffff 0018 e708 2033 0800 4500
0x0010: 0029 0000 4000 4011 30c4 0a00 0001 ffff
0x0020: ffff 1392 1392 0015 6e6e 3133 3033 3235
0x0030: 3131 3830 2e34 3500 0000
03:05:09.407508 IP 10.0.0.1.5010 > 255.255.255.255.5010: UDP, length 13
0x0000: ffff ffff ffff 0018 e708 2033 0800 4500
0x0010: 0029 0000 4000 4011 30c4 0a00 0001 ffff
0x0020: ffff 1392 1392 0015 6c6c 3133 3033 3235
0x0030: 3131 3830 2e36 3700 0000
03:05:09.615962 IP 10.0.0.1.5010 > 255.255.255.255.5010: UDP, length 13
0x0000: ffff ffff ffff 0018 e708 2033 0800 4500
0x0010: 0029 0000 4000 4011 30c4 0a00 0001 ffff
0x0020: ffff 1392 1392 0015 6b6a 3133 3033 3235
0x0030: 3131 3830 2e38 3800 0000
But, the receiving Python program never sees any packets unless the
_source_ IP address in the packets is on the same subnet as the
receiving machine. In this test case, the receiving machine has an IP
address of 172.16.12.34/16. If I change the receiving machine's IP
address to 10.0.0.123, then the receiving program sees the packets.
Even though the destination address is 255.255.255.255, the receiving
machine appears to discard the packets based on the _source_ IP. Can
anybody provide example Python code for Linux that receives UDP
broadcast packets regardless of their source IP address?
This probably is more of a Linux networking question than a Python
question, but I'm hoping somebody has solved this problem in Python.
--
Grant Edwards grant.b.edwards Yow! I want my nose in
at lights!
gmail.com
Irmen de Jong
> I'm have problems figuring out how to receive UDP broadcast packets on
> Linux.
>
> Here's the sending code:
>
> --------------------------------send.py-------------------------------
> #!/usr/bin/python
> import sys,socket,time
>
> host = sys.argv[1]
> port = 5010
>
> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
> s.bind((host,port))
I don't think you should use s.bind() at all in the sending code.
Could that be at least part of the problem?
-Irmen
Grant Edwards
If I don't call bind(), then the broadcast packets go out the wrong
interface on the sending machine.
--
Grant Edwards grant.b.edwards Yow! Vote for ME -- I'm
at well-tapered, half-cocked,
gmail.com ill-conceived and
TAX-DEFERRED!
Dan Stromberg
> I'm have problems figuring out how to receive UDP broadcast packets on
> Linux.
>
> Here's the receiving code:
>
> ------------------------------receive.py-------------------------------
> But, the receiving Python program never sees any packets unless the
> _source_ IP address in the packets is on the same subnet as the
> receiving machine.
This is just how broadcasts work.
Normally, you take your subnet mask and "bitwise and" it with the IP
addresses of the sending and receiving machines. If the results match
for two such pairs on two different machines, then the broadcast
should be visible, given appropriate code.
However, some routers have the ability to pass packets from one subnet
to another. I believe this is called a "helper", at least in the
Cisco world, and must be configured specially.
Irmen de Jong
>
> If I don't call bind(), then the broadcast packets go out the wrong
> interface on the sending machine.
>
Fair enough.
Next issue then: as far as I know, broadcast packets are by default not routed across
subnets by gateways. Which is a good thing.
That would explain why your receiver doesn't see the packets unless its interface IP
address is in the same subnet as the sender's.
However it doesn't explain (for me) why the tcpdump program running on that same
receiver machine still happily spits out received packets. Unless the routing between
the subnets is somehow done on the receiving machine itself? My knowledge of networks
and TCP/IP ends here I'm afraid.
Cheers
Irmen.
Dan Stromberg
I'm guessing there are two different subnets on the same physical
cable - which is a little unusual, but not impossible.
Grant Edwards
>>
>> If I don't call bind(), then the broadcast packets go out the wrong
>> interface on the sending machine.
>
> Fair enough.
>
> Next issue then: as far as I know, broadcast packets are by default
> not routed across subnets by gateways. Which is a good thing.
>
> That would explain why your receiver doesn't see the packets unless
> its interface IP address is in the same subnet as the sender's.
>
> However it doesn't explain (for me) why the tcpdump program running
> on that same receiver machine still happily spits out received
> packets.
The two machines are on the same Ethernet segment (they're connected
via a dumb Ethernet switch).
Tcpdump shows the packets because the packets are being received by
the receiving machine's Ethernet interface. They have a destination
MAC of ff:ff:ff:ff:ff:ff, so everybody on the logical Ethernet segment
receives them.
I guess the problem is that I expected to receive a packet on an
interface anytime a packet was received with a destination IP address
that matched that of the the interface. Apprently there's some
filtering in the network stack based on the _source_ address as well
(that seems very counter-intuitive to me).
--
Grant Edwards grant.b.edwards Yow! I'm not an Iranian!!
at I voted for Dianne
gmail.com Feinstein!!
Grant Edwards
> On Tue, Apr 19, 2011 at 3:21 PM, Grant Edwards <inv...@invalid.invalid> wrote:
>> I'm have problems figuring out how to receive UDP broadcast packets on
>> Linux.
>>
>> Here's the receiving code:
>>
>> ------------------------------receive.py-------------------------------
>
>> But, the receiving Python program never sees any packets unless the
>> _source_ IP address in the packets is on the same subnet as the
>> receiving machine.
>
> This is just how broadcasts work.
>
> Normally, you take your subnet mask and "bitwise and" it with the IP
> addresses of the sending and receiving machines. If the results match
> for two such pairs on two different machines, then the broadcast
> should be visible, given appropriate code.
That certainly looks like what's hapenning, but it seems very
counter-productive to me.
If I send a packet to ff:ff:ff:ff:ff:ff--255.255.255.255, it's because
I want everybody on the Ethernet segment to receive it. If I wanted
only people on a particular subnet (e.g. 10.0.0.0/8) to receive it, I
would have sent it to the subnet broadcast address (e.g. 10.255.255.255).
> However, some routers have the ability to pass packets from one subnet
> to another. I believe this is called a "helper", at least in the
> Cisco world, and must be configured specially.
There are no routers or firewalls involved -- just a dumb Ethernet
switch.
It seems I'm going to have to use raw sockets to do what I need to do.
That's exactly what I was trying to avoid by using UDP: I'm replacing
a proprietary (non IP) MAC-level protocol that was implemented using
raw sockets.
--
Grant Edwards grant.b.edwards Yow! I'm young ... I'm
at HEALTHY ... I can HIKE
gmail.com THRU CAPT GROGAN'S LUMBAR
REGIONS!
Grant Edwards
>> On 20-4-2011 1:21, Grant Edwards wrote:
>>>
>>> If I don't call bind(), then the broadcast packets go out the wrong
>>> interface on the sending machine.
>>>
>>
>> Fair enough.
>>
>> Next issue then: as far as I know, broadcast packets are by default not routed across
>> subnets by gateways. Which is a good thing.
>>
>> That would explain why your receiver doesn't see the packets unless its interface IP
>> address is in the same subnet as the sender's.
>>
>> However it doesn't explain (for me) why the tcpdump program running on that same
>> receiver machine still happily spits out received packets. Unless the routing between
>> the subnets is somehow done on the receiving machine itself? My knowledge of networks
>> and TCP/IP ends here I'm afraid.
>
> cable
Yes -- though technically they're on the same Ethernet segment rather
than physical cable, since there's an intervening Ethernet switch.
> - which is a little unusual, but not impossible.
OK, here's some background...
I'm trying to implement a device discovery/configuration protocol that
uses UDP broadcast packets to discover specific types of devices on
the local Ethernet segment. The management program broadcasts a
discovery command to a particular UDP port. All devices who get that
packet are expected to answer regardless of thier current IP address.
The management program can then send another broadcast packet to
configure the IP address of a device. After that, the management
program switches over to normal unicast TCP and UDP protocols (HTTP,
TFTP, etc.) to set up the device.
I had ignorantly assumed that an UDP broadcast sent to IP address
255.255.255.255 would be received by everybody who could hear it.
Apparently I'm going to have to use RAW packets and implement UDP
myself. :/
--
Grant Edwards grant.b.edwards Yow! BELA LUGOSI is my
at co-pilot ...
gmail.com
Chris Angelico
> If I send a packet to ff:ff:ff:ff:ff:ff--255.255.255.255, it's because
> I want everybody on the Ethernet segment to receive it. If I wanted
> only people on a particular subnet (e.g. 10.0.0.0/8) to receive it, I
> would have sent it to the subnet broadcast address (e.g. 10.255.255.255).
> It seems I'm going to have to use raw sockets to do what I need to do.
> That's exactly what I was trying to avoid by using UDP: I'm replacing
> a proprietary (non IP) MAC-level protocol that was implemented using
> raw sockets.
I have to ask:
1) Why do you have two subnets on the same switch? Isn't that going to
be an eternal maintenance headache? Not that it _can't_ be done -
obviously it can - but it's likely to confuse the humans involved.
2) Can you replace that protocol at a higher level? Presumably there's
a full protocol stack with application data getting wrapped up inside
(ultimately) ethernet frames; can you cut it somewhere else and make,
say, a TCP/IP connection to the remote system?
Chris Angelico
Chris Angelico
> The management program can then send another broadcast packet to
> configure the IP address of a device. After that, the management
> program switches over to normal unicast TCP and UDP protocols (HTTP,
> TFTP, etc.) to set up the device.
>
Wonder if it would be possible to (ab)use DHCP for this. If every
device registers itself with a central DHCP server, you could query
that to find out what's around, and configuring of IP addresses would
then be out of your hands.
Or can you simply use a stupid netmask like /1 that picks up all the
IP ranges? That way, the source-IP check wouldn't fail.
Chris Angelico
Grant Edwards
> On Wed, Apr 20, 2011 at 10:09 AM, Grant Edwards <inv...@invalid.invalid> wrote:
>> The management program can then send another broadcast packet to
>> configure the IP address of a device. After that, the management
>> program switches over to normal unicast TCP and UDP protocols (HTTP,
>> TFTP, etc.) to set up the device.
>>
>
> Wonder if it would be possible to (ab)use DHCP for this. If every
> device registers itself with a central DHCP server, you could query
> that to find out what's around, and configuring of IP addresses would
> then be out of your hands.
I can't require the presense of a DHCP server, and many installations
won't have one.
> Or can you simply use a stupid netmask like /1 that picks up all the
> IP ranges? That way, the source-IP check wouldn't fail.
That would require that the device somehow knows that it's not
configured correctly and should change the netmask to /1. The device
doesn't have any way to know that, and it must respond to the
discovery commands both before and after it's properly configured.
I've reread the protocol documentation and noticed that the device has
to respond not only to broadcasts to 255.255.255.255 but also to
subnet broadcasts send to subnets it's not on. That pretty much
clinches the requirement to use a raw socket. :/
--
Grant
Chris Angelico
>> Or can you simply use a stupid netmask like /1 that picks up all the
>> IP ranges? That way, the source-IP check wouldn't fail.
>
> That would require that the device somehow knows that it's not
> configured correctly and should change the netmask to /1. The device
> doesn't have any way to know that, and it must respond to the
> discovery commands both before and after it's properly configured.
Was hoping that you could make such a change _only_ on the computer
that's receiving the data - that way it's only one change, the devices
don't need any tweaking. But if it can't be, it can't be.
> I've reread the protocol documentation and noticed that the device has
> to respond not only to broadcasts to 255.255.255.255 but also to
> subnet broadcasts send to subnets it's not on. That pretty much
> clinches the requirement to use a raw socket. :/
Sounds to me like someone majorly abused IP to do weird things. Looks
like you're stuck doing the same weirdness, in whatever way you can
manage :| Sorry.
Chris Angelico
Grant Edwards
>>> Or can you simply use a stupid netmask like /1 that picks up all the
>>> IP ranges? That way, the source-IP check wouldn't fail.
>>
>> That would require that the device somehow knows that it's not
>> doesn't have any way to know that, and it must respond to the
>> discovery commands both before and after it's properly configured.
>
> Was hoping that you could make such a change _only_ on the computer
> that's receiving the data - that way it's only one change, the devices
> don't need any tweaking. But if it can't be, it can't be.
There can by any number of devices that have to receive the broadcast
packet, and any of them can be on different subnets (or have no IP
address at all).
>> I've reread the protocol documentation and noticed that the device
>> has to respond not only to broadcasts to 255.255.255.255 but also to
>> subnet broadcasts send to subnets it's not on.
It turns out that some OSes (BSD and some flavors of Windows) can't
broadcast to 255.255.255.255, only to the subnet broadcast address.
Hence the requirement for the devices to be able to receive a subnet
broadcast regardless of their IP address.
>> That pretty much clinches the requirement to use a raw socket. :/
>
> Sounds to me like someone majorly abused IP to do weird things.
Indeed. The other option is to do something that's not based on IP
but done completely at the Ethernet layer. Implementing management
programs that can do that can be very nasty on Windows, which
unfortunately matters to most customers.
So you bite the bullet on the device end and implement all sorts of
weirdness in order to allow the management program to use standard
UDP.
> Looks like you're stuck doing the same weirdness, in whatever way you
> can manage
Yup. It doesn't even appear that it can be done with a raw UDP
socket. According to all of the documentation I can find, such a
socket is supposed to receive copies of all UDP packets seen by the
kernel, but it doesn't. Even if I use a raw UDP socket, the kernel is
still dropping broadcast packets whose source address don't match any
interface's subnet.
AFAICT, I'm going to have to go completely raw and process in
user-space every single IP packet recieved. That's going to be
brutal on the CPU...
--
Grant
Dan Stromberg
>> Or can you simply use a stupid netmask like /1 that picks up all the
>> IP ranges? That way, the source-IP check wouldn't fail.
>
> That would require that the device somehow knows that it's not
> configured correctly and should change the netmask to /1. The device
> doesn't have any way to know that, and it must respond to the
> discovery commands both before and after it's properly configured.
- Actually, you Might be able to configure your device to have a
netmask of 0.0.0.0, IP address of 255.255.255.255 and broadcast of
255.255.255.255.
- I've seen something a bit similar used for detecting IP address
conflicts automatically.
- A network guru I used to work with told me that you could configure
a machine with a broadcast of 255.255.255.255 more simply than messing
around with the netmask, while still achieving the same result for
general purpose networking.
> I've reread the protocol documentation and noticed that the device has
> to respond not only to broadcasts to 255.255.255.255 but also to
> subnet broadcasts send to subnets it's not on. That pretty much
> clinches the requirement to use a raw socket. :/
With a netmask of 0.0.0.0, I suspect you will receive all broadcasts
on the wire, given appropriate listening code.
You could probably also modify a copy of tshark or tcpdump to flush
after every line of output (or run an unmodified one on a pty to avoid
maintaining your own copy of the binary), and parse that output in
Python. That should make the Python code pretty simple.
There's an old program called "pty" that makes it easy to run
something on a pty, to get unbuffered output - it's in
comp.sources.unix volumes 22, 23 and 25; it's written in C. You'd
just open a subprocess with no buffering on the python side, that runs
tcpdump or tshark under pty. Beware though - the pty program predates
GNU autoconf, so might be a little involved to compile.
I agree though that you're kind of pushing IP in a direction it wasn't
intended to go.
Dan Stromberg
> I agree though that you're kind of pushing IP in a direction it wasn't
> intended to go.
It just occurred to me: You might get some additional mileage out of
popping the network adapter into promiscuous mode. In fact, it Might
be necessary irrespective of the rest of your approach.
Roy Smith
Grant Edwards <inv...@invalid.invalid> wrote:
> I'm trying to implement a device discovery/configuration protocol that
> uses UDP broadcast packets to discover specific types of devices on
> the local Ethernet segment. The management program broadcasts a
> discovery command to a particular UDP port. All devices who get that
> packet are expected to answer regardless of thier current IP address.
Have you considered what will happen if you have, say, 1000 such
devices, and they all respond at the same time?
Heiko Wundram
> I guess the problem is that I expected to receive a packet on an
> interface anytime a packet was received with a destination IP address
> that matched that of the the interface. Apprently there's some
> filtering in the network stack based on the _source_ address as well
> (that seems very counter-intuitive to me).
Just to pitch in here (because nobody's mentioned it yet AFAICT): yes,
there's a filtering done (at least under Linux, and I'd guess something
similar on xBSD too) to packets based on the source address coming in on
an interface, and it's called the reverse path filter and is on by
default (the tunable on Linux is /proc/sys/net/ipv4/conf/*/rp_filter).
The idea behind the reverse path filter is that your machine won't
accept packets coming in over an interface when a return packet (i.e.,
the presumed response) won't be routed over the same interface, and from
what I gather, this is what makes the TCP/IP stack drop the packets
because your machine will not route packets to 192.168.x.x over the same
interface it sees the packet coming in. This is a _security_ feature,
because it makes address spoofing harder.
If you need to see the packets regardless, either use a promiscuous mode
sniffer (i.e., tcpdump, but that's relatively easy to mirror in Python
using SOCK_RAW, capturing packets at the ethernet level), or add a route
on your system for the 192.168.x.x network on the same interface.
HTH!
--
--- Heiko.
Sherm Pendley
> I'm trying to implement a device discovery/configuration protocol that
> uses UDP broadcast packets to discover specific types of devices on
> the local Ethernet segment. The management program broadcasts a
> discovery command to a particular UDP port. All devices who get that
> packet are expected to answer regardless of thier current IP address.
Have you looked at the source for Apple's Bonjour?
<http://developer.apple.com/opensource/>
<http://opensource.apple.com/source/mDNSResponder/mDNSResponder-258.18/>
Might be interesting to see how it does announcement/discovery. Or maybe
just use it directly, if this happens to be a case of "gee, I didn't
know that wheel had already been invented." :-)
sherm--
--
Sherm Pendley
<http://camelbones.sourceforge.net>
Cocoa Developer
Thomas Heller
> I'm have problems figuring out how to receive UDP broadcast packets on
> Linux.
>
> On the receiving machine, I've used tcpdump to verify that broadcast
> packets are being seen and have a destination IP of 255.255.255.255 and
> destination MAC of ff:ff:ff:ff:ff:ff
> But, the receiving Python program never sees any packets unless the
> _source_ IP address in the packets is on the same subnet as the
> receiving machine. In this test case, the receiving machine has an IP
> address of 172.16.12.34/16. If I change the receiving machine's IP
> address to 10.0.0.123, then the receiving program sees the packets.
>
> Even though the destination address is 255.255.255.255, the receiving
> machine appears to discard the packets based on the _source_ IP. Can
> anybody provide example Python code for Linux that receives UDP
> broadcast packets regardless of their source IP address?
>
> This probably is more of a Linux networking question than a Python
> question, but I'm hoping somebody has solved this problem in Python.
>
You must set the network interface to promiscous mode on the receiving side:
os.system("ifconfig eth0 promisc")
Thomas
Adam Tauno Williams
> Grant Edwards <inv...@invalid.invalid> writes:
> > I'm trying to implement a device discovery/configuration protocol that
> > uses UDP broadcast packets to discover specific types of devices on
> > the local Ethernet segment. The management program broadcasts a
> > discovery command to a particular UDP port. All devices who get that
> > packet are expected to answer regardless of thier current IP address.
> Have you looked at the source for Apple's Bonjour?
On LINUX this is called "avahi", which has Python bindings. Avahi
auto-configuration / discovery works very well.
<http://avahi.org/>
<http://freshmeat.net/projects/avahi>
<http://pypi.python.org/pypi/pybonjour/1.1.1>
<http://avahi.org/wiki/PythonBrowseExample>
See also:
<http://pypi.python.org/pypi/pybonjour/1.1.1>
at least you may be able to lift code from them (License is non-viral
MIT)
> Might be interesting to see how it does announcement/discovery. Or maybe
> just use it directly, if this happens to be a case of "gee, I didn't
> know that wheel had already been invented." :-)
--
Adam Tauno Williams <awil...@whitemice.org> LPIC-1, Novell CLA
<http://www.whitemiceconsulting.com>
OpenGroupware, Cyrus IMAPd, Postfix, OpenLDAP, Samba
Grant Edwards
The network adapter is already receiving all the packets I want to
receive, so putting it into promiscuous mode would only increase the
number of unwanted packets.
--
Grant Edwards grant.b.edwards Yow! Send your questions to
at ``ASK ZIPPY'', Box 40474,
gmail.com San Francisco, CA 94140,
USA
Grant Edwards
Yes. Firstly, there will very rarely be more than a handful of such
devices. Secondly, I plan on inserting a small, psuedo-random delay
before replying. Thirdly, the management program typically repeats
the discovery process a few times in case any packets get dropped.
--
Grant Edwards grant.b.edwards Yow! Maybe we could paint
at GOLDIE HAWN a rich PRUSSIAN
gmail.com BLUE --
Grant Edwards
> Am 20.04.2011 01:54, schrieb Grant Edwards:
>> I guess the problem is that I expected to receive a packet on an
>> interface anytime a packet was received with a destination IP address
>> that matched that of the the interface. Apprently there's some
>> filtering in the network stack based on the _source_ address as well
>> (that seems very counter-intuitive to me).
>
> Just to pitch in here (because nobody's mentioned it yet AFAICT): yes,
> there's a filtering done (at least under Linux, and I'd guess something
> similar on xBSD too) to packets based on the source address coming in on
> an interface, and it's called the reverse path filter and is on by
> default (the tunable on Linux is /proc/sys/net/ipv4/conf/*/rp_filter).
Brilliant! While I had determined that such filtering took place, I'd
been unable to figure out if it was configurable.
> The idea behind the reverse path filter is that your machine won't
> accept packets coming in over an interface when a return packet (i.e.,
> the presumed response) won't be routed over the same interface, and from
> what I gather, this is what makes the TCP/IP stack drop the packets
> because your machine will not route packets to 192.168.x.x over the same
> interface it sees the packet coming in. This is a _security_ feature,
> because it makes address spoofing harder.
And it's an eminently sensible feature.
> If you need to see the packets regardless, either use a promiscuous mode
> sniffer (i.e., tcpdump, but that's relatively easy to mirror in Python
> using SOCK_RAW, capturing packets at the ethernet level), or add a route
> on your system for the 192.168.x.x network on the same interface.
I've thought about the SOCK_RAW option, but the CPU load of looking
all received Ethernet packets in user-space would be a big down-side.
Adding the route isn't an option since 1) the device doesn't know what
route to add, and 2) adding such a route could break the normal
networking operation.
--
Grant Edwards grant.b.edwards Yow! My haircut is totally
at traditional!
gmail.com
Grant Edwards
>
>>> Or can you simply use a stupid netmask like /1 that picks up all the
>>> IP ranges? That way, the source-IP check wouldn't fail.
>>
>> That would require that the device somehow knows that it's not
>> doesn't have any way to know that, and it must respond to the
>> discovery commands both before and after it's properly configured.
>
> - Actually, you Might be able to configure your device to have a
> netmask of 0.0.0.0, IP address of 255.255.255.255 and broadcast of
> 255.255.255.255.
> - I've seen something a bit similar used for detecting IP address
> conflicts automatically.
> - A network guru I used to work with told me that you could configure
> a machine with a broadcast of 255.255.255.255 more simply than messing
> around with the netmask, while still achieving the same result for
> general purpose networking.
I'll look into that.
>> I've reread the protocol documentation and noticed that the device has
>> to respond not only to broadcasts to 255.255.255.255 but also to
>> subnet broadcasts send to subnets it's not on. ?That pretty much
>> clinches the requirement to use a raw socket. :/
>
> With a netmask of 0.0.0.0, I suspect you will receive all broadcasts
> on the wire, given appropriate listening code.
That might be an option as well, as long as it doesn't disrupt normal
operation of the interface.
--
Grant Edwards grant.b.edwards Yow! Remember, in 2039,
at MOUSSE & PASTA will
gmail.com be available ONLY by
prescription!!
Grant Edwards
Why?
The network interface is already receiving the packets I want, since
they're beign sent with a destination MAC of ff:ff:ff:ff:ff:ff.
--
Grant Edwards grant.b.edwards Yow! Spreading peanut
at butter reminds me of
gmail.com opera!! I wonder why?
Heiko Wundram
>> If you need to see the packets regardless, either use a promiscuous mode
>> sniffer (i.e., tcpdump, but that's relatively easy to mirror in Python
>> using SOCK_RAW, capturing packets at the ethernet level), or add a route
>> on your system for the 192.168.x.x network on the same interface.
>
> I've thought about the SOCK_RAW option, but the CPU load of looking
> all received Ethernet packets in user-space would be a big down-side.
Not necessarily: instead of using UDP datagrams to send the data, use
ethernet datagrams (without any IP/UDP header) with your own
ethernet-type (there is a range of "local" types that you can use for
your own local use-case), and then simply create a RAW socket that only
listens on packets that have the specified ethernet types. We use
something similar at work for a high-availability application.
The server-side looks something like:
"""
PKT_TYPE = 0x1234 # My very own ethertype.
sock = socket(AF_PACKET,SOCK_DGRAM,htons(PKT_TYPE))
sock.bind(("ethxyz",PKT_TYPE))
while True:
data, (_, _, _, _, addr) = sock.recvfrom(1500)
print "I got:", repr(data), "from etheraddr:", addr
"""
The client-side looks similar.
Because you're using UDP broacast, you have unreliable transport anyway,
and if the client-side supports sending ethernet datagrams (with a
broadcast address), I'd rather advise to use that for your use case.
This makes you independent of IP configuration (and as I can see, you're
actually not interested in the "routing" that IP gives you, but rather
interested in contacting all nodes on a local ethernet; why not use
ethernet directly?).
--
--- Heiko.
Grant Edwards
>>> If you need to see the packets regardless, either use a promiscuous mode
>>> sniffer (i.e., tcpdump, but that's relatively easy to mirror in Python
>>> using SOCK_RAW, capturing packets at the ethernet level), or add a route
>>> on your system for the 192.168.x.x network on the same interface.
>>
>> I've thought about the SOCK_RAW option, but the CPU load of looking
>> all received Ethernet packets in user-space would be a big down-side.
>
> Not necessarily: instead of using UDP datagrams to send the data,
There's the rub. The requirement is to use UDP.
> use ethernet datagrams (without any IP/UDP header) with your own
> ethernet-type (there is a range of "local" types that you can use for
> your own local use-case), and then simply create a RAW socket that
> only listens on packets that have the specified ethernet types. We
> use something similar at work for a high-availability application.
Same here. What I'm working on is a replacement for just such a
protocol. The problem is that this whole scheme has to work easily
with the management program running on Windows. While doing a custom
Ethernet protocol on Linux is dead-simple (it does require either
UID==0 or CAP_NET_RAW), doing it on Windows is painful in the extreme.
Thus the requirement to use something that works using "normal UDP" on
the manager end of things (even if it requires jumping through some
hoops on the device end).
[Insert standard whinge about how once again, Microsoft royally
botches something and we non-Windows people have to suffer.]
--
Grant Edwards grant.b.edwards Yow! ... I want to perform
at cranial activities with
gmail.com Tuesday Weld!!
Grant Edwards
> On 2011-04-20, Heiko Wundram <mode...@modelnine.org> wrote:
>>> I've thought about the SOCK_RAW option, but the CPU load of looking
>>> all received Ethernet packets in user-space would be a big down-side.
>>
>> Not necessarily: instead of using UDP datagrams to send the data,
>> use ethernet datagrams (without any IP/UDP header) with your own
>> ethernet-type (there is a range of "local" types that you can use for
>> your own local use-case), and then simply create a RAW socket that
>> only listens on packets that have the specified ethernet types. We
>> use something similar at work for a high-availability application.
>
> Same here. What I'm working on is a replacement for just such a
> protocol.
Now that I think of it, I added raw socket support to Python just so I
could implement that protocol in Python!
[A rather vain attempt to inject some Python content back into the
thread.]
--
Grant Edwards grant.b.edwards Yow! Awright, which one of
at you hid my PENIS ENVY?
gmail.com
Grant Edwards
> On Wed, 2011-04-20 at 06:07 -0400, Sherm Pendley wrote:
>> Grant Edwards <inv...@invalid.invalid> writes:
>> > I'm trying to implement a device discovery/configuration protocol that
>> > uses UDP broadcast packets to discover specific types of devices on
>> > the local Ethernet segment. The management program broadcasts a
>> > discovery command to a particular UDP port. All devices who get that
>> > packet are expected to answer regardless of thier current IP address.
>> Have you looked at the source for Apple's Bonjour?
>
> On LINUX this is called "avahi", which has Python bindings. Avahi
> auto-configuration / discovery works very well.
>
><http://avahi.org/>
><http://freshmeat.net/projects/avahi>
><http://pypi.python.org/pypi/pybonjour/1.1.1>
><http://avahi.org/wiki/PythonBrowseExample>
>
> See also:
><http://pypi.python.org/pypi/pybonjour/1.1.1>
> at least you may be able to lift code from them (License is non-viral
> MIT)
I've looked at those protocols and they aren't particularly suited to
what we want to do. One of our requirements is to be able to discover
a device with a static IP address (or no IP address?) that isn't on
the same subnet -- and I don't see how those protocols can do that.
Since the management program already has to implement the UDP based
protocol I'm working on, it would be a big win if I didn't have to
make the management program add support for a second protocol.
--
Grant Edwards grant.b.edwards Yow! Are we THERE yet?
at My MIND is a SUBMARINE!!
gmail.com
Grant Edwards
> - Actually, you Might be able to configure your device to have a
> netmask of 0.0.0.0, IP address of 255.255.255.255 and broadcast of
> 255.255.255.255.
255.255.255.255 isn't allowed as an IP address.
I tried a netmask of 0.0.0.0, and it didn't make any differnce.
--
Grant Edwards grant.b.edwards Yow! My EARS are GONE!!
at
gmail.com
Grant Edwards
>> Am 20.04.2011 01:54, schrieb Grant Edwards:
>>> I guess the problem is that I expected to receive a packet on an
>>> interface anytime a packet was received with a destination IP address
>>> that matched that of the the interface. Apprently there's some
>>> filtering in the network stack based on the _source_ address as well
>>> (that seems very counter-intuitive to me).
>>
>> Just to pitch in here (because nobody's mentioned it yet AFAICT): yes,
>> there's a filtering done (at least under Linux, and I'd guess something
>> similar on xBSD too) to packets based on the source address coming in on
>> an interface, and it's called the reverse path filter and is on by
>> default (the tunable on Linux is /proc/sys/net/ipv4/conf/*/rp_filter).
>
> Brilliant! While I had determined that such filtering took place, I'd
> been unable to figure out if it was configurable.
Bingo!
Turning off reverse-path filtering solved my original problem (not
receiving broadcasts sent to 255.255.255.255 because the source IP
wasn't reachable via the receiving interface).
It doesn't help with the "new" requirement of receiving subnet
broadcasts that don't match the interface on which they're received,
since it's destination IP filtering that's dropping those.
Apparently that requirement is due to the fact that on some OSes (some
flavors of BSD and Windows) applications can't broadcast to
255.255.255.255, they can only do subnet broadcasts.
--
Grant Edwards grant.b.edwards Yow! I wish I was a
at sex-starved manicurist
gmail.com found dead in the Bronx!!
Grant Edwards
> I'm have problems figuring out how to receive UDP broadcast packets on
> Linux.
Thanks to everybody for their help with what turned out to have little
or nothing to do with Python itself.
The main problem was reverse-path filtering (which is enabled by
default on Linux). Disabling it allowed me to receive packets sent to
255.255.255.255 regardless of the sender's IP address.
Another minor issue that cropped up was that I ended up having to use
two sockets: an Rx socket bound to ('',port) and a Tx socket bound to
(myip,port).
Even though the protocol spec said that subnet broadcasts were used,
tracing a few transactions between existing implementations showed
that only global broadcasts were used. More testing showed that
existing implementations won't respond to subnet broadcasts unless
they're already configured to be on the proper subnet. So, receiving
subnet broadcasts that don't match an interface turned out to be a
wild goose chase.
It's all working now...
--
Grant Edwards grant.b.edwards Yow! Everywhere I look I
at see NEGATIVITY and ASPHALT
gmail.com ...
Dan Stromberg
> On 2011-04-20, Dan Stromberg <drsa...@gmail.com> wrote:
>> On Tue, Apr 19, 2011 at 8:12 PM, Dan Stromberg <drsa...@gmail.com> wrote:
>>> I agree though that you're kind of pushing IP in a direction it wasn't
>>> intended to go.
>>
>> It just occurred to me: You might get some additional mileage out of
>> popping the network adapter into promiscuous mode. In fact, it Might
>> be necessary irrespective of the rest of your approach.
>
> The network adapter is already receiving all the packets I want to
> receive, so putting it into promiscuous mode would only increase the
> number of unwanted packets.
I think tcpdump and tshark (was tethereal) will put the interface into
promiscuous mode so it can see more traffic; on OSF/1 (Tru64), we had
to do this manually for said programs to see all that was possible
(barring the presence of a switch not repeating packets the way
routers and hubs would).
Heiko Wundram
> I think tcpdump and tshark (was tethereal) will put the interface into
> promiscuous mode so it can see more traffic; on OSF/1 (Tru64), we had
> to do this manually for said programs to see all that was possible
> (barring the presence of a switch not repeating packets the way
> routers and hubs would).
It actually depends on the network adapter/card that's in use: many
modern cards (especially those in the lower price segment, i.e. Realtek)
don't (properly) implement "MAC"-filtering at the hardware level, and in
this case, there's no difference for the operating system between
promiscuous mode and non-promiscuous mode (because the card will forward
all packets that it sees coming in over the ethernet bus to the
operating system, which will then discard those at the ethernet level it
doesn't deem necessary to process at a higher level, for example because
the destination MAC is unicast, but not the cards own, so the
destination wasn't the system itself).
For pricier cards/chips, this filtering (which also includes restricting
the multicast-destinations that are forwarded to the operating system,
think IPv6-multicast which uses quite a range of multicast MAC addresses
for its neighbour discovery) is implemented at the hardware level, and
the ethernet adapter throws away uninteresting packets and doesn't
signal the operating system (think of the cost of interrupts you save;
on high throughput links, this makes perfect sense). Putting the card
into promiscuous mode basically disables this filtering, so that the
card will again forward all packets to the operating system.
This is why tcpdump for example puts the network adapter into
promiscuous mode, but normally (see above, depending on the network
adapter), that's not required because the operating system "sees all"
ethernet packets anyway.
--
--- Heiko.
Grant Edwards
> On Wed, Apr 20, 2011 at 7:21 AM, Grant Edwards <inv...@invalid.invalid> wrote:
>> On 2011-04-20, Dan Stromberg <drsa...@gmail.com> wrote:
>>> On Tue, Apr 19, 2011 at 8:12 PM, Dan Stromberg <drsa...@gmail.com> wrote:
>>>> I agree though that you're kind of pushing IP in a direction it wasn't
>>>> intended to go.
>>>
>>> It just occurred to me: You might get some additional mileage out of
>>> be necessary irrespective of the rest of your approach.
>>
>> The network adapter is already receiving all the packets I want to
>> receive, so putting it into promiscuous mode would only increase the
>> number of unwanted packets.
>
> I think tcpdump and tshark (was tethereal) will put the interface into
> promiscuous mode so it can see more traffic;
It can (and by default does). I was using "-p" so it didn't.
> on OSF/1 (Tru64), we had
> to do this manually for said programs to see all that was possible
> (barring the presence of a switch not repeating packets the way
> routers and hubs would).
* The packets were being sent to MAC address ff:ff:ff:ff:ff:ff, so the
NIC does not have to be in promiscuous mode to receive them.
* tcpdump saw them even when it doesn't put the NIC in promiscuous
mode.
* The kernel was seeing the packets because it was logging them as
martians and discarding them (something I didn't notice until later).
* Turning off reverse-path filtering in the TCP stack allowed the
packets to be received as expected.
--
Grant Edwards grant.b.edwards Yow! I was making donuts
at and now I'm on a bus!
gmail.com