[Bug 193246] Bug in multicast bind(), uncovered by Jenkins

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

Craig Rodrigues <rod...@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |freebs...@FreeBSD.org,
| |freeb...@FreeBSD.org
Assignee|freebs...@FreeBSD.org |freeb...@FreeBSD.org

--
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
freebs...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-java
To unsubscribe, send any mail to "freebsd-java...@freebsd.org"

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

--- Comment #1 from Craig Rodrigues <rod...@FreeBSD.org> ---
Previously, I reported some error messages reported by Jenkins
on startup:

http://lists.freebsd.org/pipermail/freebsd-java/2014-February/010593.html

I looked at the Jenkins source code, and isolated the problem.
I wrote this simple testcase (see attached MulticastTest.java),
which I am also including inline:

================================================================================
/*
* To build this test,
* (1) Make sure that the OpenJDK is installed from ports:
*
* pkg install openjdk
*
* (2) Rename this file to: MulticastTest.java
* (3) Build it:
*
* javac MulticastTest.java
*
* (4) Run it:
*
* java MulticastTest.java
*
*/

import java.net.InetAddress;
import java.net.MulticastSocket;

class MulticastTest
{

public static void main(String[] args)
{
try {
int PORT = Integer.getInteger("hudson.udp",33848);

InetAddress MULTICAST = InetAddress.getByAddress(new byte[]{(byte)239,
(byte)77, (byte)124, (byte)213});
MulticastSocket mcs = new MulticastSocket(PORT);
mcs.joinGroup(MULTICAST);
} catch (Exception e) {
e.printStackTrace();
System.exit(-1);
}
}
}
================================================================================

If I run this testcase, I get the same error as what I reported earlier
with Jenkins:

java.net.SocketException: Invalid argument
at java.net.PlainDatagramSocketImpl.join(Native Method)
at
java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:178)
at java.net.MulticastSocket.joinGroup(MulticastSocket.java:319)
at MulticastTest.main(MulticastTest.java:31)


If I run:

ktrace java MulticastTest

I see that the error here:

13253 java CALL setsockopt(0x4,0x29,0x1b,0x7fffffbfd7dc,0x4)
13253 java RET setsockopt 0
13253 java CALL
setsockopt(0x4,SOL_SOCKET,SO_BROADCAST,0x7fffffbfd7d8,0x4)
13253 java RET setsockopt 0
13253 java CALL
getsockopt(0x4,SOL_SOCKET,SO_TYPE,0x7fffffbfd77c,0x7fffffbfd778)
13253 java RET getsockopt 0
13253 java CALL
setsockopt(0x4,SOL_SOCKET,SO_REUSEPORT,0x7fffffbfd7e0,0x4)
13253 java RET setsockopt 0
13253 java CALL
setsockopt(0x4,SOL_SOCKET,SO_REUSEADDR,0x7fffffbfd7e0,0x4)
13253 java RET setsockopt 0
13253 java CALL bind(0x4,0x7fffffbfd7a8,0x1c)
13253 java STRU struct sockaddr { AF_INET6, [::]:33848 }
13253 java RET bind 0
13253 java CALL setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4)
13253 java RET setsockopt 0
13253 java CALL getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864)
13253 java RET getsockopt 0
13253 java CALL setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14)
13253 java RET setsockopt -1 errno 22 Invalid argument


This looks like a bug in the FreeBSD networking code for multicast,
or a bug in the FreeBSD code in the OpenJDK.

This Java code works under Linux, Solaris, Windows, etc., so it
would be good to fix this problem on FreeBSD.

Can a networking person help me with this?

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

--- Comment #2 from Craig Rodrigues <rod...@FreeBSD.org> ---
I did some more analysis and found that this code:

13253 java CALL bind(0x4,0x7fffffbfd7a8,0x1c)
13253 java STRU struct sockaddr { AF_INET6, [::]:33848 }
13253 java RET bind 0
13253 java CALL setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4)
13253 java RET setsockopt 0
13253 java CALL getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864)
13253 java RET getsockopt 0
13253 java CALL setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14)
13253 java RET setsockopt -1 errno 22 Invalid argument

is happening inside the mcast_join_leave() function inside the JDK here:

http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/net/PlainDatagramSocketImpl.c

From looking at the FreeBSD header files,
IPPROTO_IPV6 = 41 (0x29)
IPV6_MULTICAST_IF =
IPV6_JOIN_GROUP = 12 (0xc)_

inside the mcast_join_leave() function what is basically happening is:

setsockopt(0x4, IPPROTO_IPV6, IPV6_MULTICAST_IF, ...)
getsockopt(0x4, IPPROTO_IPV6, IPV6_MULTICAST_IF, ...)
setsockopt(0x4, IPPROTO_IPV6, IPV6_JOIN_GROUP, ...)

the second setsockopt() is returning EINVAL

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

--- Comment #3 from Craig Rodrigues <rod...@FreeBSD.org> ---
I ran the same test program under truss with:

truss java MulticastTest

and found this:
socket(PF_INET6,SOCK_DGRAM,0) = 4 (0x4)
setsockopt(0x4,0x29,0x1b,0x7fffffbfd7dc,0x4,0x0) = 0 (0x0)
setsockopt(0x4,0xffff,0x20,0x7fffffbfd7d8,0x4,0x0) = 0 (0x0)
getsockopt(0x4,0xffff,0x1008,0x7fffffbfd77c,0x7fffffbfd778,0x83f5292d8) = 0
(0x0)
setsockopt(0x4,0xffff,0x200,0x7fffffbfd7e0,0x4,0x83f5292d8) = 0 (0x0)
setsockopt(0x4,0xffff,0x4,0x7fffffbfd7e0,0x4,0x83f5292d8) = 0 (0x0)
bind(4,{ AF_INET6 [108c:bd00:800:0:4b41:9700:100:0]:36096 },28) = 0 (0x0)
setsockopt(0x4,0x29,0x9,0x7fffffbfd7f4,0x4,0x83f54efd0) = 0 (0x0)
getsockopt(0x4,0x29,0x9,0x7fffffbfd8ac,0x7fffffbfd864,0x83f54c798) = 0 (0x0)
setsockopt(0x4,0x29,0xc,0x7fffffbfd8c0,0x14,0x83f54c798) ERR#22 'Invalid
argument'

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

Craig Rodrigues <rod...@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------

Summary|Bug in multicast bind(), |Bug in IPv6 multicast
|uncovered by Jenkins |join(), uncovered by
| |Jenkins

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

--- Comment #4 from Craig Rodrigues <rod...@FreeBSD.org> ---
I tracked this down some more.

Inside the JDK, there is this code in
http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/net/PlainDatagramSocketImpl.c

===============================================================================
/*
* IPv6 join. If it's an IPv4 multicast group then we use an IPv4-mapped
* address.
*/
#ifdef AF_INET6
{
struct ipv6_mreq mname6;
jbyteArray ipaddress;
jbyte caddr[16];
jint family;
jint address;
family = (*env)->GetIntField(env, iaObj, ia_familyID) == IPv4? AF_INET
: AF_INET6;
if (family == AF_INET) { /* will convert to IPv4-mapped address */
memset((char *) caddr, 0, 16);
address = (*env)->GetIntField(env, iaObj, ia_addressID);

caddr[10] = 0xff;
caddr[11] = 0xff;

caddr[12] = ((address >> 24) & 0xff);
caddr[13] = ((address >> 16) & 0xff);
caddr[14] = ((address >> 8) & 0xff);
caddr[15] = (address & 0xff);


===============================================================================




I can confirm that the address created by this code looks something like:

0 0 0 0 0 0 0 0 0 0 ff ff ef 4d 7c d5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0


In FreeBSD, in src/sys/netinet6/in6_mcast.c inside in6p_join_group(), there
is this:

if (!IN6_IS_ADDR_MULTICAST(&gsa->sin6.sin6_addr))
return (EINVAL);

Since IN6_IS_ADDR_MULTICAST() only checks if the first octet is 0xff, that is
what is returning the EINVAL. So the JDK is creating
an IPV4 multicast address mapped inside an IPV6 address. The FreeBSD
kernel code is rejecting this as a valid IPV6 multicast address.

I'm not sure if it is better to fix this in the kernel or the JDK.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

John Baldwin <j...@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |j...@FreeBSD.org

--- Comment #5 from John Baldwin <j...@FreeBSD.org> ---
Don't you really want an IPv4 socket here anyway? It seems rather convoluted
to create an IPv6 socket so you can listen for IPv4 multicast. My guess is
that the in6_mcast code doesn't handle IPv4 multicast groups, but bms@ might
know.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

--- Comment #6 from Craig Rodrigues <rod...@FreeBSD.org> ---
I'm looking at this from the perspective that
3rd party multicast code written in Java "just works" under Linux
and Solaris, but fails under FreeBSD.

Sure, using an IPv4 socket from the beginning would be the way to go,
but now that will require pushing patches upstream to all Java software
using multicast, in order to accomodate FreeBSD. It's possible, but not
practical.

It would be nice if the FreeBSD IPv6 stack could be modified
to deal with IPv4-mapped multicast addresses for IPv6, similar to
how Solaris does it. See comment from Andrey,
http://lists.freebsd.org/pipermail/freebsd-net/2014-September/039686.html

If that could be made to work, then no changes to upstream code would be
required.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

--- Comment #7 from Craig Rodrigues <rod...@FreeBSD.org> ---
In src/sys/netinet6, I see that there is usage of a
IN6_IS_ADDR_V4MAPPED() macro in other places in the code, like in udp6 and
sctp6,
so V4 mapped addresses are supported for other things.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

John Baldwin <j...@FreeBSD.org> changed:

What |Removed |Added
----------------------------------------------------------------------------

Status|Needs Triage |In Discussion
CC| |b...@FreeBSD.org

--- Comment #8 from John Baldwin <j...@FreeBSD.org> ---
It seems to be a non-trivial amount of work. From bms@ on IRC:

<quote>
There's no hard and fast reasons why it couldn't be done. The code as it stands
will reject that as being an API mixup (you want v4 memberships, use the v4
APIs).
The tension points are the Layer 4 ingress filtering for SSM, and actually
calling Layer 2 in the right way.

The nasty thing about IP6-mapped is that you need to track the memberships in
v6 terms, but hand-off all the work to the v4 routines to do the right thing.

The easiest way to go about doing it is to deal with the ASM case first, and
just punch a hole in ingress filtering if someone tries to use SSM (which is
what the
stack has to do anyway).

I'm not going to stick around to see what happens, though. ;-)
</quote>

I think what this means is that in_mcast6.c is very much tied to doing MLDv6
(or whatever the v6 equivalent of IGMP is), but for these groups, you need to
be somehow calling into in_mcast.c to do IGMP instead. However, I'm not sure
at what level in_mcast6.c needs to call into in_mcast.c myself. I do think
Bruce is your best bet for someone to ask.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

Ronald Klop <ronald...@klop.ws> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |ronald...@klop.ws

--- Comment #9 from Ronald Klop <ronald...@klop.ws> ---
Not a solution, but does it work as a workaround to add this option on the
commandline to java?
-Djava.net.preferIPv4Stack=true

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

--- Comment #10 from Craig Rodrigues <rod...@FreeBSD.org> ---
Ronald,

Thanks for the tip.

java -Djava.net.preferIPv4Stack=true MulticastTest

seems to work around the problem.

I would still like to see FreeBSD fixed so that this workaround is not
required.

Even though it is a lot of work, I would like to see Java on FreeBSD behave out
of the box, "just like Linux", without the FreeBSD community needing to
push lots of patches upstream to different Java software authors.
I want there to be less motivation for people to migrate from FreeBSD to Linux
if they are deploying Java applications.

That's why I've spent the time to analyze the problem and reporting my
findings in this bug report. I find this audit trail is quite interesting. :)

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

--- Comment #11 from Craig Rodrigues <rod...@FreeBSD.org> ---
-------- Forwarded Message --------
Subject: Re: [Bug 193246] Bug in IPv6 multicast join(), uncovered by
Jenkins
Date: Fri, 05 Sep 2014 03:05:14 +0100
From: Bruce Simpson <b...@fastmail.net>
To: bugzilla...@freebsd.org

On 04/09/2014 19:16, bugzilla...@freebsd.org wrote:
> I think what this means is that in_mcast6.c is very much tied to doing MLDv6
> (or whatever the v6 equivalent of IGMP is), but for these groups, you need to
> be somehow calling into in_mcast.c to do IGMP instead. However, I'm not sure
> at what level in_mcast6.c needs to call into in_mcast.c myself. I do think
> Bruce is your best bet for someone to ask.
>

Unfortunately I'm fully booked. But yes, that is a good summary.

If Craig (or someone else) is willing to volunteer to support v4-mapped
addresses: one approach would be to extend in6_mship{} to include them.
Pushing state down to IGMP will need to be added as a special case. To
keep it simple, assume that only the legacy any-source multicast (ASM)
model will be supported, i.e. listeners will not specify source filters.

Looking at the JDK source, it appears they used to handle the v4/v6
swizzle themselves because of limitations in Linux 2.4/2.6.

In other words, we do not support RFC 3493 Sec 3.7 for multicast groups
at the moment. A more appropriate errno value to return is EPROTONOOPT.
[Interestingly, Sec 5.2 discusses IPv6 multicast options, but does not
contain any reference to IPv4 at all.]

There now follows a late-night writeup of the rationale behind this code
-- and this is as concise as I can make it, I'm afraid.

in[6]_mcast.c is split into a top half (i.e. per-socket state) and a
bottom half (i.e. stack-wide state, and the IGMPv1/2/3 and MLDv1/2 wire
protocols). IPv6 mcast options are processed separately from IPv4.

Both implement a transaction scheme to protect global membership state
(in the bottom half) from failures at the granularity of a single socket
(or PCB) in the top half.

Why all the complexity? Well, this is to support source-specific
multicast (SSM, aka "inclusive mode"). To cut a long story short: as the
size of an internetwork increases, it gets more difficult for routers to
track the state of multicast listeners, unless they are aware of where
the traffic originates from. The book "Interdomain Multicast Routing" by
Brian M. Edwards discusses this in lurid detail.

So, SSM was introduced to support inter-domain multicast. In this model,
joining a multicast group is no longer a simple matter of registering
for a channel -- you must also specify the sources you are interested
in. However, the majority of multicast applications are built on the
older model: i.e. they do not cross more than one IP network hop, and do
not specify sources ("any-source multicast", aka ASM).

The network stack must be able to cope with both of these uses. It does
so by representing the legacy ASM scheme as "exclusive mode". The RFC
3678 APIs also have the advantage that the application can block
unwanted senders, even if ASM is in use. [The main API it specifies,
setsourcefilter(), does not explicitly mandate v4-mapped support.]

So, in the bottom half of mcast, each group has a shared RB-tree of
listener state. This is created from the set union of the filter state
on each subscribed socket.

If there are no filters for a particular group, i.e. all of the
sockets/PCBs in the system are in "exclusive" mode and have no filters,
then of course the RB-tree for that group will be empty. Otherwise, if
there is a mix of "exclusive" and "inclusive" mode listeners, the tree
will need to be recomputed.

The shared tree is then used to fill out the IGMP/MLD messages sent to
on-link routers to request group subscription. It is also used to filter
input traffic from within the relevant transports (e.g. UDP, raw
sockets). Previously, this filtering required that the network layer
take a socket-layer lock.

In closing: this isn't just a simple matter of adding a few defines. The
v6 code will need to check that a v4-mapped group was passed, and make
sure to perform the transaction pushdown to IGMP.

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

Marcin Cieślak <sa...@saper.info> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |sa...@saper.info

--- Comment #12 from Marcin Cieślak <sa...@saper.info> ---
Thanks for excellent analysis!

I ran into this one today at
https://discuss.gradle.org/t/messagingservicestest-cannot-join-mcast-group-on-freebsd-invalid-argument/9662

[bugzilla...@freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

ma...@bowtie.nl changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |ma...@bowtie.nl

--- Comment #13 from ma...@bowtie.nl ---
Hi,

just to pitch in with additional argumentation for resolution of this bug. I
ran into this exact problem with openHAB, the workaround works. But as the
original reporter writes, it puts of people using FreeBSD. It was also not
trivial to diagnose.

Cheers,
Marc.

--
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
freebs...@freebsd.org mailing list

https://lists.freebsd.org/mailman/listinfo/freebsd-java