Remote Debug Launcher Non-Deterministically Hangs On Connection

[Jon Passki]

Hello,

When launching a bnd debug configuration that uses the remote bnd launcher, sometimes it non-deterministically will hang on launch, stuck around 57% progress with the following text in the Progress view:

"Launching [name of debug launch config]
Connecting debugger test to localhost:5005: Establishing connection..."

Doing one of the following usually works:

* Uninstalling and reinstalling the remote bnd agent bundle in the debug target Eclipse instance. meh
* Restarting the target Eclipse instance. meh
* Restarting both the target and development Eclipse instances. :-/
* Logging off and logging back in. :(
* Restarting my development workstation. :(

I would like to understand why the above is happening and not have to either log off / restart the workstation if possible to clear out the issue. It's currently stuck at that stage right now though, so good time to troubleshoot! :)

My current setup is as follows (Eclipse Neon.2 on macOS 10.12):

* Target Eclipse with the following added to its eclipse.ini:

```
-Xdebug
-Xnoagent
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005
-Djava.compiler=NONE
```

In the target Eclipse instance I manually open the Host OSGi console and install the bnd remote agent:

```
WARNING: This console is connected to the current running instance of Eclipse!
osgi> install -start http://central.maven.org/maven2/biz/aQute/bnd/biz.aQute.remote.agent/3.3.0/biz.aQute.remote.agent-3.3.0.jar
Bundle id is 585
Host localhost 29998
Version 3.3.0.201609221907
LastModified 1485366075108
Headers Bnd-LastModified = 1474571251923
Bundle-Activator = aQute.remote.agent.Activator
Bundle-Copyright = Copyright (c) aQute SARL (2000, 2016) and others. All Rights Reserved.
Bundle-Description = This bundle acts as an agent for the remote launching architecture. When started, it will bind to port 29998 and listens for incoming requests. When this port is accessed, it will open a link that allows it to act as an OSGi management agent that is remotely controlled. When this connection is closed, it will remove all bundles that were installed through this connection. The protocol is highly optimized to make this also work for small devices and slow connections.
Bundle-Developers = pkriens; email=Peter....@aQute.biz; name="Peter Kriens"; organization=bndtools; organizationUrl=http://bndtools.org; roles="architect,developer"; timezone=1
Bundle-DocURL = http://bnd.bndtools.org/
Bundle-License = Apache-2.0; description="Apache License, Version 2.0"; link="http://www.apache.org/licenses/LICENSE-2.0"
Bundle-ManifestVersion = 2
Bundle-Name = biz.aQute.remote.agent
Bundle-SCM = url=https://github.com/bndtools/bnd, connection=scm:git:https://github.com/bndtools/bnd.git, developerConnection=scm:git:g...@github.com:bndtools/bnd.git
Bundle-SymbolicName = biz.aQute.remote.agent
Bundle-Vendor = Bndtools http://bndtools.org/
Bundle-Version = 3.3.0.201609221907
Conditional-Package = aQute.lib*, aQute.configurable.*,aQute.service.reporter, aQute.bnd.util.*
Created-By = 1.7.0_79 (Oracle Corporation)
Embedded-Activator = aQute.remote.agent.Activator
Git-Descriptor = 3.3.0.REL
Git-SHA = 3e1d384fab0837c62449d1da8c0c15674be2b406
Import-Package = org.osgi.framework;version="[1.5,2)",org.osgi.framework.launch;version="[1.0,2)",org.osgi.framework.wiring;version="[1.0,2)",javax.management,javax.management.openmbean,javax.management.remote,org.osgi.resource;version="[1.0,2)",org.osgi.util.tracker;version="[1.5,2)",org.osgi.dto;resolution:=optional;version="[1.0,2)",org.osgi.framework.dto;resolution:=optional;version="[1.8,2)",org.osgi.resource.dto;resolution:=optional;version="[1.0,2)",org.osgi.framework.wiring.dto;resolution:=optional;version="[1.2,2)"
Manifest-Version = 1.0
Private-Package = aQute.remote.api;version="1.1.0",aQute.remote.agent,org.apache.felix.service.command,org.osgi.dto;version="1.0",org.osgi.resource.dto;version="1.0",org.osgi.framework.dto;version="1.8",org.osgi.framework.wiring.dto;version="1.2",aQute.remote.util;version="1.0.0",aQute.bnd.util.dto;version="1.0",aQute.lib.collections;version="2.1.0",aQute.lib.converter;version="2.0.1",aQute.lib.io;version="1.6.0",aQute.lib.json;version="3.2.0",aQute.libg.cryptography;version="1.1.0",aQute.libg.shacache;version="1.0.0",aQute.lib.base64;version="1.3.0",aQute.lib.hex;version="1.2.0",aQute.libg.glob;version="1.2.0",aQute.libg.tuple;version="1.0",aQute.lib.strings;version="1.5.0"
Require-Capability = osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.6))"
Tool = Bnd-3.3.0.201609221844


Location http://central.maven.org/maven2/biz/aQute/bnd/biz.aQute.remote.agent/3.3.0/biz.aQute.remote.agent-3.3.0.jar
State 32
Module osgi.identity; osgi.identity="biz.aQute.remote.agent"; type="osgi.bundle"; version:Version="3.3.0.201609221907" [id=585]
Bundle 585|Active | 1|biz.aQute.remote.agent (3.3.0.201609221907)
BundleContext org.eclipse.osgi.internal.framework.BundleContextImpl@26e30bb
BundleId 585
SymbolicName biz.aQute.remote.agent
RegisteredServices null
ServicesInUse null

osgi>
```

So far so go. After the remote bnd agent loads, lsof shows the following:

```
$ lsof -lnP -iTCP:5005 -iTCP:29998
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
eclipse 55693 504 5u IPv4 0xa6c026ed3dfd0f1b 0t0 TCP *:5005 (LISTEN)
eclipse 55693 504 154u IPv6 0xa6c026ed24393b63 0t0 TCP 127.0.0.1:29998 (LISTEN)
```

Hmm, the bnd agent is listening on IPv6 but the debugger is on IPv4...

In the development Eclipse, I have the following `debug.bndrun` that I launch via the context menu, Debug As -> Bnd Native Launcher:

```
#
# DEBUG LAUNCH SPECFICATION
#

# -include: ~${p}.bndrun

-runpath: biz.aQute.remote.launcher

-runremote: test;\
shell = -1; \
jdb = 5005; \
host = localhost; \
agent = 29998; \
timeout = 10

-runproperties: gosh.args=--noshutdown

-runrequires.debug: \
${debug-bundles}

-runbundles: \
some.bundle.a.provider,\
some.bundle.b.provider,\
```

When launched, lsof sees this:

```
$ lsof -lnP -iTCP:5005 -iTCP:29998
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
eclipse 55693 504 60u IPv4 0xa6c026ed42c72bf3 0t0 TCP 127.0.0.1:5005->127.0.0.1:62079 (ESTABLISHED)
eclipse 55693 504 154u IPv6 0xa6c026ed24393b63 0t0 TCP 127.0.0.1:29998 (LISTEN)
eclipse 55823 504 116u IPv6 0xa6c026ed2910d5e3 0t0 TCP 127.0.0.1:62079->127.0.0.1:5005 (ESTABLISHED)
```

Well, that's just weird. I don't see how the OS can establish an IPv6 connection to an IPv4 port...

OK, so now seeing that IPv4 and IPv6 are in play, I stopped both Eclipse instances and added `-Djava.net.preferIPv4Stack=true` to each of their `eclipse.ini`.

lsof after restarting the target Eclipse instance and installing the bnd remote agent:

```
$ lsof -lnP -iTCP:5005 -iTCP:29998
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
eclipse 56221 504 5u IPv4 0xa6c026ed3d5f3a03 0t0 TCP *:5005 (LISTEN)
eclipse 56221 504 65u IPv4 0xa6c026ed42c722fb 0t0 TCP 127.0.0.1:29998 (LISTEN)
```

lsof after starting the development Eclipse instance and attempting to launch the remote bnd debug launcher:


```
$ lsof -lnP -iTCP:5005 -iTCP:29998
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
eclipse 56221 504 65u IPv4 0xa6c026ed42c722fb 0t0 TCP 127.0.0.1:29998 (LISTEN)
eclipse 56221 504 140u IPv4 0xa6c026ed3d83b4eb 0t0 TCP 127.0.0.1:5005->127.0.0.1:63008 (ESTABLISHED)
eclipse 56318 504 157u IPv4 0xa6c026ed3d83a2fb 0t0 TCP 127.0.0.1:63008->127.0.0.1:5005 (ESTABLISHED)
```

OK, so only IPv4 seems to be in use. However, the progress bar is still at 57% with the same text as above.

Is there something I'm doing incorrectly above? If not, what's the best way to proceed? Enable debugging on my development Eclipse instance and breakpoint on the bnd remote launcher?

Cheers,

Jon

[Peter Kriens]

You should be able to cancel the launch and retry. What happens then? Could there be a hanging connection from an earlier session? I think you can only establish 1 connection at a time.

Kind regards,

Peter Kriens

> --
> You received this message because you are subscribed to the Google Groups "bndtools-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-user...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[j...@coastalhacking.us]

OK, I debugged my debugger :)

Here's what I'm seeing:

Thread [Worker-19] (Suspended (breakpoint at line 146 in org.eclipse.jdt.internal.launching.SocketAttachConnector))

org.eclipse.jdt.internal.launching.SocketAttachConnector.connect(java.util.Map<java.lang.String,java.lang.String>, org.eclipse.core.runtime.IProgressMonitor, org.eclipse.debug.core.ILaunch) line: 146

bndtools.launch.bnd.LaunchThread.doDebug(org.eclipse.core.runtime.IProgressMonitor) line: 69

bndtools.launch.bnd.NativeBndLaunchDelegate.launch(org.eclipse.debug.core.ILaunchConfiguration, java.lang.String, org.eclipse.debug.core.ILaunch, org.eclipse.core.runtime.IProgressMonitor) line: 131

org.eclipse.debug.internal.core.LaunchConfiguration.launch(java.lang.String, org.eclipse.core.runtime.IProgressMonitor, boolean, boolean) line: 885

org.eclipse.debug.internal.core.LaunchConfiguration.launch(java.lang.String, org.eclipse.core.runtime.IProgressMonitor, boolean) line: 739

org.eclipse.debug.internal.ui.DebugUIPlugin.buildAndLaunch(org.eclipse.debug.core.ILaunchConfiguration, java.lang.String, org.eclipse.core.runtime.IProgressMonitor) line: 1039

org.eclipse.debug.internal.ui.DebugUIPlugin$8.run(org.eclipse.core.runtime.IProgressMonitor) line: 1256

org.eclipse.core.internal.jobs.Worker.run() line: 55

I noted an `java.lang.IllegalArgumentException` thrown in `org.eclipse.jdt.internal.launching.SocketAttachConnector.connect(Map<String, String>, IProgressMonitor, ILaunch)` at this line:

IDebugTarget debugTarget= JDIDebugModel.newDebugTarget(launch, vm, vmLabel, null, allowTerminate, true);

The exception was "Path for project must have only one segment". This resulted in the exception being caught back in `bndtools.launch.bnd.LaunchThread.doDebug(IProgressMonitor)` with the thread sleeping for 500 ms. It then tried to connect to an already connected VM (since the previous connection wasn't disconnected, see below) ad infinitum.

Reading around, this seems to occur when a debug configuration doesn't point to a Project. For the "bnd Native", this is settable via the "Connect" tab.

Now I think here's the "non-deterministic" part. If it's the first time I launch the configuration via the context menu, everything is OK. Everything will be OK until I do the following:

* Open the newly created "bnd Native" debug configuration

* Navigate to the "Connect" tab

* Navigate to some other configuration without making any changes. Eclipse pops up a Save Changes dialog. If I click "No", no issues. However, if I click "Yes", from that point forward, the debug configuration will throw the "Path for project must have only one segment" exception and then subsequently will throw connection refused exceptions ad infinitum.

A potential fix in bndtools would be to ensure a Project is set when the debug configuration is first created.

`SocketAttachConnector` should maybe also catch `java.lang.IllegalArgumentException` and abort the connection, which it doesn't do. That's a JDT eclipse bug I believe.

At least I can work around it now :)

[Peter Kriens]

Can you file a bug? Or even better, with a patch?

Thanks for figuring this out. Kind regards,

Peter Kriens

[Jon Passki]

I got a local patch working and tested using my use case below. I'll read up on the contributing docs to get a patch out.

You received this message because you are subscribed to a topic in the Google Groups "bndtools-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bndtools-users/Fb5AJy2uaG8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bndtools-user...@googlegroups.com.

[Nikhil Chilwant]

Hi,

I am facing the same issue. How did you fix it?

I am using OSGi enroute template, Bndtools 3.3.0 (latest Bndtool has issue with OSGi enroute), Windows 7, Java8, Oxygen.2 Release (4.7.2)

> Bundle-Developers = pkriens; email=Pet...@aQute.biz; name="Peter Kriens"; organization=bndtools; organizationUrl=http://bndtools.org; roles="architect,developer"; timezone=1 

> Bundle-DocURL = http://bnd.bndtools.org/ 
> Bundle-License = Apache-2.0; description="Apache License, Version 2.0"; link="http://www.apache.org/licenses/LICENSE-2.0" 
> Bundle-ManifestVersion = 2 
> Bundle-Name = biz.aQute.remote.agent 

> Bundle-SCM = url=https://github.com/bndtools/bnd, connection=scm:git:https://github.com/bndtools/bnd.git, developerConnection=scm:git:git...@github.com:bndtools/bnd.git 

[Jon Passki]

Hello Nikhil,

bndtools issue 1639 tracks it: https://github.com/bndtools/bndtools/issues/1639. Related is JDT issue 511236: https://bugs.eclipse.org/bugs/show_bug.cgi?id=511236.

Jon