Issues rebuilding bnd in Eclipse

[Fr Jeremy Krieg (Home)]

Hi team,

I pray that you are all well.

I rebased my working branch onto head yesterday and ever since I've been having the following issues:

1. It seems that the "Eclipse Oxygen 4.7.3a" repository is being fully refreshed (ie, downloading many, perhaps even all, of the bundles) very often at build and/or launch time. This makes rebuilding/launch painfully slow. The "max stale age" is set to 31536000s - surely it shouldn't be updating so much? I don't recall that it was doing this so often until quite recently. I looked at the log for repositories.bnd and saw that it was changed to an OSGi repository on March 22 - it's possible that this is the trigger.

2. Bndtools will frequently crash out during rebuild with the "Problem Occurred" dialog box when you try and launch/rebuild, with an error like this:

-----

Errors occurred during the build.
Errors running builder 'Bndtools Builder' on project 'aQute.libg'.
Build Error!
C:\workspace\bnd\aQute.libg\generated\aQute.libg.jar: The process cannot access the file because it is being used by another process.

Build Error!
C:\workspace\bnd\aQute.libg\generated\aQute.libg.jar: The process cannot access the file because it is being used by another process.

----- 

Here is a copy of the event details from the Eclipse error log for a similar issue (same error, but for biz.aQute.bndlib instead of aQute.lib):

----

eclipse.buildId=4.10.0.I20181206-0815
java.version=1.8.0_131
java.vendor=Oracle Corporation
BootLoader constants: OS=win32, ARCH=x86_64, WS=win32, NL=en_AU
Framework arguments:  -product org.eclipse.epp.package.jee.product
Command-line arguments:  -os win32 -ws win32 -arch x86_64 -product org.eclipse.epp.package.jee.product

bndtools.builder
Error
Fri Jun 14 13:22:05 ACST 2019
Errors running builder 'Bndtools Builder' on project 'biz.aQute.bndlib'.

org.eclipse.core.runtime.CoreException: Build Error!
at org.bndtools.builder.BndtoolsBuilder.build(BndtoolsBuilder.java:270)
at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:833)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:220)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:263)
at org.eclipse.core.internal.events.BuildManager$1.run(BuildManager.java:316)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:319)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:278)
at org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:468)
at org.eclipse.core.internal.resources.Project$1.run(Project.java:574)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2295)
at org.eclipse.core.internal.resources.Project.internalBuild(Project.java:548)
at org.eclipse.core.internal.resources.Project.build(Project.java:116)
at org.eclipse.debug.core.model.LaunchConfigurationDelegate$1.run(LaunchConfigurationDelegate.java:424)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2295)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2317)
at org.eclipse.debug.core.model.LaunchConfigurationDelegate.buildProjects(LaunchConfigurationDelegate.java:431)
at org.eclipse.debug.core.model.LaunchConfigurationDelegate.buildForLaunch(LaunchConfigurationDelegate.java:127)
at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:832)
at org.eclipse.debug.internal.core.LaunchConfiguration.launch(LaunchConfiguration.java:720)
at org.eclipse.debug.internal.ui.DebugUIPlugin.buildAndLaunch(DebugUIPlugin.java:1029)
at org.eclipse.debug.internal.ui.DebugUIPlugin$2.run(DebugUIPlugin.java:1243)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Caused by: java.nio.file.FileSystemException: C:\workspace\bnd\biz.aQute.bndlib\generated\biz.aQute.bndlib.jar: The process cannot access the file because it is being used by another process.

at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1126)
at aQute.lib.io.IO$2.visitFile(IO.java:811)
at aQute.lib.io.IO$2.visitFile(IO.java:803)
at java.nio.file.Files.walkFileTree(Files.java:2670)
at java.nio.file.Files.walkFileTree(Files.java:2742)
at aQute.lib.io.IO.deleteWithException(IO.java:803)
at aQute.lib.io.IO.deleteWithException(IO.java:784)
at aQute.bnd.build.Project.saveBuild(Project.java:1983)
at aQute.bnd.build.Project.buildLocal(Project.java:1890)
at aQute.bnd.build.Project.build(Project.java:1688)
at aQute.bnd.build.Project.build(Project.java:2316)
at org.bndtools.builder.BndtoolsBuilder.lambda$build$0(BndtoolsBuilder.java:244)
at bndtools.central.Central.bndCall(Central.java:676)
at org.bndtools.builder.BndtoolsBuilder.build(BndtoolsBuilder.java:124)
... 23 more
-----

I've had a similar error with biz.aQute.bndlib. Cleaning/rebuilding doesn't seem to make it go away. I cannot manually delete the file in Eclipse to force a rebuild - I can't even delete it through Explorer or on the command line - I get "resource busy/in use" or equivalent. The only way around it seems to be to exit Eclipse, delete manually, and then restart Eclipse. Which is made even more painful because of the first point (Bndtools re-downloads all of the Eclipse 4.7.3a repo again after restart when it does the "Load repositories").

It is possible that one or both of these issues don't manifest on Linux/Mac - I know that Windows is a bit more aggressive at protecting locked/in-use files from deletion than Linux is.

I assume that this is happening because the running Eclipse/Bndtools instance is binding to the development versions of the bnd libs - however, I don't recall it happening before. It is possible that this is simply because aQute.lib and biz.aQute.bndlib typically don't change very often and they changed in recent days. It seems to have gone away now that I have closed Eclipse, run "rm */generated/*.jar" from the project root to delete all generated jars, and then restarted Eclipse. However, the downloading issue for the Eclipse OSGi repo still remains.

Any thoughts/tips/tricks?

Blessings,

Fr Jeremy

[BJ Hargrave]

First suggestion, stop using Windows :-)

Some other process is locking the files. Do you use allow gradle to
make a daemon? You may want to make sure it it stopped and perhaps
even disable gradle from using a long running daemon.

You may also want to try a reboot. Perhaps your system is unhappy in some way.

I do not notice any issue on my system (mac) with regard to
continuously downloading the eclipse repo. This could be a symptom of
the file locking issue.

> --
> You received this message because you are subscribed to the Google Groups "bndtools-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bndtools-dev...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/CAO6F8YxE2WjHE6tjOajStjJSfwWYoAaxBopqGQ1mj%2B9-Q7pZ8A%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.



--

BJ

[Fr Jeremy Krieg (Home)]

Hi BJ, thank you for the response,

I think you've already suggested to me to stop using Windows... but then who will you have to find all the Windows bugs? :)

I'm not using Gradle, and I have the same thing happening on two different Windows machines, so I don't think it's simply my machine being unhappy.

Workaround at the moment is to put it in offline mode. I'll try and dig a bit deeper to see what's going on.

Blessings,

Fr Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/CABr4vaynW5vGETz%3DME0QSyULmnGMLRjQWz96U7DzEKoTG6fAFg%40mail.gmail.com.

[Fr Jeremy Krieg (Home)]

In order to dig deeper - any tips on how to see the debugging output from OSGiIndex/OSGiRepository when it is running from within Bndtools?

Blessings,

Fr Jeremy

[Peter Kriens]

You can start Eclipse in debug mode and then attach the Eclipse debugger.

Add:

-Xrunjdwp:transport=dt_socket,address=1045,server=y,suspend=n

To eclipse.ini in a Mac is in:

/Users/aqute/eclipse/java-photon/Eclipse.app/Contents/Eclipse/eclipse.ini

I've seen the same behavior and it would be nice if it could be fixed since it is annoying.

Kind regards,

Peter Kriens

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/CAO6F8YySfAnh%2B0e-F%3DZE71u3wpSgoBihycJT-pjmY3ocuJWo_g%40mail.gmail.com.

[Fr Jeremy Krieg (Home)]

Thanks Peter, I should be able to get the debugging working with that as I know where the ini file is.

Which behavior have you seen - the repetitive downloading of EclipseOxygen, or the locking of bndlib?

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/E75E76CF-9D96-43E5-B878-FD10D0741BC2%40aqute.biz.

[Peter Kriens]

The downloading of eclipse. However, did you not already find the culprit? The fact that OSGiRepository is clearing the cache on a reset seems very aggressive. 

I can make a patch to remove this if you want.

The locking problem is nasty. We solve a lot of problems by finding out where we kept open files and more aggressively close them. I've got a few customers that intensely use bndtools on Windows. Although we had earlier problems it seems we fixed them (also fixes in Groovy and Eclipse for this). The exact problem you indicate is new to me.

There is a file leak detector that you can use inside Eclipse. I've got the following in my notes:

fd.jar is the JAR from http://file-leak-detector.kohsuke.org/that should be added as an agent. This requires changing eclipse.ini to add a new line after -vmargs

-vmargs

-javaagent:fd.jar=strong,dumpatshutdown

You can find Eclipse.ini in the same directory as eclipse.exe. For me it was in /c/Users/micro/eclipse/java-oxygen/eclipse.
You should then start Eclipse from the command line and redirect the output to a file. Once you run into that problem, quit. This will dump the open files. Can be quite surprisingly large. I need this file to figure out who is holding an open connection.

To trace the earlier Windows lock bugs I made extensive traces and then matched the open and close instructions. You can send the output file to me if you can reliably create this situation.

Kind regards,

Peter Kriens

[Fr Jeremy Krieg (Home)]

Hi Peter,

Yes, I did find the culprit for the re-downloading problem in the recent issue that I raised. I was still curious which problem it was that you had hit too, because if it was the locking problem I was going to ask for some more data.

Thankfully, you've given some good tips for investigating the locking problem anyway - I'll try and install that file locking agent and see how I go. I made some initial progress by install Process Monitor and keeping an eye on Eclipse and the children it spawns to see which files they are holding open - I can clearly see it holding both generated/biz.aQute.bndlib.jar and generated/aQute.libg.jar open. With the addition of that agent I should be able to see what process is opening it, and from there possibly figuring out why it isn't closing. If I can't, I'll send you whatever dump files I manage to generate.

I do remember earlier versions of Bndtools having more issues with file locking and I can affirm the fact that recent versions they basically disappeared - kudos to you guys. This problem I am describing seems to only have starting happening the last time I rebased my junit-bumpted thread (sorry for the name of that branch - a typo that stuck...).

Blessings,

Fr Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/D9DCD629-C780-480C-B1A0-ED8BA3F1E796%40aqute.biz.

[Fr Jeremy Krieg (Home)]

Hi team,

I installed the file leak detector and kept using it until the file locking error appeared again. I ran it in HTTP mode, so when the error occurred I was able to get a snapshot of open files through the browser. I have attached it (zipped, because unzipped it is 1.2MB).

The file that Eclipse got stuck on was "C:\Users\Jeremy\git\bnd\biz.aQute.bndlib\generated\biz.aQute.bndlib.jar". In the dump, it appears to be open four times in three threads.

The stacktrace from the build error that complained about the locked file is here:

-----

java.nio.file.FileSystemException: C:\Users\Jeremy\git\bnd\biz.aQute.bndlib\generated\biz.aQute.bndlib.jar: The process cannot access the file because it is being used by another process.

at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:270)
at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
at java.base/java.nio.file.Files.delete(Files.java:1134)

at aQute.lib.io.IO$2.visitFile(IO.java:811)
at aQute.lib.io.IO$2.visitFile(IO.java:803)

at java.base/java.nio.file.Files.walkFileTree(Files.java:2713)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2785)

at aQute.lib.io.IO.deleteWithException(IO.java:803)
at aQute.lib.io.IO.deleteWithException(IO.java:784)
at aQute.bnd.build.Project.saveBuild(Project.java:1983)
at aQute.bnd.build.Project.buildLocal(Project.java:1890)
at aQute.bnd.build.Project.build(Project.java:1688)
at aQute.bnd.build.Project.build(Project.java:2316)
at org.bndtools.builder.BndtoolsBuilder.lambda$build$0(BndtoolsBuilder.java:244)
at bndtools.central.Central.bndCall(Central.java:676)
at org.bndtools.builder.BndtoolsBuilder.build(BndtoolsBuilder.java:124)

at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:833)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:220)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:263)
at org.eclipse.core.internal.events.BuildManager$1.run(BuildManager.java:316)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:319)

at org.eclipse.core.internal.events.BuildManager.basicBuildLoop(BuildManager.java:371)
at org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:392)
at org.eclipse.core.internal.events.AutoBuildJob.doBuild(AutoBuildJob.java:154)
at org.eclipse.core.internal.events.AutoBuildJob.run(AutoBuildJob.java:244)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

----

Hope this helps. Let me know if I can be of any further assistance in this area.

Blessings,

Fr Jeremy

[Peter Kriens]

I think the problem might be the -make facility. It seems the Builder is not properly closed there.

I've made a fix (I hope). It has been built, can you test?

Kind regards,

Peter Kriens

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/CAO6F8YxR8g%2Bu_v%2BuWSMPMF_6HVZOOxLfhReG38sxih2MQmwgBw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

<open files.zip>

[Fr Jeremy Krieg (Home)]

Happy to test it... forgive me if I'm asking a silly question, but where do I get the patched version from?

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/860C3168-5220-4AC1-8CC5-06F4C2BB8A41%40aqute.biz.

[Peter Kriens]

From the jfrog repository

https://bndtools.jfrog.io/bndtools/update/biz/aQute/bnd/

You have to select the latest snapshot by hand.

Kind regards,

Peter Kriens

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/CAO6F8Yzjo%3D1QQ8uCoUzxvuC-tuN73%2BQfXSRve2hvQ1xv8wvgxQ%40mail.gmail.com.

[BJ Hargrave]

See https://github.com/bndtools/bnd#using-the-latest-development-snapshot-build-of-bndbndtools
for how to get the latest version.

On Wed, Jun 19, 2019 at 8:28 AM Fr Jeremy Krieg (Home)

> To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/CAO6F8Yzjo%3D1QQ8uCoUzxvuC-tuN73%2BQfXSRve2hvQ1xv8wvgxQ%40mail.gmail.com.

[Fr Jeremy Krieg (Home)]

Thanks guys - I was already set up to use that update site, I just didn't want to be looking there and Peter had actually published it somewhere else.

I'm updating my Eclipse instance now with the latest Bndtools snapshot, so I'll see how I go.

Blessings,

Fr Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/CABr4vaycSgnSzpDAf3h8RBm1t8%3D7jLauREBVLhar_3PNhQLuNw%40mail.gmail.com.

[Fr Jeremy Krieg (Home)]

Ok, good news - I think it's fixed.

I updated Eclipse to the latest snapshot, cleaned my workspace and did a full rebuild. The file leak detector's results are attached at the end of the rebuild are attached.

- There were only 238 files open (the last trace I sent had 263).

- There are *no* open files that match the pattern generated\biz.aQute.bndlib.jar, or even that matched generated\. Before this patch, I noticed that there were many files left open (often across multiple threads) in generated\ at the end of the full rebuild - even if the file locking error hadn't yet triggered.

So I think this is pretty good evidence that it's been taken care of. Kudos again to Peter.

I noticed also that "file leak detector" has an API. Maybe this could be used in an integration test to make sure that there are no stray open files at the end of a build, to prevent regressions.

Blessings,

Fr Jeremy

[Fr Jeremy Krieg (Home)]

Hi all,

It looks like I spoke too soon - it happened again. Please find open files attached. Stack trace was:

java.nio.file.FileSystemException: C:\Users\Jeremy\git\bnd\biz.aQute.resolve\generated\biz.aQute.resolve.jar: The process cannot access the file because it is being used by another process.

Blessings,

Fr Jeremy

[Peter Kriens]

The trace shows the use of Groovy. Are you using the latest version of Groovy? There was a class loader bug in Groovy that interacted badly with bnd. Since we put the JARs on the classpath and then rebuild them Groovy ran into problems because they never closed them.

I worked with them about 1-2 years ago to fix this.

My dislike for Windows and Groovy are competing rather heavily :-(

PK

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/CAO6F8Yx%3DpAD0ZjXJP%2BQDx3ShBS-JwvvYT0BFTdUXLK5HMCgZEg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

<open files (again).zip>

[Fr Jeremy Krieg (Home)]

I did install GDT when I was trying to look at the Gradle plugin source and tests. That was only yesterday though, and after I installed your patch. I'll try uninstalling it and see if the problem goes away.

Blessings,

Fr Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/C2EF8202-C7A7-49A9-A39F-ECA588C8D4C1%40aqute.biz.

[Peter Kriens]

At my customers they are (unfortunately) using Groovy so it is possible to get it to work. However, you need the latest version.

Kind regards,

Peter Kriens

To view this discussion on the web visit https://groups.google.com/d/msgid/bndtools-dev/CAO6F8Yz6sjof41dvfOikzffa9Eup66PUvnMJX_SHK8ZXk-Av5g%40mail.gmail.com.