Issue 144 in q4e: Dependency Tree Viewer

[codesite...@google.com]

Issue 144: Dependency Tree Viewer
http://code.google.com/p/q4e/issues/detail?id=144

Comment #3 by jake.pezaro:
i have a tool that does this
(http://code.google.com/p/maven-dependency-browser/) but
it is a separate app at the moment. it show raw (un-resolved or
hidden) dependencies
in a tree.

is q4e interested in incorporating this? if so i will make the core functionality
available

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings

[codesite...@google.com]

[codesite...@google.com]

Issue 144: Dependency Tree Viewer
http://code.google.com/p/q4e/issues/detail?id=144

Comment #4 by amuino:
Your application looks great. We had simpler functionality in mind, but
your tool
looks awesome for dependency analysis.

Why don't we discuss in the dev mailing list how your work can be
incorporated into q4e ?

[codesite...@google.com]

Issue 144: Dependency Tree Viewer
http://code.google.com/p/q4e/issues/detail?id=144

Comment #5 by jake.pezaro:
sure. just point me in the direction of the mailing list please

[codesite...@google.com]

[codesite...@google.com]

Issue 144: Dependency Tree Viewer
http://code.google.com/p/q4e/issues/detail?id=144

Comment #6 by emantos:
The mailing list is at -> http://groups.google.com/group/q4e-dev

[jake.pezaro]

i've already separated most of the core code from the swing GUI
implementation, but i need to tidy it up and release it which i can do
in the next day or two.

what this core code does is read pom files, construct a tree of
objects representing the dependencies and lists of objects and
duplicates. it should just be a matter of skinning this with an
eclipse gui and allowing it get it's config from eclipse.

On Jan 16, 3:03 pm, codesite-nore...@google.com wrote:
> Issue 144: Dependency Tree Viewerhttp://code.google.com/p/q4e/issues/detail?id=144

[Abel Muiño]

By what you say, it should be easy to integrate at the view layer... We'll need to use Eclipse Viewer (inside a new Eclipse View) which is able to display the tree of objects. Since eclipse already allows for clean separation between views and models, in theory, it should be quite easy.

I wouldn't like to parse the pom files for drawing the dependencies (looks expensive), so maybe we could allow the "extra" information to be enabled/disabled per user request.

Also, please take a look at the Dependency Graph available in q4e, since that already uses q4e code to extract the dependencies and construct a model. Maybe it can be reused or extended so we have a single "core" implementation for all dependency viewers.

--
Abel Muiño - http://ramblingabout.wordpress.com/

[jake.pezaro]

as far as i know you have to parse the pom files to get at the
"hidden" dependency information as i belive that maven discards this
information when it does the dependency resolution. i wrote the tool
because this hidden (discarded) dependency information is very useful
when trying to work out what maven is doing, why and where the
relevant pom files are. it is particularly helpful when maven is not
doing what you think it should be.

the parsing uses maven's code and is fairly fast as it is only reading
from the local repo (not downloading the files itself). for example
the current (single threaded) browser can read and parse a dependency
tree with 5000 leaf nodes in 5-6 seconds (which would take several
minutes for q4e to download the dependencies for this pom). the other
key point is that this information is static and that once parsed it
is unlikely to change unless another version of a pom is installed
into the local repository.

what i would suggest is that this functionality is implemented in such
a way that

a) the view can only be opened for a project once q4e has finished
resolving and downloading dependencies for the project. this is to
retain the speed benefits of reading the poms from the local repo.
b) that it performs the parse operation on open and on user request
subsequently, preventing excessive use of eclipse resources.
c) that the resolved dependency tree (what you are currently using to
power your dependency graph) is overlaid on top of the hidden
dependency tree (this would be even more useful). your suggestion of
merging the codebases so that there is one source of dependency info
would be well worth examining.

if the q4e project is not that interested in the hidden dependencies
then i would suggest that my project does not have much to offer that
q4e could not write easily anyway. for example if you just want a
tree of resolved dependencies and a list of artifacts then that is
pretty easy to do once you have the model (which you already do).

let me know what you think.

jake

On Jan 16, 4:06 pm, "Abel Muiño" <amu...@gmail.com> wrote:
> By what you say, it should be easy to integrate at the view layer... We'll
> need to use Eclipse Viewer (inside a new Eclipse View) which is able to
> display the tree of objects. Since eclipse already allows for clean
> separation between views and models, in theory, it should be quite easy.
>
> I wouldn't like to parse the pom files for drawing the dependencies (looks
> expensive), so maybe we could allow the "extra" information to be
> enabled/disabled per user request.
>
> Also, please take a look at the Dependency Graph available in q4e, since
> that already uses q4e code to extract the dependencies and construct a
> model. Maybe it can be reused or extended so we have a single "core"
> implementation for all dependency viewers.
>

[Abel Muiño Vizcaino]

First, we are very interested in your contribution!

Now, as for the details... 

I was thinking on an unified "dependency" view showing both the actual and the "hidden" dependencies. 

This view might need to be updated on every modification to the pom of any project in the workspace (since there might be references between them). Also, it should follow the focus and switch to show the dependencies of the "current" project (the one selected or the parent project of the selected file).

For this scenario, we need to check if the refresh rate is good enough (i.e. it does not block the user task).

In my opinion, the best way of knowing for sure would be implementing the feature... we can do that in a branch for 0.5.0. Would that work for you?

We can help you navigate the q4e codebase and with anything else you'll need.

If we later find out that displaying the hidden dependencies is too costly, we can package it as a "Dependency analysis view". In any case, the effort invested will be useful.

--

Abel Muiño Vizcaino - http://ramblingabout.wordpress.com

[Jake Pezaro]

ok. it all sounds sensible to me.

i'm not that familiar with eclipse plugin development, so if you can
give me some suggestions as to where to start looking within your
codebase that would speed things up. i'll probably have some more
questions once I get started. you should probably hold off making any
branches until i have a bit better idea of your codebase & what i need
to do.

[Carlos Sanchez]

I don't know if you are aware of maven-dependency-tree
https://svn.apache.org/repos/asf/maven/shared/trunk/maven-dependency-tree

It builds a tree object of the dependencies and stores all the
information about them. The library and maven have been improved
lately to throw all the dependency handling events, so this library
has all the needed information.
I have used it in the bundle plugin and it's quite simple.

dependencyTree =
dependencyTreeBuilder.buildDependencyTree( project, localRepository,
factory,

artifactMetadataSource, null, collector );

in https://svn.apache.org/repos/asf/felix/trunk/bundleplugin/src/main/java/org/apache/felix/bundleplugin/BundleAllPlugin.java

--
I could give you my word as a Spaniard.
No good. I've known too many Spaniards.
-- The Princess Bride

[Jake Pezaro]

i've seen the output of dependency:tree. as far as i could see it
didn't output the dependencies that maven didn't resolve, however I
never looked at the internals. does this plugin throw events for
dependencies that don't end up in the resolved tree?

[Carlos Sanchez]

The tree is complete. Each node has a state, one of them I think is
ignored, and the reason

In the bundle plugin I did
node.getState() != DependencyNode.INCLUDED

[Jake Pezaro]

ok, i'll try implementing it with the dependency:tree plugin first.
if i can get it working it will mean less maintenance.

thanks for the pointers

[Jake Pezaro]

if i am going to reuse the dependency tree code then i think need to
have access to the embedder or plexus container from
org.devzuz.q.maven.dependency.actions.ViewDependencyGraphAction,
however the AbstractMavenAction only provides access to the
MavenProject.

what i need to know is - is there a embedder/plexus container
somewhere that i can access from the action or do i need to
instantiate one myself?

[Abel Muiño Vizcaino]

In q4e, the maven embedder is wrapped by the EclipseMaven class. You can access the current instance via MavenCoreActivator.getMavenInstance(). As for plexus... the only point were I ever needed to access it is in the EclipsePlexusContainerCustomizer, which I don't think will be useful to you. Just out of curiosity... why do you need to access the plexus container?

Regarding code reuse... it would be a bonus, but don't let that stop you. We can refactor the code later.

--

Abel Muiño Vizcaino - http://ramblingabout.wordpress.com

[Jake Pezaro]

you're right, i don't need to access the plexus container but i do
need to access the embedder. both my original code and the
dependency:tree plugin code require maven components to operate. as
you know, these components cannot be instantiated directly (using new)
as they have private class members that do not have setters.

the reason i asked is that eclipse telling me that the getEmbedder()
method is restricted when i try to access the embedder using
MavenCoreActivator.getDefault().getMavenInstance().getEmbedder();

Access restriction: The method getEmbedder() from the type
EclipseMaven is not accessible due to restriction on required library
D:\dev\eclipse\plugins\org.devzuz.q.maven.core_0.3.0.200711240237.jar

i haven't yet been able to work out why this is happening. if you
could give me a push in the right direction that would be very
helpful.

[Jake Pezaro]

sorry, what i should have asked is - what is the reason for this
restriction and is there a reason that I can't just turn it off?

[Jake Pezaro]

ignore the last 2 emails. what i need is a way of retrieving
components out of the plexus container ie:

org.codehaus.plexus.embed.Embedder.lookup(String role)

the latest (2.0.4) Maven embedder does not allow access to the plexus
Embedder.lookup() method (i think it did in earlier versions but that
was changed).

[Carlos Sanchez]

the Maven embedder / plexus container shouldn't be exposed as they are
internal implementation details, to isolate possible changes there.
Just add IMaven methods to get those components you need, eg
ArtifactResolver, and then implement them in the EclipseMaven, where
the embedder and plexus container are available

[Abel Muiño Vizcaino]

Just for reference... anything inside an *.internal.* is not meant to be used outside the plug-in. That's why you receive that message.

--

Abel Muiño Vizcaino - http://ramblingabout.wordpress.com

[Jake Pezaro]

i've got a working draft of the dependency browser based on the
maven-dependency-tree plugin (good advice, thanks carlos), it is still
a little rough around the edges but the basic workings are good.

issues outstanding
- colours and selection behaviour aren't quite right yet
- need to add the scope filtering
- code formatting does not match rest of project
- not very well tested yet

questions:
- should i submit it as is while i finish off the last bits or should
i wait until it is all done?
- should i submit it as a svn patch (in which case where to) or would
you rather i committed it myself (i would need commit access)?
- i have added the maven-dependency-tree 1.1 jar to the embedder
manually (added jar to /lib and altered manifest). is this ok? i
wasn't able to get the maven bundle plugin to work correctly (it
re-created the manifest differently to the copy in svn).
- it is currently using some java 1.5 features but they could be
removed easily. are we aiming for compatibility with a particular
java version?

[Abel Muiño Vizcaino]

Hi Jake!

issues outstanding

- colours and selection behaviour aren't quite right yet

- need to add the scope filtering

I don't any see major issue in your list, so I don't think we need to wait. We could continue improving the dependency browser incrementally. 

- code formatting does not match rest of project

We are using maven conventions. You can download the settings from http://tinyurl.com/2mwe94 and import the file in Eclipse settings (Java > Code Style > Formatter > Import).

 Personally, I have configured eclipse to format and organize imports (and other cleanups) my code on each save (Java > Editor > Save Actions) so I don't have to be concerned with those issues.

- not very well tested yet

Testing is important, but if you've done some testing and a're confident that it works we can iron the details with some help from the community.

- should i submit it as is while i finish off the last bits or should

i wait until it is all done?

+1 to submitting it now.

- should i submit it as a svn patch (in which case where to) or would

you rather i committed it myself (i would need commit access)?

I think you should submit a patch so we can take a look first. For example, it is important that the code contains the EPL header (see the wiki, near the end of http://tinyurl.com/2zszsl). The best place for submitting the patch is the Issue tracker (Issue 114 -> http://tinyurl.com/2xla2g)

As for becoming a committer, if you plan to stay around I don't think it would be a problem... you're the best person for maintaining your code :-)

- i have added the maven-dependency-tree 1.1 jar to the embedder

manually (added jar to /lib and altered manifest).  is this ok?  i

wasn't able to get the maven bundle plugin to work correctly (it

re-created the manifest differently to the copy in svn).

I would add the maven-dependency-tree jar in a different plug-in if possible, but probably adding it to the embedder is easier. Currently the embedder plug-in can't be built only with maven (it requires some manual tweaks), so it is ok that you modified it.

- it is currently using some java 1.5 features but they could be

removed easily.  are we aiming for compatibility with a particular

java version?

Q4e is targeted at Eclipse 3.3 and Java 5, so no problem. In fact q4e makes extensive use of generics and the new for syntax.

--

Abel Muiño Vizcaino - http://ramblingabout.wordpress.com

<dependency-browser-1.JPG>

[Jake Pezaro]

i'll post the patch to issues log when it's ready. the other stuff
(formatting, 1.5, EPL headers) no problem.

the code from maven-dependency-tree jar must be executed inside the
plexus container (the code that executes it is in core..EclipseMaven).
as far as I can see (which is not very far in this case as my
understanding of OSGI is basic at best) there are 4 options for where
to put this:

1 - the embedder bundle (it's current location). dependency path:
embedder > core > dependency-browser
2 - the core bundle (i opted not to put it here as there are no other
jars in this bundle) dependency path: core > dependency-browser
3 - a new bundle (can't be the dependency-browser bundle as it depends
on core) which then becomes a dependency of the core bundle (seems a
bit complicated for no benefit). dependency path: embedder &
new-bundle > core > dependency-browser
4 - bundle the viewer together with the jar and allow it to start it's
own plexus container (probably a bad idea as it replicates core
functionality and creates a separate plexus container to boot)
dependency path: embedder > dependency-browser.

there is one other issue that i forgot to mention - initialisation of
the viewer takes place on the UI thread, which is not a satisfactory
long term solution as it will definitely be slow for large projects.
i will make this a background task but i need to get to grips with
eclipse job control first, so it might take a little while. does this
need to be done before i submit the initial patch?

dynamic reaction to changes in the workspace & following focus is not
implemented. i would rather leave this out for now as it would
require re-resolving the entire tree every time which takes upwards of
3 seconds (for a small project). we can talk about this more once the
other functionality is implemented and stable.

do we want to keep the old zest-based graph? it looks nice but i
never found it useful for solving real world problems, however if we
do want to keep it then where do we want to put the new functionality?
more options:

1 - delete the old zest display from the plugin.
2 - separate plugin. easiest code solution, but confusing as we would
then have 2 very similarly named plugins doing the same thing in a
different way
3 - two-in-one plugin (tabbed panes or something). means more code

[Abel Muiño Vizcaino]

I can't say what's best since I don't have enough knowledge of the code, so here are some pointers that you can investigate.

Running code into the embedder: you can register any plug-in as a "buddy" of another plug-in (this is not OSGi, is an Eclipse-only feature). This is achieved by a special line in the plug-in manifest (Eclipse-RegisterBuddy: org.apache.maven.embedder).

Long running tasks in the UI thread: IRunnableWithProgress is one option: http://tinyurl.com/22jb7a

Dynamic selection: We can allow a link button that the user can enable (in which case the view is updated on selection changes) or disable. But this can be done later. This recent article might give some detail: http://dev.eclipse.org/blogs/wayne/2008/01/14/the-selection-service/  (is been on my to-read pile for too long...)

I would keep the Zest based viewer and have your code in a *-dependency-analysis plug-in (or similar name). The zest viewer is mostly eye candy and is already an optional feature (it can be skipped from the install)

Thanks for your work!

--

Abel Muiño Vizcaino - http://ramblingabout.wordpress.com

[Carlos Sanchez]

I'd like to see it in a new plugin, using it's own plexus container
for encapsulation. It shouldn't be too heavy. But let's see what do
you have and we can work later the details

[codesite...@google.com]

Issue 144: Dependency Tree Viewer
http://code.google.com/p/q4e/issues/detail?id=144

Comment #7 by jake.pezaro:
have attached a patch containing the dependency analysis code. it is a zipfile
containing all the new and changed files.

Attachments:
dependency-analyser.tar.gz 79.5 KB

[Jake Pezaro]

i've uploaded a patch to
http://code.google.com/p/q4e/issues/attachment?aid=1670416264429804863&name=dependency-analyser.tar.gz

the long running dependency resolution task is now a background job.

dependencies are stilll in the embedder for now. i've been thinking
about how this can be moved into a separate plugin.

option 1 - separate instance of the embedder
option 2 - register the plugin as a buddy of core and expose the
MavenEmbedder.lookup() method in EclipseMaven ie public Object
lookup(String role, String hint).
option 3 - (kind of like option 2) register the plugin as a buddy of
core and expose the a method in EclipseMaven that returns a collection
of objects that the dependency analyser can then cast. the key
difference from 2) is that this method hard codes the roles that it
needs to lookup.

still to do
a) scope filtering
b) dynamic selection
c) selection colors are not right (SWT doesn't make it easy to change
appearance of the selected row in tables & trees) but this is a minor
issue
d) icons are stolen from the dependency viewer
e) multi-module projects don't work

let me know if, how and when you want to put this stuff into the codebase.

[Abel Muiño Vizcaino]

It would be much easier to review if it was a diff generated by svn:

From q4e's root folder, type something like:

svn diff -r HEAD > patch-144.txt

Thanks!

--

Abel Muiño Vizcaino - http://ramblingabout.wordpress.com

[codesite...@google.com]

Issue 144: Dependency Tree Viewer
http://code.google.com/p/q4e/issues/detail?id=144

Comment #8 by jake.pezaro:
(No comment was entered for this change.)

Attachments:
patch-144.txt 147 KB

[Jake Pezaro]

ok. it's submitted as a patch (svn diff -r HEAD plugins/ > patch-144.txt)

http://code.google.com/p/q4e/issues/attachment?aid=-3979520537390439624&name=patch-144.txt

[Abel Muiño]

Thanks! I will review the patch ASAP.

2008/1/26, Jake Pezaro <jake....@gmail.com>:

[Abel Muiño Vizcaino]

I've already merged the patch locally and I'm starting to play around with it. It all looks really good so far.

I have the following changes in mind:

  - To follow the conventions on other plug-ins, I'll move the code to src/ and icons/ (instead of src/main/java and src/main/resources).

  - Also, since the maven embedder is generated semi-automatically, I'll move the new jar to a new plug-in.

  - I would like to split the new plug-in in two, a "core" and a "ui". Hopefully we can reuse the "core" in some other places.

  - Use an eclipse view instead of a new Shell. In the middle term, maybe 3 separate views and a new perspective (that would allow the user to organize the three panels as she wishes).

BTW, it is quite fast on my computer... collecting the data was only a matter of seconds in the projects I checked (some archiva modules and a new project generated from the appfuse-core archetype). 

Great work Jake!

--

Abel Muiño Vizcaino - http://ramblingabout.wordpress.com

[codesite...@google.com]

Issue 144: Dependency Tree Viewer
http://code.google.com/p/q4e/issues/detail?id=144

Comment #9 by carlossg:
we are in the process of committing the code, more news soon

[codesite...@google.com]

Issue 144: Dependency Tree Viewer
http://code.google.com/p/q4e/issues/detail?id=144

Comment #10 by amuino:
Merged in revision 927.

Issue attribute updates:
Status: Fixed
Owner: amuino
Labels: Milestone-0.5.0

[Jake Pezaro]

i've uploaded a patch with a couple of minor bug fixes

http://code.google.com/p/q4e/issues/attachment?aid=-1671801451596983363&name=patch-144-1.txt

can you let me know when it's merged

[codesite...@google.com]

Issue 144: Dependency Tree Viewer
http://code.google.com/p/q4e/issues/detail?id=144

Comment #12 by carlossg:
done, thanks

[codesite...@google.com]

Issue 144: Dependency Tree Viewer
http://code.google.com/p/q4e/issues/detail?id=144

Comment #13 by jake.pezaro:
fixed problems with selected items hiding the highlighting

[Jake Pezaro]

another patch with a small bug fix

http://code.google.com/p/q4e/issues/attachment?aid=-1376065294850393332&name=patch-144-2.txt

[codesite...@google.com]

Issue 144: Dependency Tree Viewer
http://code.google.com/p/q4e/issues/detail?id=144

Comment #14 by jake.pezaro:

fixed problems with selected items hiding the highlighting

Attachments:
patch-144-2.txt 5.3 KB

[Abel Muiño Vizcaino]

Applied.

I've also added a scheduling rule to the job to avoid conflicts with other maven jobs.

--

Abel Muiño Vizcaino - http://ramblingabout.wordpress.com

[Carlos Sanchez]

please create new issues for the patches. That issue is closed and
patches could be missed

[Jake Pezaro]

ok, will do