GWT build requirements and packaging for Fedora

[jlaska]

Greetings,

I previously attempted to post this to the general 'google-web-
toolkit' group. However, I thought this group might be a more
appropriate setting.

I am looking into packaging GWT for Fedora. According to the Fedora
packaging guidelines, all packages must not include any bundled
libraries or JAR files [1]. As a result, I have attempted to identify
the JAR files included in GWT and whether they are already packaged in
Fedora.

So far, the process has identified many build dependencies that are
already packaged in Fedora or JPackage (that's good news for me).
However, there remain a few questions I could use guidance from the
GWT team on. Those include ...

= Multiple Versions =

There are several instances where multiple versions of a JAR file are
included in GWT. For example, the full list of multiple JAR file
versions I've observed:

* emma.jar and emma_ant-2.0.5312.jar
* xml-apis-2.0.2.jar and xerces-2_9_1/xml-apis.jar
* tomcat/servlet-api-2.4.jar and tomcat/servlet-api-2.5.jar
* jsp-api-2.0.jar and jsp-api-2.1.jar
* ant-launcher-1.6.5.jar and ant-launcher-1.7.1.jar
* nekohtml-1.9.12.jar and nekohtml-1.9.13.jar
* jdt-3.1.1.jar, jdt-3.1.1.jar and jdt-3.1.1.jar
* ant-1.6.1.jar, ant-1.6.5.jar and ant-1.7.1.jar
* ant-junit-1.6.5.jar and ant-junit-1.7.1.jar
* htmlunit-core-js-2.5.jar and htmlunit-core-js-r5070.jar

The question I have ... is it safe to assume only the newest version
of that JAR file is required, or are multiple JAR versions truly
required at build time?

= Uncertain dependencies =

There are several JAR files where I'm not able to determine their
nature as build requirements. That complete list is available at
https://fedoraproject.org/wiki/User:Jlaska/gwt#Status_uncertain. Is
it documented, or can someone help identify how/whether the following
JAR files are used during the build process?

* selenium*
* htmlunit*
* protobuf-2.2.0.jar
* swing-worker-1.1.jar
* javaswf-20050404.jar
* servlets-common-1.0.jar
* naming-common-1.0.jar
* gwt-dev-modified.jar
* gwt-user-modified.jar

Any assistance folks can offer in answering these 2 points is much
appreciated.

Thanks,
James

[1] http://fedoraproject.org/wiki/Packaging:Java#Pre-built_JAR_files_.2F_Other_bundled_software

[John Tamplin]

On Wed, Feb 3, 2010 at 8:25 AM, jlaska <james...@gmail.com> wrote:

I am looking into packaging GWT for Fedora.  According to the Fedora
packaging guidelines, all packages must not include any bundled
libraries or JAR files [1].  As a result, I have attempted to identify
the JAR files included in GWT and whether they are already packaged in
Fedora.

So far, the process has identified many build dependencies that are
already packaged in Fedora or JPackage (that's good news for me).
However, there remain a few questions I could use guidance from the
GWT team on.  Those include ...

Great!  I run Fedora on my home system myself, and I think it would be great if people could just run yum install gwt and get everything they need setup and updated.

 

There are several instances where multiple versions of a JAR file are
included in GWT.  For example, the full list of multiple JAR file
versions I've observed:

 * emma.jar and emma_ant-2.0.5312.jar
 * xml-apis-2.0.2.jar and xerces-2_9_1/xml-apis.jar
 * tomcat/servlet-api-2.4.jar and tomcat/servlet-api-2.5.jar
 * jsp-api-2.0.jar and jsp-api-2.1.jar
 * ant-launcher-1.6.5.jar and ant-launcher-1.7.1.jar
 * nekohtml-1.9.12.jar and nekohtml-1.9.13.jar
 * jdt-3.1.1.jar, jdt-3.1.1.jar and jdt-3.1.1.jar
 * ant-1.6.1.jar, ant-1.6.5.jar and ant-1.7.1.jar
 * ant-junit-1.6.5.jar and ant-junit-1.7.1.jar
 * htmlunit-core-js-2.5.jar and htmlunit-core-js-r5070.jar

The question I have ... is it safe to assume only the newest version
of that JAR file is required, or are multiple JAR versions truly
required at build time?

 

The tools directory is add-only, since it must be usable with any version of GWT.  So, when we upgrade to a new version, the old one stays around in case you are building an old version of GWT against it.

A few issues you are going to run into:

• some of the libraries have been modified (they will be documented as such), so it probably isn't safe to replace them with standard ones
• we don't generally upgrade dependencies unless there is something broken in the one we have, for the way we use it.  This will cause problems if you just rely on the system version of jetty, as an example, since there are API changes to later versions that would break GWT.  That would mean the GWT dependency would have to be to an older version, and you may find that using a system version pins a bunch of software at an old version.
• ant 1.7.0 has problems building GWT -- earlier or later versions are fine

 

= Uncertain dependencies =

There are several JAR files where I'm not able to determine their
nature as build requirements.  That complete list is available at
https://fedoraproject.org/wiki/User:Jlaska/gwt#Status_uncertain.  Is
it documented, or can someone help identify how/whether the following
JAR files are used during the build process?

 * selenium*

Selenium is used in RunStyleSelenium directly and GWT, and indirectly for people wanting to run a Selenium-RC server the tests can run against

 

 * htmlunit*

HtmlUnit is used as the default way of running tests, since using a real browser requires some level of configuration on the part of the user.

 

 * protobuf-2.2.0.jar

This is used for the RemoteUI interface to the Google Plugin for Eclipse.

 

 * swing-worker-1.1.jar

This is no longer needed, so if you are only supporting GWT 2.0+ you can leave that out.

 

 * javaswf-20050404.jar

I think this is used for audio resources, but I am not positive.

 

 * servlets-common-1.0.jar
 * naming-common-1.0.jar

I don't know about these.

 

 * gwt-dev-modified.jar
 * gwt-user-modified.jar

These are used by the API checker for comparing the APIs of different versions of GWT, to make sure we don't make breaking changes inadvertently.  It probably isn't needed for a package intended to be used by a developer using GWT rather than working on GWT. 

--
John A. Tamplin
Software Engineer (GWT), Google

[James Laska]

Greetings John,

Just the feedback I was in need of, thanks for taking the time to respond!

On Thu, Feb 4, 2010 at 8:05 PM, John Tamplin <j...@google.com> wrote:
> On Wed, Feb 3, 2010 at 8:25 AM, jlaska <james...@gmail.com> wrote:
>>
>> I am looking into packaging GWT for Fedora.  According to the Fedora
>> packaging guidelines, all packages must not include any bundled
>> libraries or JAR files [1].  As a result, I have attempted to identify
>> the JAR files included in GWT and whether they are already packaged in
>> Fedora.
>>
>> So far, the process has identified many build dependencies that are
>> already packaged in Fedora or JPackage (that's good news for me).
>> However, there remain a few questions I could use guidance from the
>> GWT team on.  Those include ...
>
> Great!  I run Fedora on my home system myself, and I think it would be great
> if people could just run yum install gwt and get everything they need setup
> and updated.

Definitely! There are quite a few dependencies for GWT (as I'm sure
you know), so I wouldn't say no to anyone interested in helping
package those dependencies. :)

>>
>> There are several instances where multiple versions of a JAR file are
>> included in GWT.  For example, the full list of multiple JAR file
>> versions I've observed:
>>
>>  * emma.jar and emma_ant-2.0.5312.jar
>>  * xml-apis-2.0.2.jar and xerces-2_9_1/xml-apis.jar
>>  * tomcat/servlet-api-2.4.jar and tomcat/servlet-api-2.5.jar
>>  * jsp-api-2.0.jar and jsp-api-2.1.jar
>>  * ant-launcher-1.6.5.jar and ant-launcher-1.7.1.jar
>>  * nekohtml-1.9.12.jar and nekohtml-1.9.13.jar
>>  * jdt-3.1.1.jar, jdt-3.1.1.jar and jdt-3.1.1.jar
>>  * ant-1.6.1.jar, ant-1.6.5.jar and ant-1.7.1.jar
>>  * ant-junit-1.6.5.jar and ant-junit-1.7.1.jar
>>  * htmlunit-core-js-2.5.jar and htmlunit-core-js-r5070.jar
>>
>> The question I have ... is it safe to assume only the newest version
>> of that JAR file is required, or are multiple JAR versions truly
>> required at build time?
>
>
> The tools directory is add-only, since it must be usable with any version of
> GWT.  So, when we upgrade to a new version, the old one stays around in case
> you are building an old version of GWT against it.

I see, that makes sense.

> A few issues you are going to run into:
>
> some of the libraries have been modified (they will be documented as such),
> so it probably isn't safe to replace them with standard ones

You noted that the modified libraries are documented, I gather this is
in the README file in the same directory? Looking in 'tools/lib' for
README files I see the following special handling instructions:

* gwt-2.0.0/tools/lib/protobuf/protobuf-2.2.0/README
* gwt-2.0.0/tools/lib/htmlunit/README

Are there others to consider?

Have the local library modifications been sent/accepted in the upstream project?

> we don't generally upgrade dependencies unless there is something broken in
> the one we have, for the way we use it.  This will cause problems if you
> just rely on the system version of jetty, as an example, since there are API
> changes to later versions that would break GWT.  That would mean the GWT
> dependency would have to be to an older version, and you may find that using
> a system version pins a bunch of software at an old version.

Oh good to know. This shouldn't be too much trouble. I can have GWT
%buildrequire a specific version of a library. If newer versions of
that library are available, I should be able to provide the older
library as a compatibility.

Perhaps the same question as earler, is there a way to tell when GWT
requires a *specific* library version? Or should I default to using
the latest, and see where the build fails?

> ant 1.7.0 has problems building GWT -- earlier or later versions are fine

Noted.

>> = Uncertain dependencies =
>>
>> There are several JAR files where I'm not able to determine their
>> nature as build requirements.  That complete list is available at
>> https://fedoraproject.org/wiki/User:Jlaska/gwt#Status_uncertain.  Is
>> it documented, or can someone help identify how/whether the following
>> JAR files are used during the build process?
>>
>>  * selenium*
>
> Selenium is used in RunStyleSelenium directly and GWT, and indirectly for
> people wanting to run a Selenium-RC server the tests can run against
>
>>
>>  * htmlunit*
>
> HtmlUnit is used as the default way of running tests, since using a real
> browser requires some level of configuration on the part of the user.
>
>>
>>  * protobuf-2.2.0.jar
>
> This is used for the RemoteUI interface to the Google Plugin for Eclipse.
>
>>
>>  * swing-worker-1.1.jar
>
> This is no longer needed, so if you are only supporting GWT 2.0+ you can
> leave that out.

Great, one less thing to worry about :)

>>  * javaswf-20050404.jar
>
> I think this is used for audio resources, but I am not positive.
>
>>
>>  * servlets-common-1.0.jar
>>  * naming-common-1.0.jar
>
> I don't know about these.
>
>>
>>  * gwt-dev-modified.jar
>>  * gwt-user-modified.jar
>
> These are used by the API checker for comparing the APIs of different
> versions of GWT, to make sure we don't make breaking changes inadvertently.
>  It probably isn't needed for a package intended to be used by a developer
> using GWT rather than working on GWT.

Thanks again,
James

[John Tamplin]

On Fri, Feb 5, 2010 at 10:05 AM, James Laska <james...@gmail.com> wrote:

You noted that the modified libraries are documented, I gather this is

in the README file in the same directory?  Looking in 'tools/lib' for
README files I see the following special handling instructions:

 * gwt-2.0.0/tools/lib/protobuf/protobuf-2.2.0/README
 * gwt-2.0.0/tools/lib/htmlunit/README

Are there others to consider?

Have the local library modifications been sent/accepted in the upstream project?

I know they have in the case of HtmlUnit, though I have no idea if they have been accepted.  Amit Manjhi (amitm...@google.com) would be the one to talk to about HtmlUnit changes.  I don't know about others.

 

Oh good to know.  This shouldn't be too much trouble.  I can have GWT

%buildrequire a specific version of a library.  If newer versions of
that library are available, I should be able to provide the older
library as a compatibility.

Perhaps the same question as earler, is there a way to tell when GWT
requires a *specific* library version?  Or should I default to using
the latest, and see where the build fails?

Hopefully our tests are sufficient that an incompatible version would fail tests, so I would try the newest version first.

The build files would likely need some hacking -- right now, it expects to find everything in tools under ../tools or $(GWT_TOOLS), not individually located.  You could have the spec file apply a patch against it, or if the changes could be done such that the default behavior worked as it does now but supplying -D flags to ant overrode the locations I think we could make those changes directly in GWT.

[James Laska]

On Fri, Feb 5, 2010 at 10:23 AM, John Tamplin <j...@google.com> wrote:
> On Fri, Feb 5, 2010 at 10:05 AM, James Laska <james...@gmail.com> wrote:
>>
>> You noted that the modified libraries are documented, I gather this is
>> in the README file in the same directory?  Looking in 'tools/lib' for
>> README files I see the following special handling instructions:
>>
>>  * gwt-2.0.0/tools/lib/protobuf/protobuf-2.2.0/README
>>  * gwt-2.0.0/tools/lib/htmlunit/README
>>
>> Are there others to consider?
>>
>> Have the local library modifications been sent/accepted in the upstream
>> project?
>
> I know they have in the case of HtmlUnit, though I have no idea if they have
> been accepted.  Amit Manjhi (amitm...@google.com) would be the one to talk
> to about HtmlUnit changes.  I don't know about others.
>
>>
>> Oh good to know.  This shouldn't be too much trouble.  I can have GWT
>> %buildrequire a specific version of a library.  If newer versions of
>> that library are available, I should be able to provide the older
>> library as a compatibility.
>>
>> Perhaps the same question as earler, is there a way to tell when GWT
>> requires a *specific* library version?  Or should I default to using
>> the latest, and see where the build fails?
>
> Hopefully our tests are sufficient that an incompatible version would fail
> tests, so I would try the newest version first.

Great, I'll proceed that way first.

> The build files would likely need some hacking -- right now, it expects to
> find everything in tools under ../tools or $(GWT_TOOLS), not individually
> located.  You could have the spec file apply a patch against it, or if the
> changes could be done such that the default behavior worked as it does now
> but supplying -D flags to ant overrode the locations I think we could make
> those changes directly in GWT.

The strategy I've been following, as recommended by other Fedora java
packaging gurus, is to add logic during the %prep step in the spec
file that removes bundled JAR files and replaces them with symlinks to
the system versions. This seems to be progressing well so far, and
requires no downstream patching of GWT.

For more details ...
http://fedoraproject.org/wiki/Packaging:Java#Pre-built_JAR_files_.2F_Other_bundled_software

Thanks,
James

[James Laska]

Greetings,

Apologies for the long delay (and the top-post). I've made slow, but
steady progress building a Fedora gwt package (thanks to many others
in the Fedora community who understand Java and Java packaging).

As a recap, the Fedora packaging policy prohibits bundling software in
RPM's. As a result, I have been adjusting the gwt RPM spec file to
replace all bundled .jar files with symlinks to the installed system
versions prior to the build. This has been going well, and I'm now
attempting to build gwt from source using the symlinked jars.

Presently, building gwt from source fails due to several jdt issues (see below).

compiler.standalone:
[mkdir] Created dir: /builddir/build/BUILD/gwt-2.0.3/build/out/dev/bin
[gwt.javac] Compiling 5 source files to
/builddir/build/BUILD/gwt-2.0.3/build/out/dev/bin
[gwt.javac] Compiling 673 source files to
/builddir/build/BUILD/gwt-2.0.3/build/out/dev/bin
[gwt.javac] /builddir/build/BUILD/gwt-2.0.3/dev/core/src/com/google/gwt/dev/javac/JdtCompiler.java:312:
warningThreshold has protected access in
org.eclipse.jdt.internal.compiler.impl.CompilerOptions
[gwt.javac] options.warningThreshold = 0;
[gwt.javac] ^
[gwt.javac] /builddir/build/BUILD/gwt-2.0.3/dev/core/src/com/google/gwt/dev/javac/JdtCompiler.java:312:
incompatible types
[gwt.javac] found : int
[gwt.javac] required: org.eclipse.jdt.internal.compiler.impl.IrritantSet
[gwt.javac] options.warningThreshold = 0;
[gwt.javac] ^
[gwt.javac] /builddir/build/BUILD/gwt-2.0.3/dev/core/src/com/google/gwt/dev/javac/JdtCompiler.java:343:
incompatible types
[gwt.javac] found : org.eclipse.jdt.internal.compiler.lookup.TypeBinding
[gwt.javac] required: org.eclipse.jdt.internal.compiler.lookup.ReferenceBinding
[gwt.javac] type = BinaryTypeBinding.resolveType(type,
lookupEnvironment, true);
[gwt.javac] ^
[gwt.javac] Note: Some input files use or override a deprecated API.
[gwt.javac] Note: Recompile with -Xlint:deprecation for details.
[gwt.javac] Note: Some input files use unchecked or unsafe operations.
[gwt.javac] Note: Recompile with -Xlint:unchecked for details.
[gwt.javac] 3 errors

Based on the above failures, I could use some guidance/clarification.

1) I see use of org.eclipse.jdt.internal.*, my understanding is that
these internal objects are not guaranteed to be available in future
versions of jdt. I gather there are missing API's which required
using the internal objects? Are there any plans to remove use of
internal jdt objects?

2) Fedora currently ships jdt-3.6. Are there plans to update gwt to
support jdt-3.6?

Many thanks!
James

[John Tamplin]

On Thu, Jul 22, 2010 at 11:57 AM, James Laska <james...@gmail.com> wrote:

Based on the above failures, I could use some guidance/clarification.

1) I see use of org.eclipse.jdt.internal.*, my understanding is that
these internal objects are not guaranteed to be available in future
versions of jdt.   I gather there are missing API's which required
using the internal objects?  Are there any plans to remove use of
internal jdt objects?

No, as we have to have access at a lower level, since GWT doesn't primarily use the bytecode of the compilation but rather the symbol tables and AST.

 

2) Fedora currently ships jdt-3.6.  Are there plans to update gwt to
support jdt-3.6?

We have generally upgraded JDT only when we had to for some other reason.  Aside from the internal APIs issue above (and the fact that they change), the JDT integration has been the source of a number of subtle bugs and it requires significant engineering effort to safely upgrade.

I suggest one of the following:

• package a gwt-jdt that has the specific version used by GWT and make that a dependency
• incorporate this version of JDT into the GWT package, perhaps renaming the classes using jarjar/etc to avoid conflicts
• have GWT depend on the specific version of JDT, though that might result in people being unable to use other packages that need later versions of JDT