Encoding for generated sources in jenkins-core

[Rusi Filipov]

Hello Group,

I am planning to use Jenkins for a demonstration and now I am trying to play around with the sources. I must have somhow misconfigured Intellij because it starts giving me encoding problems with the generated classes and they don't compile any more:

src/main/resources/hudson/scheduler/Messages.properties (Intellij shows windows-1252):

CronTab.spread_load_evenly_by_using_rather_than_=Spread load evenly by using \u2018{0}\u2019 rather than \u2018{1}\u2019

target/generated-sources/localizer/hudson/scheduler/Messages.java (Intellij shows UTF-8):

     * Spread load evenly by using �{0}� rather than �{1}�

Intellij says: error: unmappable character for encoding UTF-8

Similar problem with the Jelly scripts:

src/main/resources/lib/layout/stopButton.jelly (Intellij shows UTF-8)

        Creates a clickable “Stop” button.

target/generated-sources/taglib-interface/lib/LayoutTagLib.java (Intellij shows UTF-8)

     *         Creates a clickable �Stop� button.

Intellij says: error: unmappable character for encoding UTF-8

I have set UTF-8 as the default project encoding in Intellij. 

Any ideas? 

[Hugh Perkins]

I checked on my own computer, and the quotation marks in src/main/resources/lib/layout/stopButton.jelly are not valid ASCII, but they are valid UTF-8.  They are these two characters:

http://www.fileformat.info/info/unicode/char/201c/index.htm

http://www.fileformat.info/info/unicode/char/201d/index.htm

[Hugh Perkins]

In the pom.xml at the root of hudson, it says the properties files are in iso-8859-1:

  <properties>

    <!-- *.html files are in UTF-8, and *.properties are in iso-8859-1, so this configuration is acturally incorrect, but this suppresses a warning from Maven, and as long as we don't do filtering we should be OK. -->

    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

[Rusi Filipov]

Hugh, thank you for pointing these out. 

As far as I remember properties files must be encoded in iso-8859-1, so if the rest of the sources are UTF-8 this should be OK. 

I guess the error was caused by me experimenting with different compiler settings and instructing the eclipse compiler inside Intellij to use UTF-8, which messed up with the annotation processor.

Besides that, what is the recommended way to configure the compilation in Intellij?

Do you enable "Use external build"?

Which compiler do you use - javac or eclipse?

Do you enable "Enable annotation processing"?

Greetings, 

Rusi

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/mLrAoiYH6bk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jesse Glick]

On Fri, Aug 2, 2013 at 9:28 AM, Rusi Filipov <rfil...@gmail.com> wrote:
> target/generated-sources/localizer/hudson/scheduler/Messages.java (Intellij shows UTF-8):
> * Spread load evenly by using �{0}� rather than �{1}�
>
> Intellij says: error: unmappable character for encoding UTF-8

Please verify whether the file is actually valid in UTF-8 encoding, or
if not, what encoding it is in. And what operating system do you use?
I am wondering if there is an issue with the platform default
encoding. You can check this on your system with:

jrunscript -e 'println(java.lang.System.getProperty("file.encoding"))'

org.jvnet.localizer.Generator calls com.sun.codemodel.CodeWriter which
uses the stupid OutputStreamWriter constructor that uses platform
default encoding, but then wraps it in a UnicodeEscapeWriter which is
supposed to generate \uXXXX escapes for any characters unmappable in
that encoding. Sounds like that escaping is not working for you. Even
if it did, this would be a bug in GeneratorMojo. For example, if
Messages.properties specified characters in U+0080–U+00FF, and the
system encoding was ISO-8859-1, these would be written to
Messages.java as single bytes which would be invalid in the UTF-8
encoding specified in the root POM (since generated sources are
obliged to use the specified encoding just like main sources).

@Kohsuke maybe we could retire this questionable Maven plugin and just
use a JSR 269 processor? That would pick up the right encoding for
Java sources without hassle, and would have some ancillary benefits;
e.g. 269-aware IDEs may immediately offer code completion on
Messages.java methods as soon as you save changes to
Messages.properties, without requiring a Maven build. It would be
slightly tricky to enumerate all the available Messages.properties
files, since JSR 269 does not offer this capability, but you could
work around that by checking explicitly for Messages.properties
resource in each package for which a root element was encountered in
the round environment.

[Rusi Filipov]

Thank you for the insight Jesse.

Please verify whether the file is actually valid in UTF-8 encoding, or

if not, what encoding it is in. And what operating system do you use?
I am wondering if there is an issue with the platform default
encoding.

My laptop has the following Environment:

- Windows 7, English

- JDK 1.7.0_25

- Maven 3.0.5

- Git Bash 1.8.3

- Intellij IDEA 12.1.4

jrunscript -e 'println(java.lang.System.getProperty("file.encoding"))'

This outputs Cp1252.

 

I have been digging deeper into this. When I compile the project in the Terminal I see this output:

fidel@ARULCO /d/ccd/workshop/jenkins/core (cleanup)

$ mvn clean compile

...

[INFO] --- build-helper-maven-plugin:1.7:add-source (add-source) @ jenkins-core ---

[INFO] Source directory: d:\ccd\workshop\jenkins\core\target\generated-sources\antlr added.

[INFO] Source directory: d:\ccd\workshop\jenkins\core\target\generated-sources\localizer added.

...

[INFO] --- maven-localizer-plugin:1.14:generate (default) @ jenkins-core ---

...

[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ jenkins-core ---

[INFO] Changes detected - recompiling the module!

[INFO] Compiling 916 source files to d:\ccd\workshop\jenkins\core\target\classes

[WARNING] /d:/ccd/workshop/jenkins/core/target/generated-sources/taglib-interface/lib/LayoutTagLib.java:[730,36] unmappable character for encoding UTF-8

...

[WARNING] /d:/ccd/workshop/jenkins/core/target/generated-sources/localizer/hudson/scheduler/Messages.java:[33,36] unmappable character for encoding UTF-8

...

The compiler warnings about unmappable characters in LayoutTagLib.java and in Messages.java indicate an encoding issue. When I open these two files in Notepad++ he detects them as UTF-8 without BOM.

But when I try to generate the sources in the Terminal (mvn clean generate-sources) and launch a subsequent build of the module inside Intellij (Make Project) it results in compile errors:

D:\ccd\workshop\jenkins\core\target\generated-sources\localizer\hudson\scheduler\Messages.java

error: unmappable character for encoding UTF-8

@Kohsuke maybe we could retire this questionable Maven plugin and just
use a JSR 269 processor? That would pick up the right encoding for
Java sources without hassle, and would have some ancillary benefits;
e.g. 269-aware IDEs may immediately offer code completion on
Messages.java methods as soon as you save changes to
Messages.properties, without requiring a Maven build. It would be
slightly tricky to enumerate all the available Messages.properties
files, since JSR 269 does not offer this capability, but you could
work around that by checking explicitly for Messages.properties
resource in each package for which a root element was encountered in
the round environment.

This seems a questionable plugin indeed. I could not find this plugin's home page and mvn localizer:help produced an error message saying he does not know the 'help' goal, which regular Maven plugins should support. Furthermore, I could not trigger its execution in Intellij during the 'generate-sources' phase, even when I set this explicitly in the plugin's configuration in the POM. So this basically makes "Rebuild Project" inside Intellij impossible and forces the developer to use both Terminal and IDE for this basic task. But even with a Terminal, the encoding issues from above emerge so I cannot compile the module. 

Not as good as it should be, but I will try to find a workaround for this... 

[Rusi Filipov]

As a workaround I replaced the unicode quotes  by regular quotes in Messages.properties and Messages_ja.properties (patch for core is attached).

[Baptiste MATHUS]

Hi,
FWIW, .properties files are by definition encoded (and parsed as) in iso-8859-1 (see http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html). The platform encoding shouldn't have any impact, if so then this might be a bug somewhere.

Cheers

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

[Jesse Glick]

OK, finally managed to reproduce locally (on Linux) using

MAVEN_OPTS=-Dfile.encoding=Cp1252 mvn -DskipTests -f core/pom.xml clean compile

My earlier suspicion about the bug in the Localizer plugin seems to be
correct, though I thought it would not apply here since I was thinking
about ISO-8859-1, which does not encode ‘’ quotes. Cp1252 does however
encode these characters as 0x91/0x92, meaning Windows users would be
hit by the bug. Looking for a fix.

[Jesse Glick]

Fixed the Localizer problem in trunk sources; needed a newer version
of that plugin.. There is still a problem with
core/target/generated-sources/taglib-interface/lib/LayoutTagLib.java
which will only be fixable when maven-hpi-plugin 1.97 is released (I
just committed a fix to that plugin’s sources).

In the meantime, I think your workaround for all such issues is likely to be:

C:\> set MAVEN_OPTS=-Dfile.encoding=UTF-8

[Rusi Filipov]

Thank you for the effort to fix this Jesse! Meanwhile, after working around the encoding issue I found a way to rebuild the project with the Terminal and Intellij:

1. mvn clean process-resources

2. Switch to Intellij and refresh

3. Eventually Reimport the Maven Project in order to detect the generated source folders

4. Make the Project (use Compiler Settings "javac" and "Use External Build")

Best Regards

Rusi

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/mLrAoiYH6bk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.

[Kohsuke Kawaguchi]

30ca0b818e0ae07c877dc63740f073c9782d79a4 added some .idea/*.xml files
that would get your IDEA set up correctly.

> <mailto:jenkinsci-dev%2Bunsu...@googlegroups.com>.

> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email

> to jenkinsci-de...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
Try Jenkins Enterprise, our professional version of Jenkins

[Jesse Glick]

On Mon, Aug 5, 2013 at 10:57 AM, Jesse Glick <jgl...@cloudbees.com> wrote:
> There is still a problem with
> core/target/generated-sources/taglib-interface/lib/LayoutTagLib.java
> which will only be fixable when maven-hpi-plugin 1.97 is released

That should now be fixed in trunk sources too.