Started writing a new Xtend based code generator for jOOQ 3.0

[digulla]

Hi,

I've started to write a new Xtend-based code generator for jOOQ 3.0. The main focus here is to simplify the code generator and to make it easy to reuse.

For my test cases, it would be great if I had a "MockDatabase" where I can define schemas, tables and columns in a unit test. Does something like that already exist?

Regards,

A. Digulla

[Lukas Eder]

Hello,

> I've started to write a new Xtend-based code generator for jOOQ 3.0. The
> main focus here is to simplify the code generator and to make it easy to
> reuse.

Great news!

> For my test cases, it would be great if I had a "MockDatabase" where I can
> define schemas, tables and columns in a unit test. Does something like that
> already exist?

jOOQ and jOOQ-codegen are integration tested against various similar
databases, containing all the weird corner-cases that can happen in
real life database designs (naming collisions, etc). For instance
Oracle:
https://github.com/jOOQ/jOOQ/blob/master/jOOQ-test/src/org/jooq/test/oracle/create.sql
https://github.com/jOOQ/jOOQ/blob/master/jOOQ-test/src/org/jooq/test/oracle/reset.sql

However, there is no such thing as a MockDatabase. The simplest way
for you to get going quickly is to use H2 and create this schema here:

https://github.com/jOOQ/jOOQ/blob/master/jOOQ-test/src/org/jooq/test/h2/create.sql
https://github.com/jOOQ/jOOQ/blob/master/jOOQ-test/src/org/jooq/test/h2/reset.sql

That wouldn't be "mocking" a schema, but fast enough, I guess.

Some side-notes:

- Beware that I have deprecated master data types and that I'm
planning to remove that functionality in jOOQ 3.0:
https://github.com/jOOQ/jOOQ/issues/1740. jOOQ should have enough
conversion features, such that these kinds of data types can be
generated transparently in client code.
- Neither jOOQ-meta nor jOOQ-codegen are "stable" APIs. This means
that unlike the jOOQ core, I do not officially support them and keep
them backwards-compatible. For jOOQ 3.0, they can be heavily changed
if you find some issues to be disturbing. Specifically, their
inter-dependency is probably an annoying issue for a bigger
refactoring.

[digulla]

Am Freitag, 31. August 2012 09:57:57 UTC+2 schrieb Lukas Eder:

Great news!

Thanks. I think I'll have something to show by the middle of next week.

Do you have an Eclipse formatter config for me?

 

That wouldn't be "mocking" a schema, but fast enough, I guess.

I also use H2 but it does take 1-2s to start; with my mock database, I can run about 100 tests in that time :-)

Also, the mock database allows to define a lot more corner cases since you can extend an existing "schema" -> much more efficient.

 

- Beware that I have deprecated master data types and that I'm
planning to remove that functionality in jOOQ 3.0:
https://github.com/jOOQ/jOOQ/issues/1740. jOOQ should have enough
conversion features, such that these kinds of data types can be
generated transparently in client code.

Right now, I don't have big issues with meta; I would prefer if things like configuredSchemata would be elsewhere (so the AbstractDatabase would really be just an information pool about the database) and all mapping options would be in the config of the code generator. But it doesn't hurt me.

My new code will make it pretty easy to refactor that later when we have all those nice test cases :-)

- Neither jOOQ-meta nor jOOQ-codegen are "stable" APIs.

Thanks for the info, I'll keep that in mind.

Regards,

A. Digulla 

[Lukas Eder]

> Do you have an Eclipse formatter config for me?

Yes, all you need is checked in here:
https://github.com/jOOQ/jOOQ/tree/master/jOOQ-codegen/.settings

> I also use H2 but it does take 1-2s to start; with my mock database, I can
> run about 100 tests in that time :-)
>
> Also, the mock database allows to define a lot more corner cases since you
> can extend an existing "schema" -> much more efficient.

Yes, well if we start afresh, it's a good idea to create these things.
Currently, there are probably too many integration tests and not
enough unit tests

> Right now, I don't have big issues with meta; I would prefer if things like
> configuredSchemata would be elsewhere (so the AbstractDatabase would really
> be just an information pool about the database) and all mapping options
> would be in the config of the code generator. But it doesn't hurt me.

Yes, that was never cleanly separated

> My new code will make it pretty easy to refactor that later when we have all
> those nice test cases :-)

You're setting the expectations high! :-)

Cheers
Lukas

[Lukas Eder]

Hi

> first cent: please please seperate the generation code into two phases a)
> prepare completely all meta-information b) write the Javacode

Yes, this is one of the main goals of the source code generation
refactoring in jOOQ 3.0, as this has been requested time and again.

> second cent: i would like to have more flexibility. E.g. make it possible
> that i access the results of the db examination and write the results in my
> own XML file. Than let me write code to build the Factory,Schema, Table,
> etc. objects in memory (no code generation) by myself. What is it good for?
> It opens possibilies for the dynamic usage of jooq. D'accord, this is a
> niche but on the other hand, if you follow advise 1, you are almost there.

Would you mind elaborating on that idea? What would be your
requirements? And use cases?

Cheers
Lukas

[Lukas Eder]

> I used a custom generator for renaming generated objects as some of my DB
> tables have prefix and pluralization that were repercussed. However I was
> surprised that I couldn't find a nice and easy way to do this, also on the
> documentation side, there isn't much regarding the Definition object that
> gets passed to your overriden methods..

There would be a way using the generator strategy, but it isn't nicely
documented (or even well-designed)

> I thought an easier way to solve this issue(which is probably a common one)
> would be to use an XML file like this
>
> [...]

Yes, the intermediate XML step would also be my favourite. In fact,
the solution I'm looking for would involve two code-generators:

1. A generator to generate this XML from a physical database schema
2. A generator to generate Java classes from this XML

Users may then choose whether they really need step 1. Also, they may
use XSL transformation (or any other means) to transform the XML
meta-data into any other format, including - for instance -
Hibernate's hbm.xml format (or the other way round).

> (or perhaps a syntax closer to... mybatis-generator... don't misunderstand,
> no offense here, I am not pretending they do everything so much better, but
> a couple of ideas they had are perhaps worth a look, right?)

No offense taken. I had been exploring this topic earlier this year.
Ideally, I'd like to have an XML schema modelling the SQL standard
INFORMATION_SCHEMA:
http://stackoverflow.com/q/8184849/521799

But any other model may do as well, including that of mybatis.

One thing about your XML format suggestion, though: Modelling the
actual Java outcome through elements like <tableClass/>,
<tableStaticField/>, etc will be quite hard to maintain on the
jooq-generator side (maintaining this XML language, adding new
elements with new feature requests, integration testing all corner
cases, etc.). I prefer to have an XML format only containing database
meta-information, moving code generation "styling" elsewhere.

> I was even thinking about contributing such a functionality depending on my
> free time but it's probably not a good idea if you are going to redesign the
> generator API in v3.

Let's wait and see what Aaron produces with Xtend this week.

[digulla]

Just as a teaser, here is the configuration of the new code generator. I'm not a big fan of XML-based configurations, I prefer DSLs but the API is public, so writing an XML front-end for the config is just an issue of someone wanting to scratch this itch.

Here is the config:

        BluNamingStrategy namingStrategy = new BluNamingStrategy();

        namingStrategy

        .renameColumn( "BLU.ADM_LANGUAGE.ISO_CODE", "locale" )

        .renameColumn( "BLU.GBL_LOCALISED_TEXT.ISO_CODE", "locale" )

        .renameBooleanColumn( "BLU.ADM_LANGUAGE.IS_DEFAULT", "default" )

        ;

        generator.config()

        .dataSource( dataSource )

        .dialect( SQLDialect.H2 )

        .packageName( "com.avanon.blu.jooq.gen" )

        .rootFolder( new File( "tmp/gen2" ) )

        .schema( "PUBLIC", "BLU" )

        .includeAll()

        .namingStrategy( namingStrategy )

        .factory( new BluGeneratorFactory( generator.config() ) )

        .typeMapping( DateTime.class, DateTimeConverter.class )

            .columnPatterns(

                new ColumnPattern( "(VALID_FROM|VALID_TO)" ),

                new ColumnPattern( "ADM_USER", "(PASSWORD_EXPIRES|LAST_LOGIN|FIRST_FAILED_LOGIN|LAST_MODIFIED_DATE|DEACTIVATED_DATE)" ),

                new ColumnPattern( "GBL_LOCK", "LAST_UPDATE" )

            )

        .typeMapping( Boolean.class, BooleanConverter.class )

            .columnPatterns(

                new ColumnPattern( "ADM_LANGUAGE", "IS_DEFAULT" ),

                new ColumnPattern( "ADM_USER", "ACCOUNT_LOCKED" ),

                new ColumnPattern( "ADM_USER", "MANDATORY_NOTIFS_DISABLED" ),

                new ColumnPattern( "ADM_ORG_UNIT_REV", "IS_PROFIT_CENTER" )

            )

        .typeMapping( Locale.class, LocaleConverter.class )

            .columnPatterns(

                new ColumnPattern( "GBL_LOCALISED_TEXT", "ISO_CODE" ),

                new ColumnPattern( "ADM_LANGUAGE", "ISO_CODE" )

            )

        ;

* The naming strategy has support for renaming columns in Java code. So the column ISO_CODE will be mapped to setLocale()/getLocale() while IS_DEFAULT will be setDefault()/isDefault()

* The PUBLIC schema is renamed to BLU

* I include all tables of the schema

* Three custom types are defined which map columns to joda-time's DateTime, Boolean and java.util.Locale. Each type gets a list of column patterns where to apply it.

* I'm using my own Generator factory because I need custom code in my POJOs, Records and interfaces.

Config is the most complex step; the rest of the code is two lines (create the JavaGenerator and start it).

Regards,

A. Digulla

[Lukas Eder]

Hello Aaron,

> Just as a teaser, here is the configuration of the new code generator. I'm
> not a big fan of XML-based configurations, I prefer DSLs but the API is
> public, so writing an XML front-end for the config is just an issue of
> someone wanting to scratch this itch.

A DSL contribution is nice, too. If it works well, I can add the
XML-parts, which will be required to be able to run code generation
through Maven, for instance. That's just like what we have today,
where the XML configuration is loaded by GenerationTool or by the
Maven plugin, and passed on to the DefaultGenerator.

> Here is the config:

From what I've seen so far, most of the configuration does more or
less what the current configuration can do? Or am I missing something?
I do like the more explicit naming strategy though. There had been a
feature request for some time, asking for precisely that.

Some remarks:

> .renameBooleanColumn( "BLU.ADM_LANGUAGE.IS_DEFAULT", "default" )

That shouldn't be necessary. The boolean type information is available
on the ColumnDefinition. If you prefer isXXX() instead of getXXX(),
this should be placed elsewhere. In your case, it should be placed
into the BluNamingStrategy implementation, not the API

> .dialect( SQLDialect.H2 )

I'd prefer if the org.jooq.util.Database implementation is still
referenced rather than a SQLDialect, for two reasons

1. Users can override the Database if they want support for an older
version. E.g. the default SQLServerDatabase is tested against SQL
Server 2008. Someone wanting to generate code for SQL Server 2000
should be able to provide their own custom Database (I've just had
this feature request)
2. If we have an intermediate step (e.g. XML), the Database would have
to read its data from that XML file. This would then be
dialect-independent. Also the JDBCDatabase that I'm planning to add to
jooq-meta would be dialect-independent

> .schema( "PUBLIC", "BLU" )

Did you plan to keep multi-schema support?

Also, I wonder whether object renames should still be possible in two
places, i.e. naming strategy AND the overall schema reference. The
current generator strategy is a strategy for giving names to Java
objects (it would affect what you call the GeneratorFactory)

A new "database strategy" that would contain database object renames
(such as physical schema renames), and also the include() / exclude()
/ includeAll() methods, and maybe the type mapping.

> .factory( new BluGeneratorFactory( generator.config() ) )

I'm guessing that this will replace what is called DefaultGenerator
today, reading the meta data and generating Java classes?

Anyway, we should coordinate your goals. Your own personal goals seem
to diverge a little from the general jOOQ 3.0 roadmap.

Here's the jOOQ 3.0 roadmap:
1. There should be a clean API for jooq-meta. It should be completely
independent of the code generator.
2. The 1) jooq-meta API can be implemented in dialect-independent
ways, reading meta-information from non-JDBC data sources, such as
XML. This can be used to read non-jOOQ XML formats, such as
Hibernate's .hbm.xml
3. The jooq-meta API should have a "database strategy", which models
today's include/exclude features, database type mappings, database
object renames
4. The output produced by 3) "database strategy" is fed into a
"JavaGeneratorFactory" or into a "Generator". Today's DefaultGenerator
could be a sample implementation thereof
5. An new "XMLGeneratorFactory" could be created to use the feed from
3) "database strategy", to render an XML representation of the schema
meta-information. This could then again be fed into 2)
6. Today's "GeneratorStrategy", or what you call the "Naming Strategy"
could be called "JavaNamingStrategy", used to map database objects to
Java objects. It is specific to 4) "JavaGeneratorFactory". Today's
"Naming Strategy" is too low-level, more explicit features would be
better.

For me, the most important features to add are 2) (new data sources),
5) (new target formats) and maybe 6) (improved Java naming
strategies). For you, 6) seems to be the most important improvement,
as well as the fact that 4) can be re-implemented in your own way,
using Xtend.

If we can agree on the roadmap, I can try to devise an API for 1), 3),
4) and 6), and post it on this user group within the next 1-2 weeks.
You could then again challenge this API with your requirements

Cheers
Lukas

[digulla]

Am Montag, 3. September 2012 23:30:25 UTC+2 schrieb Lukas Eder:

Some background info: I needed a more powerful code generator now, so my 2nd gen code generator reuses most of the old API and code. This isn't a goal, it's pure necessity.

But since the new code generator is more flexible (and needs less lines of code) than the old one, it will help you when you start with refactoring the old API because less things will break / it will be easier to fix the broken parts.

> Just as a teaser, here is the configuration of the new code generator. I'm
> not a big fan of XML-based configurations, I prefer DSLs but the API is
> public, so writing an XML front-end for the config is just an issue of
> someone wanting to scratch this itch.

A DSL contribution is nice, too. If it works well, I can add the
XML-parts, which will be required to be able to run code generation
through Maven, for instance. That's just like what we have today,
where the XML configuration is loaded by GenerationTool or by the
Maven plugin, and passed on to the DefaultGenerator.

I'm now finished with the prototype. You can find the code here: https://bitbucket.org/digulla/jooq-codegen2

I used Mercurial because that was easier for me to set up. Let me know if you need a Git repo.

Overview of the features:

* Most types can be generated (table definition, interfaces, records and pojos)

* It's easy to write tests for the code generator. All output is written via an OutputWriter. Test override that with a MockOutputWriter that collects all output in a map (path -> content)

* 70% test coverage

* Code generators are written in Xtend

* Each output type gets their own generator to make it easy to extend the generated code

* Configuration uses a DSL

* There is a MockDatabase to help writing test cases (no need to actually create a database). It uses DSL to quickly define schemas, tables and columns.

* A pluggable strategy to create a serialVersionUID. I also created a new strategy which removed comments and white space before calculating the UID. I also made sure the UID is never negative to make Cut&paste easier.

I suggest to have a look at the existing test cases to get an idea how everything works together. There are quite a few TODOs left because I only implemented those features which I needed to create the code for my application. But near the end, I created a new generator in about 4 hours.

A comment about XML based configs. I need a good way to define configs in Java so I can create configs for unit tests. When creating custom code generators, it's very important to be able to run the generator and check the output against a "known good" baseline. Otherwise, it's too hard to see what effects a change has. So I end up with this setup:

1. A project which extends jOOQ-codegen2 and which contains the custom generators

2. There is a "mini model" project that defines a minimal database that demonstrates all the features of the custom generators. It uses a MockDatabase.

3. The real model with the complete DDL. This one uses an XML-based config and a Maven plugin to generate the code.

In the past, it wasn't really possible to extend the code generator, so there was no need to be able to define a config with 2-3 lines of code. Test cases for #2 look like this:

    @Test

    public void testSchema() throws Exception {

        check( "Blu.java" );

    }

    

    @Test

    public void testFactory() throws Exception {

        check( "BluFactory.java" );

    }

    

    @Test

    public void testKeys() throws Exception {

        check( "Keys.java" );

    }

    

    @Test

    public void testTables() throws Exception {

        check( "Tables.java" );

    }

    

    @Test

    public void testClientInfo() throws Exception {

        check( "tables/adm/ClientInfo.java" );

    }

    

    @Test

    public void testGlobalPrivGrant() throws Exception {

        check( "tables/adm/GlobalPrivGrant.java" );

    }

... hundreds of tests; one per generated file ...

    private void check( String file ) throws IOException {

        

        JavaGenerator2 generator = runCodeGen();

        String path = generator.config().getPackageName().replace( ".", "/" ) + "/" + file;

        File expected = new File( "src/main/java", path );

        File actual = new File( generator.config().getRootFolder(), path );

        TestUtils.compareFiles( expected, actual, Encoding.UTF_8.charset() );

    }

This means I define one test case for each generated file. When I run JUnit, I get failures for each output file that did change. That way, I can quickly see whether my changes have the desired/expected impact on the code. And when I run the tests in an IDE, I can use the result comparison to see the expected and actual output side by side.

This approach makes it very safe to add features to the code generator.

After checking the output, I either copy the updated code into the source tree or run the tests with the System property "overwriteSource=true" to let the tests write into src/main/java/.

> .renameBooleanColumn( "BLU.ADM_LANGUAGE.IS_DEFAULT", "default" )

That shouldn't be necessary. The boolean type information is available
on the ColumnDefinition. If you prefer isXXX() instead of getXXX(),
this should be placed elsewhere. In your case, it should be placed
into the BluNamingStrategy implementation, not the API

You're right. I'll add that to the list of TODOs.

> .dialect( SQLDialect.H2 )

I'd prefer if the org.jooq.util.Database implementation is still

referenced rather than a SQLDialect, for two reasons [...]

Actually, this information is used just in a single place:

   log.info("  dialect: {}", config.getDialect());

Deleted.

 

> .schema( "PUBLIC", "BLU" )

Did you plan to keep multi-schema support?

I didn't plan to drop anything because I don't know what's safe to drop.

 

Also, I wonder whether object renames should still be possible in two
places, i.e. naming strategy AND the overall schema reference. The
current generator strategy is a strategy for giving names to Java
objects (it would affect what you call the GeneratorFactory)

Yes, this is very confusing, especially since it leads to getInputName() and getOutputName() in a lot of places and I'm always wondering which one to use...

A new "database strategy" that would contain database object renames
(such as physical schema renames), and also the include() / exclude()
/ includeAll() methods, and maybe the type mapping.

Sounds good to me. Why is this necessary at all? Shouldn't we use the original names and then allow to override those at runtime? What renames to we really need? In my limited view, we only need to allow to set a different schema to allow people to run jOOQ against one database server with two schemas that have the same tables but different names.

 

> .factory( new BluGeneratorFactory( generator.config() ) )

I'm guessing that this will replace what is called DefaultGenerator
today, reading the meta data and generating Java classes?

No, that's an instance factory. In the current version of the prototype, I have already replaced that with DI based on Guice (com.google.inject). This makes it easy to define a custom code generator for POJOs. It's not needed at runtime, only to generate the code.

 

Anyway, we should coordinate your goals. Your own personal goals seem
to diverge a little from the general jOOQ 3.0 roadmap.

Here's the jOOQ 3.0 roadmap:
1. There should be a clean API for jooq-meta. It should be completely
independent of the code generator.
2. The 1) jooq-meta API can be implemented in dialect-independent
ways, reading meta-information from non-JDBC data sources, such as
XML. This can be used to read non-jOOQ XML formats, such as
Hibernate's .hbm.xml
3. The jooq-meta API should have a "database strategy", which models
today's include/exclude features, database type mappings, database
object renames

I agree with the include/exclude and type mappings but isn't renaming part of the naming strategy? Do we really want to strategies?

I'm asking because the Java naming strategy will contain a lot of code to determine what to rename. A lot of that would have to replicated in the database naming strategy when there are two instances since you can only extend a single type in Java.

One thing that would be great is custom attributes. I would like to add something like "this table has columns for a time range" to the TableDefinition so I can quickly look that up in custom generators.

 

4. The output produced by 3) "database strategy" is fed into a
"JavaGeneratorFactory" or into a "Generator". Today's DefaultGenerator
could be a sample implementation thereof

I like the idea of creating a database model and then run a (bunch of) postprocessors on that before its fed into the code generator.

 

6. Today's "GeneratorStrategy", or what you call the "Naming Strategy"
could be called "JavaNamingStrategy", used to map database objects to
Java objects. It is specific to 4) "JavaGeneratorFactory". Today's
"Naming Strategy" is too low-level, more explicit features would be
better.

I already improved that a bit. There is now a method getType() which returns a JavaType instance. This encodes the type information (package + type name, generic parameters, isArray). For the code generator, it has methods like getPath() which returns the path this type would have.

 

For me, the most important features to add are 2) (new data sources),
5) (new target formats) and maybe 6) (improved Java naming
strategies). For you, 6) seems to be the most important improvement,
as well as the fact that 4) can be re-implemented in your own way,
using Xtend.

That sounds about right.

 

If we can agree on the roadmap, I can try to devise an API for 1), 3),
4) and 6), and post it on this user group within the next 1-2 weeks.
You could then again challenge this API with your requirements

Well, the prototype works for me, so you can now have fun with it. I'll improve it over the next few weeks as I add features which we need for our application. In the process, I'll try hard to meet your goals, so the final result can be included in jOOQ 3.0.

Regards,

A. Digulla

 

[digulla]

Hi Lukas,

Since this is a lot of code, maybe you'd like to meet with me so I can give you a tour plus an introduction to the (somewhat odd) Xtend syntax.

Regards,

A. Digulla

[Lukas Eder]

> Since this is a lot of code, maybe you'd like to meet with me so I can give
> you a tour plus an introduction to the (somewhat odd) Xtend syntax.

I have been contacted through other channels about this ;-)
Let me have a look at it first, and then we can discuss things next
week (I'll meet Danilo on Tuesday). In the mean time, I'll have a look
at your suggestions as soon as I have the time...

[Lukas Eder]

Hello,

I think I'm ready to take a superficial stand on this topic.
Essentially, I can say that Xtend / Xtext will not be added as a
dependency for jOOQ 3.0's code generator. I feel that these
technologies are not

1. stable enough (Aaron himself pointed out various flaws repeatedly,
e.g.: http://blog.pdark.de/2012/10/02/xtend-for-java-developers/)
2. independent enough (They create additional required compile-time
and runtime-dependencies)
3. java enough (The tools impose a new non-Java but Javaesque syntax.
jOOQ is about Java)
4. interoperable enough (Xtend / Xtext code cannot be mixed with Java code)

While they seem to be nice tools to be used in highly specialised
client environments, they cannot be imposed on a broader audience.

However, the need for improved customisability of the code generator
have been well received. What Aaron has shown to me proves that it is
relatively simple to create a highly extensible, generic code
generation API that allows for arbitrary artefact generation
overrides. It should be possible to override default code generation
with any other Java-based means, including Xtend / Xtext.

I have also thought about moving away from string-based source code
generation. For instance to the way XJC does with an object-oriented
Java syntax model. I don't think jOOQ should go down that path either,
as code generation isn't necessarily about Java (think of generating
Scala, XML or any other code/document type).

Cheers
Lukas

[digulla]

Am Mittwoch, 3. Oktober 2012 15:06:30 UTC+2 schrieb Lukas Eder:

I think I'm ready to take a superficial stand on this topic.
Essentially, I can say that Xtend / Xtext will not be added as a
dependency for jOOQ 3.0's code generator. I feel that these
technologies are not

1. stable enough (Aaron himself pointed out various flaws repeatedly,
e.g.: http://blog.pdark.de/2012/10/02/xtend-for-java-developers/)

Just like with any technology, Xtend has flaws. From my experience, the advantages outweigh the flaws.

 

2. independent enough (They create additional required compile-time
and runtime-dependencies)

We'd need a version of Xtend on Maven Central but that shouldn't be a show stopper. Other projects need the same plus the generated code doesn't have any dependencies. The dependencies are only necessary when someone works on the code generator itself.

I understand that you're reluctant to add new dependencies to jOOQ (because problems in them will end up on your mailing list) but, as always, it's a question of "how much do I get and how much do I have to pay?"

 

3. java enough (The tools impose a new non-Java but Javaesque syntax.
jOOQ is about Java)

Correct. Java simply doesn't have what it takes. You CAN write a Mandelbrot Set generator in SQL (you can find the query in my blog somewhere) but is that a SMART thing to do? All languages are Turing complete. That doesn't mean you can solve each problem with an equal amount of effort.

 

4. interoperable enough (Xtend / Xtext code cannot be mixed with Java code)

Bullshit. I'm extending Java classes from Xtend and I'm extending Xtend classes in Java. The output of Xtend is pure Java. What gives you the idea that Java isn't compatible with Java?

While they seem to be nice tools to be used in highly specialised
client environments, they cannot be imposed on a broader audience.

Code generators are a specialized niche. There are people who believe that you can write and maintain code generators in Java but these people simply don't know what they're talking about or they have unreliable memory.

 

However, the need for improved customisability of the code generator
have been well received. What Aaron has shown to me proves that it is
relatively simple to create a highly extensible, generic code
generation API that allows for arbitrary artefact generation
overrides. It should be possible to override default code generation
with any other Java-based means, including Xtend / Xtext.

... as long as you don't care about maintenance. Sure, Xtend is pure Java, so you can do everything in plain Java.

I implemented your code generator in ONE WEEK. How many months did the Java version take? How often did you try to fix something just to break it even worse? If you compare my code to the Java version, which one can you understand?

 

I have also thought about moving away from string-based source code
generation. For instance to the way XJC does with an object-oriented
Java syntax model. I don't think jOOQ should go down that path either,
as code generation isn't necessarily about Java (think of generating
Scala, XML or any other code/document type). 

All this approaches have one major drawback: You need many lines of code to produce one line of output. When writing a flexible code generator, your tool should fit the bill. I bet you don't use a road train to go shopping. Unlike all the other tools, Xtend was designed to write code generators.

Yes, it's a moving target. Yes, there are flaws.

But ...

All the other solutions have much bigger flaws. I've been writing code generators for 20 years. I wrote them in C, tcl, Python, Java, Perl, ksh, XML/XSLT, AST-based approaches (like XJC), JSP. They all had one major drawback: You couldn't mix source code with templates.

We all hate JSPs. They suck. But why?

- There is no good editor.

- If you make a change, it's hard to see what the result will be.

- You don't reliably get compile errors in the editor

- You need to deploy the file to a web server (edit -> deploy -> wait 5 minutes -> test)

Xtend is a lot like JSPs but it doesn't suffer from these because there is a good editor. Code is compiled right away as you type (and more complex validations happen when you safe it). There is code completion, even inside templates. The code is readable (unlike JSPs) because you can indent it. Unlike JSPs, the output is also readable.

Regards,

A. Digulla

[Adam Gent]

I just checked out the Xtend code generator and I do have to say it is easier to understand and may use it as starting point for my own code generation.

My compliments to Digulla.

Right now I tried to look into using jOOQ's code generator to just generate simple static String constants for table and column names that have no jOOQ dependency.

String constants are nice because you can use them in Annotation arguments.

When you regenerate the code string constants from the database and your code doesn't compile then you know that there is a mismatch.

I looked at the generator (albeit briefly) but felt like I would have to almost rewrite the generator to do such.

-Adam

[Lukas Eder]

> I just checked out the Xtend code generator and I do have to say it is
> easier to understand

I agree, it looks more intuitive.

> Right now I tried to look into using jOOQ's code generator to just generate
> simple static String constants for table and column names that have no jOOQ
> dependency.
> String constants are nice because you can use them in Annotation arguments.
> When you regenerate the code string constants from the database and your
> code doesn't compile then you know that there is a mismatch.

That's a good idea

> I looked at the generator (albeit briefly) but felt like I would have to
> almost rewrite the generator to do such.

Yes, today's code generator is an organically evolved mess. As I said,
I'll happily look into Aaron's generator API re-engineering
suggestions. I feel that a formally supported and extensible public
generator API will be a big plus in jOOQ 3.0. Re-engineering this API
is independent from using Xtend, though. You will still be able to
implement overriding elements using Xtend.

In any case, don't be discouraged by my decision not to use Xtend.
Feel free to publish, maintain and support your own generator
implementation. If you wish, I can then reference to such an
inofficial generator implementation from the jOOQ website.

Cheers
Lukas

[Lukas Eder]

Hi Aaron,

> Just like with any technology, Xtend has flaws. From my experience, the
> advantages outweigh the flaws.

Yes, I have seen you debugging it :-)

> We'd need a version of Xtend on Maven Central but that shouldn't be a show
> stopper. Other projects need the same plus the generated code doesn't have
> any dependencies. The dependencies are only necessary when someone works on
> the code generator itself.

From what I understand, running such a code generator will need some
runtime dependencies?

>> 4. interoperable enough (Xtend / Xtext code cannot be mixed with Java
>> code)
>
> Bullshit. I'm extending Java classes from Xtend and I'm extending Xtend
> classes in Java. The output of Xtend is pure Java. What gives you the idea
> that Java isn't compatible with Java?

I cannot write Xtend code in a ".java" file. Neither can I write Java
code in a ".xtend" file. I see no strong reason for that not to be
possible.

> Code generators are a specialized niche. There are people who believe that
> you can write and maintain code generators in Java but these people simply
> don't know what they're talking about or they have unreliable memory.

Hehe, that paragraph clearly carries Aaron's signature, alright :-)

> I implemented your code generator in ONE WEEK. How many months did the Java
> version take? How often did you try to fix something just to break it even
> worse?

I was pretty happy with the current generator. It didn't suffer from
many regressions, as it is fully integration-tested. The fact that you
could write yours in one week is also related to the fact that all the
thinking about *what* should be generated was already done. You just
translated mine, you didn't create a generator.

> If you compare my code to the Java version, which one can you
> understand?

I do agree, the current generator suffers from pretty unreadable code
for first-time and second-time readers.

> Xtend is a lot like JSPs but it doesn't suffer from these because there is a
> good editor. Code is compiled right away as you type (and more complex
> validations happen when you safe it). There is code completion, even inside
> templates. The code is readable (unlike JSPs) because you can indent it.
> Unlike JSPs, the output is also readable.

Well, I did understand how Xtend works, and I found it quite
interesting, too. But for now, I rest my case. For me to implement,
maintain, and support such an Xtend-based generator, the platform is
not:

1. stable enough
2. independent enough
3. java enough
4. interoperable enough

I'm sure, this will improve in the future, when I can still rethink
this decision. In the mean time, as I've told Adam Gent in a previous
mail:

[Lukas Eder]

> I'm sure, this will improve in the future, when I can still rethink

> this decision. [...]

On a less serious side-note, Xtend has less traction than jOOQ on
Stack Overflow:

http://stackoverflow.com/questions/tagged/xtend
http://stackoverflow.com/questions/tagged/jooq

;-)

[digulla]

Am Donnerstag, 11. Oktober 2012 19:29:18 UTC+2 schrieb Lukas Eder:

> Just like with any technology, Xtend has flaws. From my experience, the
> advantages outweigh the flaws.
Yes, I have seen you debugging it :-)

As I said: The advantages outweigh the flaws. I didn't try 2.3.1 but I'm pretty sure many bugs have already been fixed.

Also, I'm worried that you favor anecdotal "evidence" over facts here :-(

 

> We'd need a version of Xtend on Maven Central but that shouldn't be a show
> stopper. Other projects need the same plus the generated code doesn't have
> any dependencies. The dependencies are only necessary when someone works on
> the code generator itself.

From what I understand, running such a code generator will need some
runtime dependencies?

To edit the code generator, you need Eclipse and the Xtend plugins at the moment. There is a plugin for Maven which can compile Xtend to Java but I didn't test/try/use that.

To run the code generator, you need org.eclipse.xtext.xbase.lib on the classpath.

The output of the code generator has only the dependencies which you coded into the generator.

 

>> 4. interoperable enough (Xtend / Xtext code cannot be mixed with Java
>> code)
>
> Bullshit. I'm extending Java classes from Xtend and I'm extending Xtend
> classes in Java. The output of Xtend is pure Java. What gives you the idea
> that Java isn't compatible with Java?

I cannot write Xtend code in a ".java" file. Neither can I write Java
code in a ".xtend" file. I see no strong reason for that not to be
possible.

For the same reason why you can't write SQL in a .java file or C or Smalltalk: It's a different language with different goals. Xtend wants to be compact and readable. Java wants to be unambiguous. Xtend supports templates in the language, Java has only very basic support for inlining multi-line strings.

Xtend can be converted to Java. But that doesn't mean you can write a useful grammar that can mix the two.

 

In any case, don't be discouraged by my decision not to use Xtend.
Feel free to publish, maintain and support your own generator
implementation. If you wish, I can then reference to such an
inofficial generator implementation from the jOOQ website.

I'm completely discouraged by your decision. I simply don't have to time to maintain such an effort. For me, your decision means that I've wasted two weeks of my life :-( Plus all the effort I'll have to waste in the future to port my extensions to the code generator to whatever you might eventually come up with.

Regards,

A. Digulla

[digulla]

That's one way to see it.

The c't Magazine once wrote: "Our magazine publishes much more articles on Windows than on Mac or Linux. This isn't because we like Windows better, it's just that Windows make a) most problems and b) most of them are hard to fix even for us."

Regards,

A. Digulla

[Lukas Eder]

> Also, I'm worried that you favor anecdotal "evidence" over facts here :-(

Humour seems to me the appropriate reaction to a rant :-)
I've said this before. I hope you don't take things personal on this list...

> I'm completely discouraged by your decision. I simply don't have to time to
> maintain such an effort.

I don't have that time either, right now. However, when Xtend meets my
expectations, we can discuss again.

> For me, your decision means that I've wasted two
> weeks of my life :-( Plus all the effort I'll have to waste in the future to
> port my extensions to the code generator to whatever you might eventually
> come up with.

You didn't waste your time. You've made a point that was generally
well received. When you offered your help, I was looking forward to a
proof of concept from you, not something that must be integrated into
jOOQ without discussion.

Then again, of course, you could've asked first :-)

I'm sorry you're feeling discouraged by my decision.

Cheers
Lukas