Liquibase vs dev mode

[Stuart Douglas]

Hi Everyone,

At the moment Liquibase runs on every restart and this slows down development mode restarts by quite a bit.

I was thinking it might be good if this only ran when the liquibase files changed, and I came up with: https://github.com/quarkusio/quarkus/compare/master...stuartwdouglas:liqibase?expand=1

Unfortunately this is wrong, as liquibase files are not registered as hot replacement resources, so this won't really work. Looking at the code it does not seem to be possible to actually get the location of these files without running the parser. 

I propose that we adopt a Quarkus convention, where all the changesets need to live in a liquibase folder so we can identify them. Then we can do a dev mode restart if they change, and also only run Liquibase on restart if something has actually changed.

Does this seem like a reasonable solution? I have never used Liquibase, so I want to get some additional opinions before I implement it.

Stuart

[Max Rydahl Andersen]

What are those files besides what quarkus.liquibase.change-log points to ?

/max
https://xam.dk/about

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cwXppn8nW_yORYXMeyRWCk5wtHf3-f_DHjS2JLCELszWA%40mail.gmail.com.

[Guillaume Smet]

The changelog file points to the other files.

But, Stuart, don't we have the list considering we are parsing the changelog file at build time? Or do you want to avoid that altogether?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/35FCD7FA-9F60-47F4-BDC8-046DB1A180EE%40redhat.com.

[Max Rydahl Andersen]

> The changelog file points to the other files.

ah, darn includes ;)

> But, Stuart, don't we have the list considering we are parsing the
> changelog file at build time? Or do you want to avoid that altogether?

I think the issue is that if a user adds a changelog file, we won't know
the list without parsing.
And Stuart is looking to avoid call out to liquibase all together by
only trigger it
when related files changed.

I assume Stuart is finding that the parsing is big part of the 2-300 MS
overhead...but thats my guess.

If not, then yeah, just doing liquibase parsing and then check should
work?

/max

>> <https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cwXppn8nW_yORYXMeyRWCk5wtHf3-f_DHjS2JLCELszWA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .

>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups
>> "Quarkus Development mailing list" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an
>> email to quarkus-dev...@googlegroups.com.
>> To view this discussion on the web visit

>> https://groups.google.com/d/msgid/quarkus-dev/35FCD7FA-9F60-47F4-BDC8-046DB1A180EE%40redhat.com
>> <https://groups.google.com/d/msgid/quarkus-dev/35FCD7FA-9F60-47F4-BDC8-046DB1A180EE%40redhat.com?utm_medium=email&utm_source=footer>
>> .
>>

[Guillaume Smet]

Yes, the parsing is a bit heavy unfortunately.

[Stuart Douglas]

We don't parse at build time, except for native image. The parsing is slow, so we want to avoid it if possible.

[Max Rydahl Andersen]

I guess what we could do is a hybrid that adds a liquidbase folder to monitor for changes + get list from the parser if run.

Then it would "just work" except for the case where a new include is added referring to a file outside the liquibase folder.

could be that "get list from the parser + monitor it" can be a significant overhead but at least it would work for existing setups.

/max
https://xam.dk/about

[Stuart Douglas]

I think just have a convention of 'put in this folder' is all we need. If you put it outside you lose Dev mode support but I think that is ok.

Stuart

[Max Rydahl Andersen]

make it default the base dir of the existing change-log property ?
/max
https://xam.dk/about

[Guillaume Smet]

Well, we could be a bit more clever than a convention.

We could check every file in the directory containing the changelog as it's defined at build time: https://github.com/quarkusio/quarkus/blob/master/extensions/liquibase/runtime/src/main/java/io/quarkus/liquibase/runtime/LiquibaseDataSourceBuildTimeConfig.java#L29

The only rule would be: put your changelog files in a specific directory where you don't put other files.

[Max Rydahl Andersen]

just realised our flyway integrations have this:

https://quarkus.io/guides/flyway#quarkus-flyway_quarkus.flyway.locations

mimic that ?

make liquibase.locations=${basedir:liquibase.change-log} but you can give a list if you want ?

then the semantics are somewhat the same in devmode ? (though I think flyway don't need a starting file)

/max
https://xam.dk/about

[Stuart Douglas]

So in jhipster in goes in src/main/resources/config/liquibase/ with the change logs in src/main/resources/config/liquibase/changelog, so this would work well as long as we include subdirectories.

[Max Rydahl Andersen]

Well, we could be a bit more clever than a convention.

We could check every file in the directory containing the changelog as it's
defined at build time:
https://github.com/quarkusio/quarkus/blob/master/extensions/liquibase/runtime/src/main/java/io/quarkus/liquibase/runtime/LiquibaseDataSourceBuildTimeConfig.java#L29

The only rule would be: put your changelog files in a specific directory
where you don't put other files.

Yes, thats what my proposal was - use base directory of existing change-log property, but have a optional .locations property similar
to flyway integration to make the configuration similar.

On Fri, Apr 24, 2020 at 12:33 PM Stuart Douglas <sdou...@redhat.com> wrote:

I think just have a convention of 'put in this folder' is all we need. If
you put it outside you lose Dev mode support but I think that is ok.

Stuart

On Fri, 24 Apr 2020, 8:30 pm Max Rydahl Andersen, <mand...@redhat.com>
wrote:

I guess what we could do is a hybrid that adds a liquidbase folder to
monitor for changes + get list from the parser if run.

Then it would "just work" except for the case where a *new* include is

[Guillaume Smet]

On Fri, Apr 24, 2020 at 1:35 PM Max Rydahl Andersen <mand...@redhat.com> wrote:

Yes, thats what my proposal was - use base directory of existing change-log property, but have a optional .locations property similar
to flyway integration to make the configuration similar.

The configuration is not similar because both components behave differently.

Liquibase starting point is a changelog file, not a directory as for Flyway.

Being consistent is not really needed here given people will only use one or the other, not both.

[Manyanda Chitimbo]

Hi, 

We can also keep the Liquibase team on the loop, I remember that Steve, Community Manager commented on a PR to improve perfs. 

--

You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALt0%2Bo9Vp3PzAR%2BkqRk_JUk2Z970QdfBOLj4g8BRTGRKSom%2B9Q%40mail.gmail.com.

--

Manyanda Chitimbo.

[Daniel Petisme]

Hello,

Convention sounds good. 

My 2cts, defining the current dir `quarkus.liquibase.change-log` and sub-dirs are candidates for hot reload ?

Since, the default value is db/changeLog.xml, Quarkus would listen db/**. For jhipster, I define `src/main/resources/config/liquibase/master.xml` so It should be `src/main/resources/config/liquibase/**`.

This would make sense ?

Another different approach would be to force a changelog file convention. Proposing a configurable quarkus.liquibase.change-log.dir property and expecting a `changelog.xml` (or whatever.xml). With this option, Liquibase and Flyway would be consistent for hot reload and IMHO forcing a file name is an acceptable price.

 

Daniel PETISME
daniel....@gmail.com

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANwCmXJM6b6OdsF9f%3D15xaAWcq1CQThNyFSYFyphAyrdsHY37A%40mail.gmail.com.

[Stuart Douglas]

I have actually figured out a way to improve the performance here: https://github.com/quarkusio/quarkus/pull/8934/files

Liquibase has a fairly slow service discovery mechanism, that I replaced with a Jandex based one, and more importantly limited the Database implementations to just the ones in use. It looks like it generates SQL for every known database so this made a big impact on startup time.

Stuart

[Max Rydahl Andersen]

> I have actually figured out a way to improve the performance here:
> https://github.com/quarkusio/quarkus/pull/8934/files
>
> Liquibase has a fairly slow service discovery mechanism, that I
> replaced
> with a Jandex based one, and more importantly limited the Database
> implementations to just the ones in use. It looks like it generates
> SQL for
> every known database so this made a big impact on startup time.

very cool! I wonder why they generate for all database...seems "too
much" even
as their own default implementation. Awesome Quarkus can help it do more
with less :)

The improvement on monitoring change log file and its base dir content
would
then not be so much help performance wise but just make it smoother to
have it reload if you changed the files between events, right ?

/max

>>> <https://github.com/quarkusio/quarkus/pull/8028#discussion_r395782240>

>>>> <https://groups.google.com/d/msgid/quarkus-dev/CALt0%2Bo9Vp3PzAR%2BkqRk_JUk2Z970QdfBOLj4g8BRTGRKSom%2B9Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .

>>>>
>>>
>>>
>>> --
>>> Manyanda Chitimbo.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups
>>> "Quarkus Development mailing list" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an
>>> email to quarkus-dev...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/quarkus-dev/CANwCmXJM6b6OdsF9f%3D15xaAWcq1CQThNyFSYFyphAyrdsHY37A%40mail.gmail.com

>>> <https://groups.google.com/d/msgid/quarkus-dev/CANwCmXJM6b6OdsF9f%3D15xaAWcq1CQThNyFSYFyphAyrdsHY37A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>

[Stuart Douglas]

On Wed, 29 Apr 2020 at 16:57, Max Rydahl Andersen <mand...@redhat.com> wrote:

> I have actually figured out a way to improve the performance here:
> https://github.com/quarkusio/quarkus/pull/8934/files
>
> Liquibase has a fairly slow service discovery mechanism, that I
> replaced
> with a Jandex based one, and more importantly limited the Database
> implementations to just the ones in use. It looks like it generates
> SQL for
> every known database so this made a big impact on startup time.

very cool! I wonder why they generate for all database...seems "too
much" even
as their own default implementation. Awesome Quarkus can help it do more
with less :)

The improvement on monitoring change log file and its base dir content
would
then not be so much help performance wise but just make it smoother to
have it reload if you changed the files between events, right ?

Yea, it would help hot reload performance, basically just figure out if it actually needs to run or not and skip it if there are no changes.

Stuart