Microprofile config yaml overwrite with environment variable

[Tobias Dittrich]

Hi all,

I have an injected MP config value in my class:

@Inject
@ConfigProperty(name = "app.product.minQuantityOfValidProducts")
int minQuantityOfValidProducts;

In project-stages.yml:

app:
  product:
    minQuantityOfValidProducts: 10000

which works fine - the variable is initialized with 10000 as expected. However I want to be able to override this yml config via environment. Background is that the application is dockered and running for several different sales channels in a docker swarm environment. The channel for each docker instance is configured like this

docker service create \
--replicas 1 \
[..]
-e "swarm.project.stage=live" \
-e "product.channel=ALTERNATE_B2C_DE" \
[..]

Now I thought I could just add another -e "app.product.minQuantityOfValidProducts=10" but this is not working. From the smallrye code I found out that the io.smallrye.config.EnvConfigSource is doing a .replaceAll("[^a-zA-Z0-9_]", "_"); so I thought -e "app_product_minQuantityOfValidProducts=10" should work but it doesn't.

Using System Properties works fine bzw. (-Dapp.product.minQuantityOfValidProducts=10 initializes minQuantityOfValidProducts with 10) but unfortunately this is not an option in my docker scenario.

Is there a misunderstanding from my side? Is there a way to override the yml values via environment or do I have to find another way?

Many thanks in advance for your help.

[Ladislav Thon]

On 07. 03. 19 13:50, Tobias Dittrich wrote:

Now I thought I could just add another -e "app.product.minQuantityOfValidProducts=10" but this is not working. From the smallrye code I found out that the io.smallrye.config.EnvConfigSource is doing a .replaceAll("[^a-zA-Z0-9_]", "_"); so I thought -e "app_product_minQuantityOfValidProducts=10" should work but it doesn't.

Per the MicroProfile Config specification, exact match should work, and it seems to be implemented correctly in SmallRye Config: https://github.com/smallrye/smallrye-config/blob/master/implementation/src/main/java/io/smallrye/config/EnvConfigSource.java#L53 Can you place a breakpoint in EnvConfigSource.getValue to make sure it's actually invoked?

BTW it seems you're using the legacy project-stages.yml format (with multiple stages in the same file). I'd recommend moving to the new project-defaults.yml style, with each stage in a different file -- there, you can have one file with defaults embedded in the application and another file, environment-specific, outside of the uberjar. You can find more info about that in https://docs.thorntail.io/2.3.0.Final/#configuring-a-thorntail-application-using-yaml-files_thorntail

Also you can check https://docs.thorntail.io/2.3.0.Final/#configuring-a-thorntail-application-using-environment-variables_thorntail if that helps in your particular case.

LT

[Tobias Dittrich]

Ok I found out what's happening: the environment has a lower priority than yaml file. If I delete the property from the yaml file the variable is injected with the value from system env as expected.

In my opinion this is not expected behaviour (aka a bug). What do you guys think?

[Ladislav Thon]

Ha, I know what's going on, and it's actually a little more complicated. During the boot, we create a system property for each and every YAML configuration value. In MP Config, system properties have higher priority than environment variables. Together, these 2 things explain the behavior you see.

Now, creating a system property for each YAML config value seems weird, but this is actually what allows injecting YAML config using MP Config. We could build a more direct bridge, but I know there are other usecases, so I don't think we can stop it entirely.

At the same time, I can see why environment variables are the easiest way for you. Besides the option of using multiple project-*.yml files I mentioned last time, I've got one more suggestion: you could create your own ConfigSource with a very high priority (ordinal, per MP Config wording) and load all the configuration keys your application "owns" from environment variables in there.

Hope that helps,

LT

--
You received this message because you are subscribed to the Google Groups "Thorntail" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thorntail+...@googlegroups.com.
To post to this group, send email to thor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/thorntail/849c07f3-e855-4d43-995c-9506ff411aa1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tobias Dittrich]

Hey Ladislav,

thanks for your explanations. Your suggestion is a valid workaround I already have come up by myself - but in my opinion it's exactly that: a workaround. My feeling is that forcing people to implement their own config source for every project where environment overwrite is needed is suboptimal.

But it might as well be that I am wrong in how I use the docker mechanics - it seemed to me straight forward to use environment variables for individual docker instances. What are your thoughts about this?

Tobias

[Ladislav Thon]

Hi Tobias,

I see your point, and I agree that the way you use environment variables is completely natural. However, remember that there are two configuration systems at play here: Thorntail's YAML files, and MicroProfile Config. The bridge between them isn't totally seamless, and at this point, I know of people that rely on that unintentional side effect of building the bridge on top of system properties. (Actually I think that we exposed all YAML configs as system properties even before we had MP Config, so the bridge came for free.)

One more suggestion: if you only used MP Config, that is, if you placed your app.product.* configs into microprofile-config.properties and not to the YAML, then I believe you wouldn't have a problem.

LT

To view this discussion on the web visit https://groups.google.com/d/msgid/thorntail/1139edc7-8e7e-4769-b4d2-0bf2f51b8967%40googlegroups.com.

[Tobias Dittrich]

Hi Ladislav,

thanks again for explaining I can totally understand the problems. Your suggestion with microprofile-config.properties works fine and I have to admit that this is the "cleaner" / more standard way to do it. We just got used that much to the yaml files and I like the syntax more than plain .properties :)

But for now I'm content with that solution.

Many thanks

Tobias