Config API Prototype

[Emily Jiang]

I proposed some Config API under https://github.com/microprofile/microprofile-config. The API was created with some input from Config JSR presentation on JavaOne from 

Dmitry Kornilov and Mark Strugberg's initial proposal. There is some open questions I would like to get some input:

1. ConfigProvider: At the moment, I made it an interface. To make it more useful and easier for applications to use, it should be a class. Should we use serviceloader to create an instance of this?

2. Dynamic configuration support: Should we add one more method on ConfigSource to specify the time interval for accessing the new configuration values?

3. Performance issue: for massive ConfigSource, should we specify some policy to ready configuration lazily?

Thoughts?

Emily

[Viktor Klang]

Hi Emily,

Where can I read about the requirements & design rationale?

[Gordon Hutchison]

Hi Emily,

In response to 1. below, I was sent this link today that discusses the use of Interfaces versus Classes, the use of statics and bootstrapping
access to an API:

   http://blog.bjhargrave.com/2013/10/api-design-practices-that-work-well.html

Gordon

[Gordon Hutchison]

Regarding the interface ConfigBuilder at
   https://github.com/microprofile/microprofile-config/blob/master/src/main/java/io/microprofile/config/ConfigBuilder.java
I noticed there was a method to add an individual ConfigSource via the varargs addSources but no means to
remove a particular ConfigSource apart from removeAllSources()?
(A user could implement one though using get/add sources).

Also with ConfigProvider we have:
    /**
     * Create a builder so that custom config resources can be added.
     * @return an instance {@link ConfigBuilder}
     */
    ConfigBuilder builder();

If we have multiple builders with different sources created using builder() which builder (where the sources are set) builds the config returned by
    myConfigProvider.getConfig();

Do we need a definition of the ConfigProvider's current builder?
(preferable to  singleton builder or moving the getConfig to only be on builder?)

Gordon.

On Sunday, 13 November 2016 23:35:01 UTC, Emily Jiang wrote:

[Emily Jiang]

I will try to write the readme this week to describe the flow. However, I recommend you read the other two config related discussions so that you will understand the rational.
Thanks
Emily

[Emily Jiang]

The design pattern is for OSGi environment. In OSGi, the implementation can be looked up via OSGi service registry. The reason to use interfaces instead of concrete classes, it will cater for multiple implementations. The service providers can provide multiple implementations of one particular service and the client can specify which one to wire to. However, in non-OSGi environment, there is no OSGi service registry and the applications cannot look up implementation easily. In order to make this API usable for applications, it is better if we provide ConfigProvider class so that a user can do "ConfigProvider.builder().addSources().build".

Emily

[Kevin Sutter]

Emily,
The readme is definitely required.  If people take a look at your repo, they need some context on what's being proposed.

And, to be quite honest, we have so many conversations happening on the MicroProfile Google Groups it's hard to keep them all straight.  And, there's nothing that states this thread necessarily has anything to do with the other threads.  So, although I agree that participants should try to keep up with the other threads and take all of this information into account, it would be worthwhile to discuss any questions that come up on this particular thread.

Thanks, Kevin

[Viktor Klang]

Emily, thanks for taking the time to write the README, that's going to be very helpful to me and others.

Sadly/unfortunately, I did read both those [prior] threads, and I am unable to deduce what the actual agreed-upon design goals are or how they are intended to be evaluated.

Furthermore, I'm not sure how decisions are designed to be made or how compliance to an eventual standard is verified.

As developers (myself very much included) it is natural to start thinking about the solution space early, in my (limited) experience—when it comes to creating standards across vendors—the most important thing is to reach a decision on the scope and design goals first. Now, I am most likely probably missing where the aforementioned information is located, and would really like to get a pointer in that direction so I can return with some actual feedback. :)

[Emily Jiang]

Thank you Viktor for taking time to read all threads! I tried to express rational in my PR https://github.com/microprofile/microprofile-config/pull/1/files. Please read it and let me whether it addresses your concern. Let me know if it is not quite you are asking for.
Thanks
Emily

[Ondrej Mihályi]

This week at Java2Days in Sofia, Dmitry Kornilov unofficially disclosed that the proposal for a new config JSR for Java EE 8 is already ready to be submitted to the JCP. It should be submitted as soon as it gets proper internal approvals within Oracle. Dmitry expects he can submit the proposal to the JCP in 1 or 2 months. From what I saw, the proposal is heavily based on Apache Tamaya - the API was already presented several times, last time last week in Sofia - here are the slides with API examples: http://www.slideshare.net/DmitryKornilov/configuration-for-java-ee-config-jsr-and-tamaya

I recommend that if we approve the microprofile config proposal, we should do it after we discuss how much we would like to be compliant with the upcoming Java EE config JSR. I'd like to have a vision on how to potentially converge with the new JSR once it eventually gets final in 2017 or later. Or even if we want to collaborate with the new JSR EG on the spec. However, in my opinion, we should not delay acceptance of the MicroProfile config proposal. I would only be careful not to stray too much from the new Java EE config spec.

--Ondrej  

Dňa piatok, 18. novembra 2016 14:39:14 UTC+1 Emily Jiang napísal(-a):

[Werner Keil]

Together with Dmitry I showed some of the basic aspects he presented already implemented and working in Tamaya. 

In the sense, that a config standard will naturally separate an API from a RI, Tamaya does that already. While several prior examples like Apache Commons Config (it improved a bit with V2) Spring Config or Typesafe Config usually don't distinguish between API and implementation. Neither of them are standards, so it may not be in their interest.

While it is far from known, if the new Config JSR could use an Apache or Apache License project as RI, having multiple implementations including Microprofile certainly can't hurt. In the other session Dmitry and I held together on JSON-P and JSON-B a question from the audience was why, e.g. some of the most widely used JSON frameworks like Jackson do not implement those standards (at least JSON-P is already final) 

Having multiple implementations at an early stage, many of them actively involved in the EG could certainly help. Defining a "shadow" API could be a first step similar to what e.g. Tamaya did so far, but even if the RI might have to be in a different place (e.g. Glassfish) Tamaya aims to implement it in a way e.g. Johnzon or Open WebBeans do with the JSON standards or CDI.

Werner

[Emily Jiang]

The aim of this Config API exercise is to standardise the APIs. We are trying to separate the APIs from impls. After the APIs are standardised, the current configuration providers e.g. Apache DeltaSpike, Archaius, Tamaya can be updated to use this APIs. We can choose one implementation as RI. Once this is achieved, the microservices will be portable.

The APIs I am trying to propose is very close to the JSR proposal and also close to what Mark Strugberg brought up (Thanks Mark!). Both of them look very close anyway:o.

Thanks
Emily

[Werner Keil]

It's usually the Spec Lead who picks a RI;-) 

Whether or not Dmitry considers a project under Microprofile or elsewhere, we'll learn soon enough. If there are multiple implementations, it is certainly not a bad thing, of course some like Archaius or Commons Config have a significant "market share" already, but especially Archaius by itself is portable and can be backed by either Commons Config or Typesafe Config, so let's see, maybe it'll also pick up a future standard.

Werner 

[Emily Jiang]

Yes, Werner. The JSR spec lead will pick the RI for JSR config. Ondrej and I meant the Config API RI in MicroProfile. When the MicroProfile community standardises the Config API, ideally the RI should be identified as well, which might be same or different from the JSR config RI.

Thanks
Emily

[Werner Keil]

There is only one RI for the JSR, but MicroProfile could pick another implementation of its choice or create its own;-)

Since Microprofile doesn't do a spec, there is no real room for a RI or TCK of its own. All it could do (and probably should if implementing either JSR by itself) is pass the TCK of such standard.
The interoperability test suites or reference apps (here the word "reference" is probably OK) are a different thing, and good to progress further.
Arjan added CI jobs for some of the most popular containers (many of them contributors here) running Soteria: https://travis-ci.org/javaee-security-spec/soteria

At least once the transition to Eclipse was made, this would be great to have for demo apps like the conference one.

Cheers,
Werner

[Ondrej Mihályi]

Hi Werner,

To be honest, I must disagree with your claim that MicroProfile doesn't do specs. You probably meant "JSR" specs, which is true. On the other hand, one of the visions of MP is to come up with new Java specifications agreed on by vendors and the community. These specifications cannot be called JSR and would have nothing to do with the JCP unless submitted to the JCP eventually.

But there's nothing stopping us to have both a Microprofile config spec and a Java EE config JSR. Actually, it is necessary to have both as the JCP and MicroProfile have different lifecycles. I would only strongly suggest to not diverge too much so that the efforts can easily become compatible once both specifications are final.

Thanks, Emily, for clarifications about the similarities between the MP proposal and the other activities/projects. I think we're on a good path being close to the expected Java EE config JSR proposal.

--Ondrej

[James Carman]

So, why are we saying we need an EE-based spec?

[John Clingan]

We prefer not to implement two different specifications. There are a couple of reasons:

1. We prefer not to split the Java EE community. MicroProfile is trying to bring the traditional Java EE community together around microservices.
   
   
2. Vendors do not want to integrate two separate configuration APIs. While this may sound self serving, it is pragmatic. We need to define a consistent platform that multiple vendors agree on. It gets harder to do if we have disparate approaches to configuration.

The above applies to Java EE 8 (which is where the Configuration spec falls). Health Check also falls under this umbrella. Personally, I think we need to figure out a better approach for Java EE 9 which I covered here.

Open to other thoughts and opinions.

[James Carman]

Okay, cool.  That's the part I was missing that MicroProfile is inherently EE-based.  Is that right?  Are there more "lightweight" efforts going on?  I really don't like the idea of having to have a full EE stack in order to run a microservice, which is why I chose Apache Karaf as my container for "Microbule".  With Apache Karaf, I can kind of pick and choose the technology pieces I want to use.  

[Mark Little]

We have started out at being based on EE. However, if you check the mail archives we have discussed possibilities where certain MP "profiles" are not EE compatible any more. These aren't necessarily profiles that all implementers will support and that's fine as that is a choice and freedom. But at least developers will know what implementations they can migrate their applications between or interop with. Of course some of those non-compatible profiles may still use many of the core capabilities from EE.

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "MicroProfile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/a30a6ad6-77d1-4f3c-ab3b-287be05cf4eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gordon Hutchison]

Some of the servers, such as Liberty, are now excellent at being shrunk down to just what is needed
but I also spotted Hammock, which is coming from the other direction, in relation to microprofile:
https://github.com/hammock-project/hammock/wiki/Microprofile

[Werner Keil]

A profile may "mix and match" things, but also Eclipse where the journey seems to go with steady pace does not consider itself standard-setting.

You find plenty of Eclipse projects or working groups implementing standards by OGC (Open Geospacial Consortium), OMA (Open Mobile Alliance), JCP and many others. So does Apache in some cases, e.g. SIS is an important implementation of the GeoAPI standard (there may be others, it seems OGC does not insist on the term "Reference Implementation" but SIS led by Martin and others who shape GeoAPI could be seen as such) and Pluto is the RI of the Portlet standard including version 3. We saw other RIs at Apache in the past.

So neither Apache nor Eclipse consider themselves "standard-setting", thus it makes sense to respect that and leave those standards where they are defined.

Werner

[Emily Jiang]

MicroProfile community is a new community and one of the missions is to address the gap left by Java Community. You might be right that in the past, there was no standard-setting, but in the future, it might be different. World is changing...

May I suggest we use this thread to discuss the technical aspects of the Config API? For 'the relationship of MicroProfile and JCP' discussions, can we comment on the thread "MicroProfile and JCP" initialed by Ivan, which was created for this type of discussion"? It will help the late comers who would like to know some particular areas by filtering the contents via topics. 

Thanks
Emily

[Emily Jiang]

After some useful and constructive discussions with Mark and Viktor via PR comments(Thanks Mark, Viktor!), the PR has become very difficult to be followed. As per Steve Millidge's request, I have merged the PR. Please take a look at the API and raise issues/PRs if necessary. The next stage is to discuss the implementation.

Thanks
Emily

[Werner Keil]

It's very minimalistic so far, but until an actual JSR does make it through the trenches inside Oracle, it is not bad to sketch some things.
Converting from strings is fairly close to what Dmitry proposed, so there should be no big deal to implement a future standard when that's available.

The difference between "getPropertyValue" and "getString" sounds a bit inconsistent. I may also raise either an issue or PR if I get to.

Regards,
Werner

[Emily Jiang]

Thank you Werner for your comments! I have changed getPropertyValue to getValue. As for the copyright, I will start another discussion.

Are there any other comments? If not, we can start focusing on the implementations.

Thanks
Emily

[Werner Keil]

You're welcome.

Please note, there is a thread on correct headers by John, maybe best to keep such discussion there https://groups.google.com/forum/?hl=en#!topic/microprofile/UvPWRMOtcfk

Werner

[Heiko Braun]

Emily, before the year closes, can you gives us a quick heads up about the state of the project? I.e. what you have achieved, plans for next year, etc?

[Werner Keil]

There seem to be just a few API elements, no implementation as of now.

Since the API is relatively close to what Dmitry spoke about between JavaOne and Java2Days, I assume we'll have to see, how it goes with a JSR proposal in that case. Of course having some implementation bits could not hurt, either to swap for a future JSR or (in the unlikely event it somehow shouldn't work out;-O) a more complete API here

[Emily Jiang]

Heiko,

I think the APIs are pretty much defined and at the moment we are doing the implementation by adapting Archaius. Will try to contribute once it is done. If anyone else would like to help, please shout. To look ahead in 2017, I plan to get API and implementation standardised in 1Q.

Emily

[Werner Keil]

Standardised?

Do you (IBM) plan to propose a Config JSR together with Dmitry/Oracle?

Or instead of them? There are not too many cases of competing specs filed in parallel, but e.g. JSR 330 and 299 (CDI) were in competition at first before finding synergies. 

Or JSR 347 was filed to leverage or compete with JSR 107. Till it was eventually withdrawn by Red Hat.

Curious to see, how to implement this API with Archaius, given at least Archaius 2 https://github.com/Netflix/archaius/tree/2.x/archaius2-api has quite a strong and extensive API of its own. Backed by at least 2 configuration frameworks like Apache Commons Config or Typesafe.

Werner

[John D. Ament]

Emily,

I'm a bit concerned about what I see here.  Are you/IBM implementing this in private?  Or is it public?

John

[Emily Jiang]

We will contribute here. There is nothing operating in private. This is the whole point of the standardisation. I would like to see the API, impl and tests all available in the repo. Actually, it will be good if anyone else contributes the implementation and tests.

Emily

[Emily Jiang]

Wrener,

I am referring in MicroProfile.io community when I talk about standardisation. Not trying to compete with Config JSR. This effort should be considered as laying out foundation for config JSR and eventually feeding into Config JSR. We are not trying to overrule JSRs, just a prototype exercise and help/ease the effort in JSRs. As you know, this exercise is purely trying to overcome the release gap in JavaEE. Hope this helps!
 
Archaius2 has a great reputation for coping with dynamic config changes. As mentioned in another reply, I recommend all other config implementation to use this APIs. Anyone who would like to contribute impl or tcks are welcome.

Emily

[Anatole Tresch]

Hi Emily

I am alsp happy to add some thoughts to the discussion here:

• When using Java 8 as a base we might even have no need for a ConfigProvider, since we can add accessor methods on the Config itself. 
  
• The biggest issue with the ServiceLoader is that it is very minimalistic and it does not work in OSGI. In Tamaya we therefore defined a ServiceContext, which implement the logic that makes sense for the current runtime. This also allows us to load multiple implementation of a type (e.g. providers) using the same entry point. In the ladder case we also honor @Priority annotations for overriding and ordering.
  
• I would leave off dynamic config at the beginning. In Tamaya we have implemented something similar with DynamicValue. It basically works but creates quite a lot of subsequent questions. So it is better to leavfe out that kind of features for now.
  
• Related to dynamic config support in Tamaya the sources must not return the same values all the time. The config tree can be regularly reevaluated for changing (in Tamaya done by the events module, which triggers corresponding change events). This behaviour can be easily factored out into decoraors tha may cache or freeze a sources initial state. Therefore the source API itself must not refelct this kind of requirements.
• Performance: in Tamaya and Deltaspike a config source must not return anything usable for getProperties() (a config source in Tamaya declares this by the scannable property). So with single value and map access also big configs can be supported easily.

Now working a couple of years on the topic, my biggest concern is the concept of ordinals. I personally think in the meantime that ordering should not be controlled (or at least not only controlled) by ordinals. On a first sight this concept looks charming. But thinking on getting a configuration system as easily configured as a logging system things are getting complicated. Imagine the following snippet:

<sources>

  <source type="..." ordinal="100" />

  <source type="EnvProperties" />

  <source type="SystemProperties" />

  <source type=*...* refreshable="true" />

</source>

Assuming lower ordinals map to lower priorities the following problems now occur:

1. Refreshing may also change the ordinal of a config source, how to be aware of such changes?
2. The SystemProperties may have some hardcoded (default) ordinal value. When I now want to let EnvProperties override SysProperties. Must I create a wrapper to redefined the ordinal, so it matches? Just to apply some reordering. In Tamaya I have operations on the builder like increasePriority(PropertySource), decreasePriority(PropertySource), etc. This is much more intuitive than using ordinals, which even may not be known a all.
3. The XML configuration above already defines an intuitive ordering (from top to bottom). The ordinals added may conflict with this order.
4. ...

In Tamaya we were comming up with the following solution:

1. Basically a Config is created based on a ConfigurationContext.
2. The ConfigurationContext is
1. an ordered list or config sources
2. an oredered list of global filters (not important for this topic here)
3. an oredered lis of converters per target type.
3. A ConfigurationContextBuilder allows to create the ConfigurationContext exactly the way I want it to be without having to care on ordinals. Basically when I use a builder these ordinals are useless, since I define my ordering by the order of adding my sources/converters etc. to the builder. This ends up in a nice programmatic builder API.
4. If I want to have some unified ordering for prioritizing my config sources, using some ordinals might be only one choice, there may also be others. As a result the context builder in Tamaya allows to pass the comparator to be used as an argument to sort-methods on the builder. Of course sorting by ordinal is still possible, typically by using the default comparators in place.
5. With the context built the ConfigProvider allows to transform it into a Config. This Config is not registered, so it can be used as component where useful in the current context. Similarly the DEFAULT Config can be replaced with new Config created by calling ConfigProvider.setConfig(Config). With that Tamaya supports both: the factory based approach and the service location styled approach.

These are my 5 cents for now, so hopefully we end up in an API that is powerful enough and as minimalistic as possible ;)

J Anatole (PPMC Member of Apache Tamaya)

[Ondrej Mihályi]

Hi Emily, Anatole,

I agree with Anatole that using the static ServiceLoader is an anti-pattern and it is hard to reuse in all contexts. I'm strongly in favor of using ConfigProvider as plain object without static methods, and provide a default @ApplicationScoped bean instead of the static factory method. Outside of CDI, the I recommend to provide 2 public constructors:

• the default one without parameters (already available in the current proposal)
• additional constructor accepting an instace of ConfigProviderResolver, which override the default ConfigProviderResolver

This makes it possible to reuse ConfigProvider in any context - CDI, plain SE or OSGi.

I agree with Anatole again about the ordinals. They are the most confusing part of the proposal. I didn't like them in the Config JSR presentations, neither I like them here. I don't even think that we need to support configuration of the sources in XML, but if we do, we should rethink ordinals. I suggest that we name the configurations instead, and provide means to insert a configuration before/after another named configuration. It would make the setup much more readable and deterministic:

<sources>

  <source type="Custom" name="defined-in-code"/>

  <source type="EnvProperties" name="env"/>

  <source type="SystemProperties" />

</source>

ConfigBuilder.withSourcesBefore(String nameBefore, ConfigSource... sources);

ConfigBuilder.withSourcesAfter(String nameAfter, ConfigSource... sources);

If called as configBuilder.withSourcesBefore("env", secondConfigSourceDefinedInCode), then there would be sources in this order:

• name="defined-in-code" (defined in code, but ordered by XML before all others)
  
• secondConfigSourceDefinedInCode (defined in code, ordered in code)
  
• EnvProperties
  
• SystemProperties
  

This is only a suggestion, it might not be bullet proof. But I strongly suggest to find an alternative to ordinals, as they are really confusing and old-school.

--Ondrej

Dňa štvrtok, 22. decembra 2016 1:11:43 UTC+1 Anatole Tresch napísal(-a):

[Werner Keil]

Hi,

Full agree, especially now that Microprofile was accepted as Eclipse project it seems even more beneficial if any implementation or solution defined here works as good citizen with JSR 330 (not just CDI, or at least the SE options planned for CDI 2) compliant dependency injection and OSGI. Maybe Jigsaw (where that isn't compatible with existing mechanisms) in future, too.

Whether ConfigProvider changes to an interface or abstract class, it is worth considering it as SPI element, see CDI.

Of coure that is mostly subject to an upcoming JSR and looking at the results from the new Java EE Survey:

https://blogs.oracle.com/theaquarium/entry/java_ee_8_community_survey2.

Configuration is among the most popular choices, so it would be more stupid than ignorant of Oracle and everyone else to ignore the community's demand by not doing anything about it ;-)

Werner

[Emily Jiang]

Hi Anatole, Ondrej, Wrener
Thanks for for your comments! Let me try to clarify/answer your queries.

• When using Java 8 as a base we might even have no need for a ConfigProvider, since we can add accessor methods on the Config itself.
  

I don't quite understand what you mean here. Do you mean Config.setProvider().getBuilder().build?

• The biggest issue with the ServiceLoader is that it is very minimalistic and it does not work in OSGI. In Tamaya we therefore defined a ServiceContext, which implement the logic that makes sense for the current runtime. This also allows us to load multiple implementation of a type (e.g. providers) using the same entry point. In the ladder case we also honor @Priority annotations for overriding and ordering.

ServiceLoader works in OSGi by using SPI Fly in Apache Aries (part of OSGi Enterprise specification). ServiceLoader pattern is not anti-pattern and exists in a large number of JSRs to separate API and implementations. Tamaya also uses service loader pattern to load ServiceContext.
I do have plan to get this working in OSGi, which is to add setter/getter and the impl bundle can direct set it. This should work well in OSGi environment.
As for priority for the provider, I don't see any user case to supply more than one ConfigProvider. Perhaps I miss something. Can you explain your user case here?

• I would leave off dynamic config at the beginning. In Tamaya we have implemented something similar with DynamicValue. It basically works but creates quite a lot of subsequent questions. So it is better to leavfe out that kind of features for now.

Agreed. Nothing should surface to the api level as for the dynamic config handling. It is down to implementation to decide.

• Related to dynamic config support in Tamaya the sources must not return the same values all the time. The config tree can be regularly reevaluated for changing (in Tamaya done by the events module, which triggers corresponding change events). This behaviour can be easily factored out into decoraors tha may cache or freeze a sources initial state. Therefore the source API itself must not refelct this kind of requirements.

Good to know. Agreed!

• Performance: in Tamaya and Deltaspike a config source must not return anything usable for getProperties() (a config source in Tamaya declares this by the scannable property). So with single value and map access also big configs can be supported easily.

Can you elaborate a bit more on this. Do you return all the usable properties then. I guess they are not huge.

As for your comment on ordinal, the value is not baked in the config source files. The config source file content changes should not have any impact on this. The ordinal is defined when adding the config source to the config builder. I think from what you say, the ordinal in Tamaya was defined implicitly by the order being added. I think the ordinal concept here is very similar to that. Besides the configSrouces.xml is just a meta file and it will not be treated dynamically as config source property file and no changes will be taken into consideration. Hope this clarify your concerns on the ordinal.


Ondrej,
The Config APIs and the whole concept will work under all environment. See my explanation earlier. The other thing to note is that applications should NOT see any implementation classes. They should only import API packages and nothing more. Therefore, I don't think the extra constructor on the ConfigProvider is usable for any application. I hope the ordinal explanation above makes sense. Let me know if it is not clear.

By the way, I am fine if we think we should NOT support configSources.xml:o. It seems it caused more confusion rather than helping.

Wrener,
Interesting question. I will think more about whether to place ServiceProvider to SPI package.

I'll come back to this after new year! Please feel free to comment and contribute. If you would like to contribute towards implementation or tests, please feel free to do so.

My apologies if I have not understood your questions or concerns, please reiterate:o.

Thank you all for your great contribution so far! Let's work together to get this done!
Emily

[Ondrej Mihályi]

Hi Emily,

Thank you for the explanation. It makes sense why ordinals were proposed.

However, although the problem it tries to solve is clear, the solution with ordinals is really confusing.

Honestly, I looks like we couldn't find anything better. Any concept of magic numbers smells, and is not easy to follow.

Imagine, that you are going to add a new source. 

 - You would need to search for all the other source declarations to find out the proper ordinal value. 

 - If you accidentally clash with ordinals of other sources (you want to insert a source with ordinal 500 between two sources with 500 and 501), you need to modify one of the other source's ordinal. This is very error-prone.

 - as Anatole described, you may easily end up with sources declared in a different order than declared by ordinals, making it very hard to read the order of the sources applied at runtime.

I'm strongly in favor to describe exactly in points, what the ordinals try to solve. And then at least to try to find an alternative solution, without the above drawbacks. Even if we don't find a better solution, we would have an argument for using the ordinals, because currently it is very vaguely described.

I'll try to propose a PR with my concept of naming the sources, to fuel the discussion.

--Ondrej

Dňa piatok, 23. decembra 2016 1:26:10 UTC+1 Emily Jiang napísal(-a):

[Emily Jiang]

By the way, I like the idea of introducing an OPTIONAL dependency on CDI or any injection framework (JSR330).
In the ConfigProvider, I can introduce a constructor injection so that the ConfigProviderResolver can be injected  by the container. In OSGi environment, the Config integration will be achieved via the CDI extension pattern (registering the ConfigProvider bean). In other models, service loader pattern can be used. The only downside for non-Injection mode, the client needs to create an instance of ConfigProvider unnecessarily as there is no instance variable.

Emily

[Emily Jiang]

Hi Ondrej,

+1 on opening up a proposal. One more thing for the usage:
ConfigProvider.getBuilder().addConfigSource(source1, 10).addConfigSource(source2, 20).build()
They are in line and visible in one place. Let's discuss more on the issue so that people can follow the discussion.
Thanks
Emily

[Werner Keil]

All,

Not sure if you heard about it, but at least via the Tamaya dev list Anatole forwarded us, that Dmitry/Oracle is not planning to propose a Configuration JSR, at least not for Java EE 8.
As he leads 2 JSON JSRs now, it is somewhat understandable to focus on those and get them done for Java EE 8. For the overall EE roadmap and Oracle's credibility especially since the topic ranked 4th in the latest Java EE survey it may not look so good. Wondering if those who are in the JCP (e.g. Red Hat or IBM) and actively discussing things here could step in before Java EE 9 or Oracle feels it has enough capacity again?

After all it is a brand new topic where no JSR existed before. Thus if either IBM, Red Hat or say Tomitribe (other vendors like Payara are no JCP Members and while a co Spec Lead role could be possible for a Full Individual, too, examples from JSR 107 to 377 show, it is very hard if not impossible without at least one corporate vendor also backing it and helping as Spec Lead, that's also what we did in JSR 363) stepped up and were willing to propose a JSR sooner, there is no reason against that.

We should have a chance to discuss both Microprofile and Java EE in some detail during the London EC F2F. And a large number of people or companies involved here or on GitHub are also in the EC (IBM, Red Hat, Hazelcast, Tomitribe, JUGs SouJava and LJC as well as Ivar and myself) so if there was a consensus about something whether it's configuration, monitoring/metrics or another topic that was best suited for actual standardization in the JCP, there's no need to wait for Oracle.
And don't forget, the TCK situation would be different if IBM or Red Hat were individually or jointly Spec Leads of such a JSR. Patrick said there is no reason to believe, Oracle could open their commercial TCKs any time soon. And certain parts of even Java SE or "Professional Editions" being subject to license fees if you use them for your own business is another reason why we should not hope this will change. After all it is a revenue source for Oracle and with other areas (DB or "traditional" app servers and portals) shrinking they won't let those go away either.

Merry Christmas and let's hope we can get some of these areas going in 2017.
If not an integral part of Java EE 8, then a "module" usable either by a future version of Microprofile or new flexible profiles of Java EE 9 in a post-Jigsaw environment;-)

Werner

[Mark Struberg]

I agree with most of the answers Emily already wrote.

> The biggest issue with the ServiceLoader is that it is very minimalistic and it does not work in OSGI.

> I agree with Anatole that using the static ServiceLoader is an anti-pattern and it is hard to reuse in all contexts

ServiceLoader is not natively supported in OSGI gut ALL OSGi environments provide a special Service discovery hook for this standard mechanism.
So it _does_ work in OSGi.

We have to separate 2 things:
1.a.) how the SPI looks to the end user
1.b.) how the SPI does the lookup internally and how it finds the impl.

The SPI in 1.b) might be different depending on the environment. In Apache Geronimo we provide spec API jars which work perfectly fine even in OSGi, even if he Oracle ones do not. Still for the USER they are perfectly portably useable.

> <sources>...
That would totally defeat pluggability. Means whenever you like to just use a library provided by someone else you would have to change your own configuration.
Example: you write a CDI Extension which integrates Apache Oak and have the default config in a oak.properties. How would a user know that? This is reall similar to the jar hell we had before maven...

I agree that magic numbers are not perfect, but the closest you can get to a well working solution.
The hardcoded list of ConfigSources is a show stopper. We would not need any Config mechanism at all. Just add all your config we WEB-INF/web.xml then. You say that makes no sense? EXACTLY!

> The config tree can be regularly reevaluated for changing

What you propose is a mutable Configuration. That's exactly what got ruled out from day 1.
Config should be immutable regarding it's structure. Otherwise it's very hard to get it done in a well performing manner and it's also a pain to maintain it. Example: you got some weird behaviour in production and you cannot reproduce nor even judge where you get the values from. It's also not needed in practice.

> Refreshing may also change the ordinal of a config source,

No, changing the property at runtime would _not_ change the ordinal of the ConfigSource.

> ConfigBuilder.withSourcesBefore(String nameBefore, ConfigSource... sources);
> ConfigBuilder.withSourcesAfter(String nameAfter, ConfigSource... sources);

Wow, that's like in servlets, right? Did you ever work with that stuff? I mean in real world projects? It's totally broken. Why? Because you then need to know all the names of the web-fragments. Because you have to say after NAME or before NAME. In practice you will end up with most fragments saying after OTHERS. But who is the 'others'. In the end you'll end up with undefined ordering again :(

> - as Anatole described, you may easily end up with sources declared in a different order than
> declared by ordinals, making it very hard to read the order of the sources applied at runtime.

No, just add the configuration in the other ConfigSource then.
The nice point with ordinals and freely added ConfigSources is that you can ALWAYS override a configured value in a higher ConfigSource.

Oh and to give you some context: when the sytem with the Ordinals was designed I did not want to write a cool configuraiton system in the first place. I had real problems in real projects. And I just wanted to solve them in a good way. Please do always keep the real problems in your head when you think about any solution.

[Ondrej Mihályi]

> Oh and to give you some context: when the sytem with the Ordinals was designed I did not want to write a cool configuraiton
> system in the first place. I had real problems in real projects. And I just wanted to solve them in a good way. Please do
> always keep the real problems in your head when you think about any solution.

 - That's why I suggested to create a list of issues we want to solve, and then discuss if ordinals or any other solution is the best. Without any context, I can only imagine a small subset of real problems, and I see more problems with ordinals than without them.

In regards to ordinal vs. before/after, I realize that ordinals help ensuring that the application config will get higher priority than multiple configs in third-party libraries. With after/before, it would be necessary to define multiple names for before to ensure that the app config goes before both. However, pluggability may suffer too - if a new third-party library is added with higher priority ordinal, you need to increase the ordinal in your app. It may be better to define more exactly the implicit ordering set by order in xml/code, and by the origin if possible (application, library, etc.)

So the first usecase to solve would be:
 - how to easily define that a config source should get higher priority than multiple other sources (possible defined in a third-party dependency that can't be modified)

I suggest to have multiple competing solutions so that we may compare. I couldn't support a solution that is not backed by clear reasoning an is not easy to use. Ordinals have exactly that status right now. Saying that they are based on real world problems with any other solutions is just not enough.

--Ondrej

Dňa sobota, 24. decembra 2016 10:31:49 UTC+1 Mark Struberg napísal(-a):

[Mark Struberg]

Hi Ondrej!

We moved the discussions to private emails a long time ago to stop certain trolling. I don't like that, but it was the only solution to continue with productive discussions.
I'd be happy to move to a fully open list again, but that requires that people really stick to the topic and don't start trolling again.

I'd like to start with prefacing the technical discussion with a few formal things (copying all that stuff 1:1 from the mails)
----
As usual I like to split each topic into 4 categories. The categorisation is always personal of course:
[BLOCKER]: if something doesn't technically work or is not wide enough. Or if some important use case is not covered. Those are topics which we either must solve or you convince me that it's not really a blocker.
[COMPROMISE]: for topics someone thinks _should_ get solved, but can live if not (yet) addressed. Typically things like 'it works but it's imo ugly or too complicated'.
[DONTCARE]: Not perfect, but can live with it.
[ACK]: all fine and perfectly executed.
----

----
In my opinion why only should make certain design decisions, because
* we know the use case AND
* we know the background AND
* we know how it technically works AND
* we know the reason behind a certain design decision AND
* we know about the benefits of a certain design desision AND
* we of course also know about the downsides of a certain design decision AND
* we thus conclude that we like to include this feature.

We certainly should not make decisions just because 'someone else did it that way'.
----

Now for the question why I don't like the list of XML (or whatever) configured ConfigSources:
This brings us back to the days of EJB XML, Spring-XML and stuff. Anyone likes this still?
So what are the benefits of all those 'hardcoded' way to define things? You have all under your control! You can specify EXACTLY how it should look like.
What is the dowside? You HAVE to EXACTLY define how things should work. The app developer really needs to know all the subtle internals about how all the libraries are to be used and which Filter, Servlet, Listener etc he needs to configure in his web.xml (as an example).

Sticking with Sevlets, how did they solve this? You now have a way to dynamically use plug&play mechanism like Annotation driven activation (@WebSevlet, etc), ServiceLoader based dynamic ways (ServletContainerInitializer) plus you can override it in web.xml.

Coming back to Configuration, the problem is exactly the same. Here is the use case:
You have some container (and it's libs) and you have the developer building his own application.
But inbetween you often have many jars which don't know of either the container (means they need to be potrable across containers) nor the application.
This happens if you write some OSS project. Or if you have an inhous platform group which tprovides common tools for the various inhouse applications.
More general: it happens every time you write some reusable parts which need some configuration. There are tons of such reusable parts!

Those libs will provide a certain default configuration out of the box (by registring their own ConfigSource with an ordinal within the 'defaults' range of < 100). And most of the times the user don't need to think about it. But IF he wants to tweak the configuration, then he can easily just override the values in his own ConfigSource (because it has higher ordinal).

Another use case:
Say Payara (or any other EE server) likes to provide a special cluster mode. In this mode you have a single config source which shares the Config over the cluster (via DB or a common config server doesn't matter to the app)
The ordinal would be between the the defaults (<100) and the ENV and System (>=200). Let's say 170.
That would allow the Ops team to e.g. switch a REST endpoint in a single step without having to change the config on every single cluster node. And if it uses ConfigValue.cacheFor(5, TimeUnit.MINUTES) then you don't even need a restart.
All that is totally transparent to the application! The app (and even more important some reusable code parts) do NOT need to take any care about the existance of such a cluster-wide ConfigSource.

To give you the full picture: we of course have all the ConfigSources (with their Ordinals) and all the configured values available via JMX. And we even have a page where the Ops team can look at the effectively configured values of all apps and from which ConfigSource the effective value was taken.

[Mark Little]

Guys, let's keep everything public. Even before moving to Eclipse, when we started MicroProfile we had agreed this was an open group to encourage broad participation. We can deal with any "trolling" in alternative ways, but we shouldn't let anything drive us to private only.

Thanks,

Mark.

Sent from my iPad

> --
> You received this message because you are subscribed to the Google Groups "MicroProfile" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.
> To post to this group, send email to microp...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/8af09cf0-52dd-4334-9639-8244de4fa362%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Gunnar Morling]

> That's why I suggested to create a list of issues we want to solve, and then discuss if ordinals or any other solution is the best.

> Without any context, I can only imagine a small subset of real problems, and I see more problems with ordinals than without them.

+1

Is there a list of commonly agreed-on requirements somewhere? I remember Mark S. wrote up something, but I can't seem to find it anymore.

If there is no such list yet, we could take a Google doc, wiki page or similar, spend some time on gathering requirements so we can first reach some rough consensus on what exactly it is we want to achieve here. As others have sad, it seems we are in the solution space already, but apparently it seems different people have different visions of what is needed actually.

That said, I also have doubts about the notion of ordinals. In my picture, an application should be in control of the order of config sources and could easily do so by just specifying that order when bootstrapping the configuration. 3rd party libs used by the application could receive the effective list of configuration sources when being initialized by the application. In case of a packaged application where the source code can not be altered (e.g. a commercial product), the list of configuration sources could be passed using a simple JVM option or a defined config file.

So no ordinals would be needed really, but then this is just based on the agreement that I personally have in mind, and it may well be that it wouldn't suffice to satisfy the requirements of other people. Hence compiling a list of requirements should be the first step for diving into these solution details.

--Gunnar

[Ondrej Mihályi]

Hi Mark,

I completely get your point, you may save words for the explanation how ordinals solve a problem. I'm more interested in describing the problem which ordinals are supposed to solve. I proposed an alternative solution, which you probably misunderstood. But it's easy to misunderstand if we don't specify the problem and talk about different solutions. Maybe we are even trying to solve a different problem, so let's focus on that first.

One of the problems you are trying to solve with ordinals is extensibility. What else? What problems have you experienced with other solutions besides ordinals? Can you update the proposal with the list of usecases?

--Ondrej

[Mark Struberg]

The 

Am Sonntag, 25. Dezember 2016 22:40:22 UTC+1 schrieb Ondrej Mihályi:

One of the problems you are trying to solve with ordinals is extensibility. What else?

A plugin mechanism is another one. 

Use case: You have some functionality in a very basic or default behaviour. And you like to allow any framework user or application developer to replace the default solution with a slightly different one. Often this means you need to amend configured values. And sometimes you have to override them again in your application. And some other time the Ops team uses yet another value on different servers. The author of the original framework often has not the slightest how the config gets changed in prouction.

 

What problems have you experienced with other solutions besides ordinals? Can you update the proposal with the list of usecases?

a.) Name based with before and after: As already explained with the Servlets example: Any name based solution is imo doomed to fail as every lib needs to have information about the other ConfigSources. But  this totally defeats separation of concerns.

b.) A final list of ConfigSources defined by the applcation itself. Also as explained above: The app has no clue about any possible ClusterConfigSource or CloudConfigSource. The Ops team otoh has not enough info about what internal ConfigSources each application and framework used by those applications needs. 

If I understood your mail correctly then you did propose b.), right? Can you imagine building e.g. a library which implements a circuit-breaker as CDI Interceptor? This needs quite a few values to configure. Max response times, lowerbound connection threshold, higherbound, hysteresis numbers, a regexp to specify on which services the interceptor gets applied, etc. And of course you have no clue where your library ends up. How to configure the default values with your proposed solutions in a way that the devs and Ops team doesn't need to know any subtle details of your library?

LieGrue,

strub

[Werner Keil]

+1

Those bashing Oracle or its likes for lack of transparency should not start discussions in "private clubs" just because some may not always like to hear other people's opinions. 

JCP is perfectly transparent now thanks to reforms like JCP.next. Other ecosystems are equally transparent anld value the voice and opinion by everyone, whether you take Apache or Eclipse. Don't even want to bother Wayne or others on how Eclipse would view such private backroom whispering.

There is nothing I recall at Eclipse that is not publicly shared and transparent. Ok, you need to be logged in to see certain details like CQs and IP logs, but even that is transparent and visible to everyone. Disagreements in the past among even founding members were also discussed very much in the open, see https://www.cnet.com/news/eclipse-tells-ex-community-director-to-go-away-10406490/

Werner

[Werner Keil]

Guess the kind of trolling we should all be worried about are statements like those by Gartner about the death of Java EE 

Thanks to ex Red Hatter John Clingham http://johnclingan.com/2016/12/19/reanalyzing-the-state-of-java-ee/

(like his "Gang-of-Four" analogy, thought it may be a bit more now;-)

or InfoQ https://www.infoq.com/news/2016/12/Gartner-downgrades-Java-EE

this was quickly analyzed and responded to by Java EE Guardians and others who believe in Java and Java EE.

I pointed out, that even though some parts of Java EE 8 were dropped others especially those used by Microprofile are very active and alive.

The JSON-P EG listened to input from the community, including those who would like to introduce proprietary back room mingling here. 

Imagine the Spec Lead or Expert Group of JSRs like 374 acted in a similar manner;-)

Cheers,

Werner

[Emily Jiang]

Thanks guys for the good discussions while I switched off! I happened to log on to check my emails and add my 2cents.

As for the ordinal discussion, if the configure sources are added explicitly to ConfigBuilder, what Ondrej proposed should work. The only downside is that the end user cannot visibly view each config source ranking.

One +1 with ordinal is that it can be extended to support the configure sources discovered automatically via service loader per se and added to ConfigBuilder implicitly. This feature can not be provided by other solutions.

Remember: ordinal is not embedded in the configsource files but it is part of modeling.

Emily

[Reza Rahman]

You are perhaps more than a little correct on the "trolling" part, particularly with the comments. Indeed I at least mildly suspect now there is anti-Java EE "Internet shilling" going on from the usual parties that someone else was smart enough to alert me to (honestly I should have spotted the possibility myself right away).

The "shilling" started shortly after this message was posted here. Maybe that's just innocent coincidence and maybe it isn't...

Anyway, that entry on InfoQ is proving to be extremely well read. If you care to chime in, I suggest doing so. For any vendors, as usual I suggest clearly identifying one's self to avoid any appearance of impropriety...

--

You received this message because you are subscribed to the Google Groups "MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/985039d2-6d84-4f13-964a-f1882cd44b3c%40googlegroups.com.

[John Clingan]

Mea Culpa :-)

[Reza Rahman]

For the record, I am very thankful you and the Payara guys spoke up on this. That is the kind of vendor leadership that gives inspiration to the community. I am hoping the TomEE community will also make its voice heard and I am trying to see if Oracle can be convinced to do something about this sort of thing. I am pretty sure back in the day Sun wouldn't have just stood by idly in this sort of scenario.

--

You received this message because you are subscribed to the Google Groups "MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/87c56fa2-9b52-4188-9ee4-8bf5b60c5f09%40googlegroups.com.

[John Clingan]

I noticed that "default" sources do not include anything YAML. Is this because there is no formal recognition of YAML within Java EE? It's getting pretty hard to ignore YAML these days. For microservices, me-thinks developers would choose YAML over XML about 99% of the time.

[Werner Keil]

When this was last discussed and presented with Dmitry under an assumption, he may sooner or later propose a JSR we also heard, YAML was not officially supported by Oracle or its upcoming standard effort.

From what I remember it should be XML, JSON and existing formats like Properties.

It should be an optional module on top of core now, but Tamaya had YAML support for quite a while.

[Mark Struberg]

Reza, John at al. While I agree with what you wrote, I still ask you to move this kind of discussions to a new thread. Please let's keep this very thread purely technical. Otherwise it will be hard to follow. Especially any JSR or JCP discussions are totally offtopic and should not be a topic in this technial discussions.

txs and LieGrue,

strub

Am Sonntag, 1. Januar 2017 00:15:58 UTC+1 schrieb Reza Rahman:

For the record, I am very thankful you and the Payara guys spoke up on this. That is the kind of vendor leadership that gives inspiration to the community. I am hoping the TomEE community will also make its voice heard and I am trying to see if Oracle can be convinced to do something about this sort of thing. I am pretty sure back in the day Sun wouldn't have just stood by idly in this sort of scenario.

On Dec 31, 2016 6:09 PM, "John Clingan" <jcli...@redhat.com> wrote:

Mea Culpa :-)

On Wednesday, December 28, 2016 at 12:04:50 PM UTC-6, Werner Keil wrote:

Guess the kind of trolling we should all be worried about are statements like those by Gartner about the death of Java EE 

Thanks to ex Red Hatter John Clingham http://johnclingan.com/2016/12/19/reanalyzing-the-state-of-java-ee/

(like his "Gang-of-Four" analogy, thought it may be a bit more now;-)

or InfoQ https://www.infoq.com/news/2016/12/Gartner-downgrades-Java-EE

this was quickly analyzed and responded to by Java EE Guardians and others who believe in Java and Java EE.

I pointed out, that even though some parts of Java EE 8 were dropped others especially those used by Microprofile are very active and alive.

The JSON-P EG listened to input from the community, including those who would like to introduce proprietary back room mingling here. 

Imagine the Spec Lead or Expert Group of JSRs like 374 acted in a similar manner;-)

Cheers,

Werner

--
You received this message because you are subscribed to the Google Groups "MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

[Mark Struberg]

Hi John!

Whether to support YAML and JSON out of the box is still an open question. 

It's mostly a matter of weighting the pros and cons. Of how much use will it be vs how much code or dependencies does it add? 

In any case support for both (and any other format) can easily get added in a portable way anyway.

As an example: do we like to support a JSON based config? 

pro: JSON is nice

con: JSON doesn't support comments (at least not portably), you need an additional dependency, etc

If people prefer to write their config in YAML then they can easily add a YAML config source themselves.

Got my point?

LieGrue,

strub

[James Carman]

I wouldn't make YAML (or JSON) a requirement, but provide extension API to allow folks to roll their own.  

--


You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.


To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.


To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/b44110a8-6d61-4e28-99d3-6cd303458108%40googlegroups.com.

[Werner Keil]

That's about how Tamaya does it on top of an SPI/API.

Unless efforts here are considered "incubator only" and everyone would rather wait for Oracle to propose a standard, should there be an RI, I guess it's up to the EG and Spec Lead how many formats can be supported. Of course other implementations Open or Closed Source may support anything they feel appropriate.

[Mark Struberg]

Werner, I have no clue why you keep going on with mentioning other projects? This thread is about TECHINCAL questions of this very microprofile proposal. The currently proposed ConfigSource mechanism of this very spec in discussion already provides all that.

[Emily Jiang]

Yes James. Mark made the exact point that an end user can provide the YAML config source themselves. I think we all agree that YAML reader is not provided out of box but an end user can write their own reader.

Let's not worry about JSR for now as we can try to standardise in MicroProfile,io and try to feed into JSR in the future. The current propose is to standardise the API in MicroProfile.io and then focus on the implementation and tests.

Thanks
Emily

[James Carman]

Yep, totally agree there, but one stipulation I would make is to design the API with extensibility in mind.  I haven't had a chance to review the proposed APIs (if there are any), but I was going to dig in a bit this week to see what folks have put together.  Have we taken a look at Apache Commons Configuration at all?  They've (perhaps "we've" is a better way to phrase that) had pluggable configuration sources for a long time.  Might be worth a look for inspiration at least.

James  

--

You received this message because you are subscribed to the Google Groups "MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/14eda7dc-9745-4907-8058-c905c159b6c3%40googlegroups.com.

[Werner Keil]

I did not ask about YAML.

And everybody does that, Emily repeatedly mentioned looking at Archaius, so stop trolling around and stick to the FAX maam, please ;-)

[Gunnar Morling]

> I think we all agree that YAML reader is not provided out of box but an end user can write their own reader. 

I'm not convinced yet that YAML shouldn't be supported out of the box. At least it should be considered conceptually right from the beginning, esp. its support for lists and dictionaries. It should be clear how these would be targeted and exposed through the config API.

So even if not providing a YAML source by default, one should be implemented as part of this work in order to make sure the API and SPI concepts line up with such more powerful configuration sources.

But then, if YAML is what people use these days for config most of the time, I don't see why we should put the burden onto users to implement their own reader.

> The current propose is to standardise the API in MicroProfile.io and then focus on the implementation and tests.

Not sure whether that's what you wanted to say, but I think these three things must go hand in hand, i.e. iteratively happen at the same time. Otherwise it's too easy to "spec" something which cannot be implemented or is lacking when using it (as found out during testing).

--Gunnar

--

You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.

To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/14eda7dc-9745-4907-8058-c905c159b6c3%40googlegroups.com.

[Werner Keil]

I hope nobody gets offended if we discuss "other projects" here as you mentioned another Apache project like Commons Config?;-)

I used it on several occasions in large scale customer projects. And based on external references or download stats it must be among the most widely used solutions out there trying to reuse and somewhat "standardize" the idea of configuration.
It was quite cumbersome and complex to use in 1.x. Some of that seems to have improved with the 2.x version, but it is still a single JAR. No modularity nor a notion of remote data sources. It does support a whole lot of data formats (neither YAML nor JSON are mentioned in https://commons.apache.org/proper/commons-configuration/ though) including the system registry on Windows or NEXTstep (some of which seems to have survived on Mac OS)

Archaius which Emily and others mentioned earlier wraps around Apache Commons Config by default (1.x it could also offer both versions now) but on an API level offers an abstraction, so backing it by Typesafe/Lightbend Config instead is also an option, and Archaius mentions Scala support, too.

While some prefer to ignore that or phantasize otherwise, there are at least 3 projects created with Distributed Configuration sources and servers in mind at the moment:
 - Netflix Archaius
 - Apache Tamaya
 - Spring Cloud Config

Tamaya is still a relatively "New Kid" and also still in Apache Incubator, but if you follow the regular hangouts and minutes or mailing lists, you'll get an idead that most of the active committers have one or more large clients (mostly Germany and Switzerland) where Tamaya in its current state is used in production. Spring is also used by its customers of course. And there should be others than Netflix internally using Archaius.

Werner

[Werner Keil]

If something is modular by design which pretty much all projects other than Apache Commons Config are (see http://jcenter.bintray.com/com/netflix/archaius/ for Archaius) then it's up to those using it whether they need YAML, Properties files or the Windows Registry ;-)

Werner

On Tuesday, January 3, 2017 at 12:05:51 PM UTC+1, Gunnar Morling wrote:

> I think we all agree that YAML reader is not provided out of box but an end user can write their own reader. 

I'm not convinced yet that YAML shouldn't be supported out of the box. At least it should be considered conceptually right from the beginning, esp. its support for lists and dictionaries. It should be clear how these would be targeted and exposed through the config API.

So even if not providing a YAML source by default, one should be implemented as part of this work in order to make sure the API and SPI concepts line up with such more powerful configuration sources.

But then, if YAML is what people use these days for config most of the time, I don't see why we should put the burden onto users to implement their own reader.

> The current propose is to standardise the API in MicroProfile.io and then focus on the implementation and tests.

Not sure whether that's what you wanted to say, but I think these three things must go hand in hand, i.e. iteratively happen at the same time. Otherwise it's too easy to "spec" something which cannot be implemented or is lacking when using it (as found out during testing).

--Gunnar

2017-01-02 23:55 GMT+01:00 'Emily Jiang' via MicroProfile <microp...@googlegroups.com>:

Yes James. Mark made the exact point that an end user can provide the YAML config source themselves. I think we all agree that YAML reader is not provided out of box but an end user can write their own reader.

Let's not worry about JSR for now as we can try to standardise in MicroProfile,io and try to feed into JSR in the future. The current propose is to standardise the API in MicroProfile.io and then focus on the implementation and tests.

Thanks
Emily

On Monday, January 2, 2017 at 7:21:57 PM UTC, Mark Struberg wrote:

Werner, I have no clue why you keep going on with mentioning other projects? This thread is about TECHINCAL questions of this very microprofile proposal. The currently proposed ConfigSource mechanism of this very spec in discussion already provides all that.

--
You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.

To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

[Werner Keil]

Sorry not to have mentioned cfg4j http://www.cfg4j.org/ in that list earlier.

Last but not least the term "ConfigurationSource" looks heavily inspired by cfg4j. And it shows a few other interesting facettes, e.g. a ConfigSource that's automatically monitored by Dropwizard Metrics ;-)

[Anatole Tresch]

Hi all

I put together some of my thaughts, why I do not like the ordinal concept on my blog here,:

https://maketechsimple.wordpress.com/2017/01/04/composing-configuration-sources/ 

Summarizing I think ordinals have too many downsides and I consider them only as one implementation of a more wider concept: A mapping function of ConfigSources to a Configuration. Even the fact that I have a well defined unidirectional tree of sources is a design decision. Even, when I define that a configuration value is determined by walking along a well ordered list of config sources, the ordering is not necessaRILY AUTOMATICALLY APPLIED:

1) When using a Builder for defining my Configuration, I explicitly define my config source ordering. NO AUTOMATISM AT ALL.

2) When collecting config sources as they are registered, I can evaluate some ordinal for each of them to determine the order. This does not require the ordinal to be defined on the ConfigSource API, I can access it as an ordinary config value from the Config Source.

3) When I have a use case, where I want to have an additoinal criteria in place, with only ordinals I can get into an big mess. Imagine I want to have my configuration sources not only ordered by their ordinal, but also by some other criteria, such as a grouing. E.g. there are use cases, where file based configuration must be handled differently than classpath based conifig, but there are also different priorities (default and explicit values) within each configuration group. In this case ordinals are not sufficient anymore and I end up with hyper complex logic to make these things working by wrapping everything, so I can define all ordinals as needed at the end of my ConfigSource collection phase. Nobody wants to do that.

All these aspects can be solved by

- defining a Builder, which by default does not sort the sources added.

- let the builder perform sorting on its config sources by passing the Comparator to be applied. This could be the one using some ordinal values calculated, or a more complex logic).

- the builder supports adding property sources explicitly, or loading the automatically registered at the end of the current chain (also not performing any sorting implicitly). This also allows to define parts of the configuration setup programmtically, but still using the default loading logic for registered config sources.

See Tamaya's ConfigurationContextBuilder as an example for this. BTW the metamodel module in Tamaya's sandbox basically is nothing more than an XML based configuration that is backed by this kind of Builder apporach. It allows as well to add preregistered config sources, but also allows to explicitly hard code property sources being added.

Am Samstag, 31. Dezember 2016 00:00:21 UTC+1 schrieb Emily Jiang:

Thanks guys for the good discussions while I switched off! I happened to log on to check my emails and add my 2cents.

As for the ordinal discussion, if the configure sources are added explicitly to ConfigBuilder, what Ondrej proposed should work. The only downside is that the end user cannot visibly view each config source ranking.

Why not? We can query the current configuration and provide that information, e.g. via a Java and REST API. As long as the Java API allows access to the underlying ConfigSources in order, this information is still accessible. 

One +1 with ordinal is that it can be extended to support the configure sources discovered automatically via service loader per se and added to ConfigBuilder implicitly. This feature can not be provided by other solutions.

See my blog. This feature is only one way of defining the order of sources and has severe deficencies IMO. I agree that in many cases it works perfectly, but I dont think it should be baked into the API. My blog mentioned above explains why. And Tamaya also shows how the oirdinal feature can be implemented as only one of many ordering policies possible.

[Anatole Tresch]

And in ten years another format may be on vogue. Tamaya as well as commons-config solves this by defining an intermediate representation that can be mapped to one or more configuration sources, making formats a pluggable aspect of configuration. Also the mapping of YAML to this intermediary format can be changed by registering an SPI of the formats module in Tamaya.

[Mark Struberg]

Anatole, who creates the ConfigurationBuilder? And how does a 3rd party lib in an application get to this very configuration?

[Gunnar Morling]

Hi Mark,

I suggested something on that very point a few days ago. In case it got lost:

===

In my picture, an application should be in control of the order of config sources and could easily do so by just specifying that order when bootstrapping the configuration. 3rd party libs used by the application could receive the effective list of configuration sources when being initialized by the application. In case of a packaged application where the source code can not be altered (e.g. a commercial product), the list of configuration sources could be passed using a simple JVM option or a defined config file.

===

I find such explicit ordering and passing much easier to grasp than relying on ordinals scattered across different places.

--Gunnar

2017-01-04 11:23 GMT+01:00 Mark Struberg <markst...@gmail.com>:

Anatole, who creates the ConfigurationBuilder? And how does a 3rd party lib in an application get to this very configuration?

--
You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.

To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/34139c71-2e00-418f-875f-f6518097e8cf%40googlegroups.com.

[Anatole Tresch]

Basically the ConfigurationBuilder is a separate part, using Java 8 something like

public interface ConfigurationBuilder{

   ...

   static ConfigurationBuilder create(){...}

   Configuration build();

}

It allows to use the Config API similar to what commons-config et al are offering. Obviously building a configuration does not make it accessible. To make it the current default configuration, it must be applied, e.g. by something like:

Configuration.setCurrent(ConfigurationBuilder.create()

   .addConfigSources(new EnvPropertiesSource(), new SysPropertiesSource, new EtcSource())

   .build());

So we have decoupled the concerns of configuration creation from making it accessible, which IMO is a very important aspect.

J Anatole

2017-01-04 11:23 GMT+01:00 Mark Struberg <markst...@gmail.com>:

Anatole, who creates the ConfigurationBuilder? And how does a 3rd party lib in an application get to this very configuration?

--
You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/34139c71-2e00-418f-875f-f6518097e8cf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Anatole Tresch

PPMC Member Apache Tamaya

JCP Star Spec Lead

Switzerland, Europe Zurich, GMT+1

maketechsimple.wordpress.com 

Twitter:  @atsticks, @tamayaconf

[Anatole Tresch]

Hi Gunnar

+1. I see multiple use cases, which all have to be covered by a uniform configuraition solution IMO:

1. Building a Configuration for use in whatever context that makes sense, under fully control of the developer that created that Configuration instance. This matches also what you describe, where the Configuration managed explicitly and passed to the components explicitly.
2. We access configuration as a shared resource through a well defined access point. This maps to the ServiceLocation pattern, which also can be implemented more decoupled using IoC/injection mechanisms.
1. Automatically initialized: loading of the configuration sources is done fully automatically by whatever strategy that is useful for a given use case.
2. Explicitly initialized: the configuration is managed explicitly by some custom/application bootstrap code and then made accessible through the shared access point.

One way this can be achieved is 

• by defining a ConfigurationBuilder, which implements 1.
  
• by letting the shared Configuration being settable, e.g. by Configuration.setCurrent(Configuration), implementing 2.2 (in combination with 1).
• by defining the default loading strategy, if no explicit initialization was triggered, implementing 2.1 (the internal implementation could still rely on 1).

If you did not see it have a look at my critical blog about ordinals here: https://maketechsimple.wordpress.com/2017/01/04/composing-configuration-sources/

J Anatole

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CADGJaX97Q0-Xb242rs-iP8RKTPMpk%3Dt3UKr1F9J9e-DJnEztcg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Werner Keil]

Gunnar/all,

cfg4j which I mentioned yesterday answering e.g. which offer YAML support, it also follows a programmatic approach.
Where so called "fallback" sources can be chained defining the order in which they are used.

Werner

[James Carman]

I don't like the idea of statics.  Can we try to avoid them?  They can cause all sorts of issues.

On Wed, Jan 4, 2017 at 5:45 AM Anatole Tresch <atst...@gmail.com> wrote:

Basically the ConfigurationBuilder is a separate part, using Java 8 something like

public interface ConfigurationBuilder{

   ...

   static ConfigurationBuilder create(){...}

   Configuration build();

}

It allows to use the Config API similar to what commons-config et al are offering. Obviously building a configuration does not make it accessible. To make it the current default configuration, it must be applied, e.g. by something like:

Configuration.setCurrent(ConfigurationBuilder.create()

   .addConfigSources(new EnvPropertiesSource(), new SysPropertiesSource, new EtcSource())

   .build());

So we have decoupled the concerns of configuration creation from making it accessible, which IMO is a very important aspect.

J Anatole

2017-01-04 11:23 GMT+01:00 Mark Struberg <markst...@gmail.com>:

Anatole, who creates the ConfigurationBuilder? And how does a 3rd party lib in an application get to this very configuration?

--


You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.


To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.

To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/34139c71-2e00-418f-875f-f6518097e8cf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Anatole Tresch

PPMC Member Apache Tamaya

JCP Star Spec Lead

Switzerland, Europe Zurich, GMT+1

maketechsimple.wordpress.com 

Twitter:  @atsticks, @tamayaconf

--

You received this message because you are subscribed to the Google Groups "MicroProfile" group.

To unsubscribe from this group and stop receiving emails from it, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/CAFJiDbcbnwkfuMmaXDCpD5MHhSGDxWzUnMFd5kU0F7%3DWFR8SMg%40mail.gmail.com.

[Mark Struberg]

@Gunnar:

> 3rd party libs used by the application could receive the effective list of configuration sources when being initialized by the application.

How? Elaborate on the details please. How do the 'Configuration Subsystem' know about all it's users? To me that sounds like reversing the scanning from automatically picking up the producers to automatically picking up the consumers, right? But it still requires some scanning in that case...

> I find such explicit ordering and passing much easier to grasp than relying on ordinals scattered across different places.

Note that the ordering picked up via 'scanning' is only a default and can be changed by the application anyway.

> the list of configuration sources could be passed using a simple JVM option or a defined config file.

you know how good that works with JVM properties? What do you do in case of 3 WAR files running on the same container? Ever tried to do that for jul? How would you solve the problem with 3 WAR files on a small EE server?

@Anatole
You didn't answer my question.
Let me reiterate:
1.a.) What visibility or 'scope' does such a produced configuration have?
1.b.) Who is in charge to assemble the configuration and how do you ensure that there is only one?
1.c.) How do you know that randomlib.jar needs some configuration? Sounds very error-prone.

@James

> I don't like the idea of statics.

If there is some 'factory' involved then I know no other way. Directly or indirectly (via a factory provider) is only an implementation detail.
How to avoid that?

All those proposals in the end lead to *not* having one 'Application Configuration' but each lib has to roll/assemble it's own.
That's not user friendly and the reason why the existing libraries have not been used a lot.

The only solutions which got pretty wide adoption have been DeltaSpike Config and Spring Config.
And both feature an out-of-the-box auto-discovery of ConfigSources.

LieGrue,
strub

[James Carman]

 I guess I am not really sold on the whole idea of standardizing configuration. How often do people actually pull configuration information directly from an API? Most people will use their container ( whatever that is) to inject configuration values into their objects.   I can understand perhaps creating some standard "reader" API, but standardizing on how to build configuration data seems a bit silly to me.   If somebody really wants to have access to that reader API, they can inject it into themselves. 

--

You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.

To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/40b38b11-37d0-4cd9-9e35-cceb5eb596a2%40googlegroups.com.

[Mark Struberg]

> I can understand perhaps creating some standard "reader" API

Yes, that's the main purpose. And you'll see that the actual API only consists of 2 or 3 interfaces.
The rest (5 interfaces) is SPI for extending the default behaviour.

Nonetheless 'reading' actually means 2 steps:
1.) reading the configured values in your code
2.) configure your project values either within your project (default values, etc) or by the Ops team

The main benefit of DeltaSpike and Spring config systems is exactly that any 'user' does not need to care where the configured values come from or whether they get 'overridden' somehow.

[Gunnar Morling]

2017-01-04 13:52 GMT+01:00 Mark Struberg <markst...@gmail.com>:

@Gunnar:

> 3rd party libs used by the application could receive the effective list of configuration sources when being initialized by the application.
How? Elaborate on the details please. How do the 'Configuration Subsystem' know about all it's users?

Not the "configuration subsystem" does, but the application does know its components. E.g. your application uses JPA/Hibernate, so it can pass the config (or config builder if you want) when bootstrapping JPA.

E.g. the application could define the config and the order of sources it wants using a callback class like this:

    class MyConfigBuilder implements ConfigBuilder {

        void buildConfig(Context ctx) {

            ctx.addFileSource("myconfig.properties")

                .addSystemPropertiesSource()

                .addEnvVariablesSource();

        }    

    }

Then when e.g. bootstrapping JPA, simply pass the callback (or its FQN) into the JPA property map or specify it in persistence.xml. Same for other libs you use.

If you think that's too cumbersome, it could be made discoverable via the service loader so that components can look it up (exactly one such callback may exist).

Or the config API could standardize on a file such as META-INF/config.xml containing that information (either the config sources directly, or the FQN of such builder callback) and let you get access to it a la Persistence#createEntityManagerFactory() or registration as a CDI bean etc.

 

To me that sounds like reversing the scanning from automatically picking up the producers to automatically picking up the consumers, right? But it still requires some scanning in that case...

My main point is that there should be single "point" of declaring the config sources and their order. As described above, there are several ways then to let users of the config API get hold of that information.

Generally, I'm less and less a fan of "magically" looking for stuff and somehow applying it adhering to some mechanism that's hard to reason about. Instead I'm more preferring a simple, potentially a tad more verbose yet much more comprehensible, explicit wiring of things these days. As we are talking about microservices here, I'm not expecting a plethora of libs and components thrown into the mix, but rather a small set of components which can be reasonably handled in such way.

> I find such explicit ordering and passing much easier to grasp than relying on ordinals scattered across different places.
Note that the ordering picked up via 'scanning' is only a default and can be changed by the application anyway.

> the list of configuration sources could be passed using a simple JVM option or a defined config file.
you know how good that works with JVM properties? What do you do in case of 3 WAR files running on the same container? Ever tried to do that for jul? How would you solve the problem with 3 WAR files on a small EE server?

Personally I don't think that such shared deployment style should be done, nor is it what I hear people recommending for microservices (or any kind of architectures, actually). One app, one container (if you even have a container at all).

But even if you share a container between several apps, each of them could have - for instance - their own META-INF/config.xml describing its config sources. Just as persistence.xml, validation.xml or other (EE) descriptors. Personally, I've never had problems with these.

The remaining case I can see is where you deploy several instances of one and the same "product" (i.e. something you cannot modify and add stuff to it) and need different configs for those. I don't have an answer to that out of my head. Is it a case common in the context of microservices, though? I I'm inclined to say no and thus wouldn't see this case worth supporting, but that's just my take on it. Seems we are circling back to the fact that there is no list of requirements that's finally agreed upon.

@Anatole
You didn't answer my question.
Let me reiterate:
1.a.) What visibility or 'scope' does such a produced configuration have?
1.b.) Who is in charge to assemble the configuration and how do you ensure that there is only one?
1.c.) How do you know that randomlib.jar needs some configuration? Sounds very error-prone.

@James
> I don't like the idea of statics.
If there is some 'factory' involved then I know no other way. Directly or indirectly (via a factory provider) is only an implementation detail.
How to avoid that?

All those proposals in the end lead to *not* having one 'Application Configuration' but each lib has to roll/assemble it's own.
That's not user friendly and the reason why the existing libraries have not been used a lot.

The only solutions which got pretty wide adoption have been DeltaSpike Config and Spring Config.
And both feature an out-of-the-box auto-discovery of ConfigSources.

LieGrue,
strub

--
You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/40b38b11-37d0-4cd9-9e35-cceb5eb596a2%40googlegroups.com.

[Werner Keil]

It's also important to have ways of propagating and consuming configuration especially if your system gets broken down into many, sometimes hundreds of "microcontainers" instead of having everything in a single container calling e.g. JNDI or the local file system of just a handful or large servers.

The idea was sometimes called "DevOps JSR" in the past. Whether you have a JSR or widely accepted projects like Archaius is not ultimately important, but if containers and vendors want interoperability, so the same app can be deployed to TomEE, Wildfly, Payara/Glassfish or any other container that claims to be compatible, then having an actual standard and API certainly helps.

Werner

[Emily Jiang]

Thanks guys for your valuable comments!
Defining API is definitely needed. +1 to Mark and Werner's comments!

ConfigSource ordering discussion:
1. implicit ordering
2. ordinal

For the user case of programmatically adding ConfigSources to the ConfigBuilder. Both solutions are fine. I think this covers most user scenarios.

When the config sources are loaded automatically via serviceLoader pattern, using ordinal is sufficient to order the config sources.
Without ordinal, with Anotole's approach, a comparator needs to be registered to order the config sources, which adds more work but solves the problem. I see with the Comparator approach, more complex logic can be used to sort the config sources.

It seems both solutions can solve the ordering issues. If I didn't miss anything, I would rather not to bake the ordinal in the APIs to keep the APIs simple, at least for the first version. There are a few unanswered questions posted by Mark.
@Anatole

1.a.) What visibility or 'scope' does such a produced configuration have?
1.b.) Who is in charge to assemble the configuration and how do you ensure that there is only one?
1.c.) How do you know that randomlib.jar needs some configuration? Sounds very error-prone.

@James
> I don't like the idea of statics.
If there is some 'factory' involved then I know no other way. Directly or indirectly (via a factory provider) is only an implementation detail.
How to avoid that?

=============
Gunner,
1. On YAML support: I think the first version of Config api will not support out of box, as it can be easily extended by developers. We can consider it in the next version.
2. Config testing and implementation, I agree what you they will go hand in hand with APIs. The APIs are not final until implementation and tests are ready.
==============

Are there any other issues with the APIs? I would like to encourage the config projects, e.g. DeltaSpike config, Tamaya, etc, to implement the APIs. Thoughts?

Thanks
Emily

[Gunnar Morling]

Emily,

> 1. On YAML support: I think the first version of Config api will not support out of box, as it can be

> easily extended by developers. We can consider it in the next version.

Putting the question whether to support it out of the box at the side for a minute, how would such an extension look like? E.g. YAML allows for nested values:

    connection:
        url: jdbc:mysql://localhost:3306/db
        poolSize: 5

How does the proposed API/SPI deal with that? How would a user access the url value? Dot syntax ("connection.url") may be one idea, but is this described somewhere?

Then YAML has lists and hashes:

    protocols:
       - http
       - https

    users:
        tom: passwd

        bob: passwd

How would these be handled? ConfigSource defines the method Map<String, String> getProperties(). How would this one be implemented for lists and hashes? To me it seems not like a good match.

How would a user access lists/hashes? Currently I could only do

    config.getValue( "protocols", List.class )

resulting in ugly raw type usage and unchecked casting for the user. So you may need to foresee something like Java EE's TypeLiteral (or simply use that one):

    config.getValue( "protocols", new TypeLiteral<List<String>>{}() )

This kind of question is why formats like YAML cannot be left out of the picture entirely. Even when not supporting them OOTB, there really needs to be at least a PoC-level implementation in order to make sure the APIs/SPIs fit. Currently, it seems they don't.

--Gunnar

--

You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/4e03f335-ddde-4066-b9b0-5e7a5bbf5b32%40googlegroups.com.

[Anatole Tresch]

Thanks Emily for winding up things, here some comments and answers to the unanswered questions from Mark...

J Anatole

2017-01-04 22:56 GMT+01:00 'Emily Jiang' via MicroProfile <microp...@googlegroups.com>:

Thanks guys for your valuable comments!
Defining API is definitely needed. +1 to Mark and Werner's comments!

ConfigSource ordering discussion:
1. implicit ordering
2. ordinal

For the user case of programmatically adding ConfigSources to the ConfigBuilder. Both solutions are fine. I think this covers most user scenarios.

When the config sources are loaded automatically via serviceLoader pattern, using ordinal is sufficient to order the config sources.
Without ordinal, with Anotole's approach, a comparator needs to be registered to order the config sources, which adds more work but solves the problem. I see with the Comparator approach, more complex logic can be used to sort the config sources.

​We can define a default Comparator that actually uses ordinal semantics. Given that for "most" cases the user does not have to do additional work, but still has the flexibility if needed.

​

It seems both solutions can solve the ordering issues. If I didn't miss anything, I would rather not to bake the ordinal in the APIs to keep the APIs simple, at least for the first version. There are a few unanswered questions posted by Mark.
@Anatole

​+1​

 

1.a.) What visibility or 'scope' does such a produced configuration have?

​That is out of scope of the configuration framework. The created instance is completely decoupled (a good thing), unless

it is applied as current shared configuration accessible via Configuration.getInstance(). As mentioned creating a configuration and making it accessible are two separate concerns not to be mixed up IMO. This gives the flexibility to use configuration logic in any scope as needed by the current usage scenario.

​

 

1.b.) Who is in charge to assemble the configuration and how do you ensure that there is only one?

​There is no need that there is only one configuration. Dont constrain users what they can do with config and it can be used. Said that, the commonly shared instance is managed by the SPI implementation backing the Configuration's static accessor methods., so this implementation defines the lifecycle anhd scope of configuration. Also here even in a Java EE context there is not necessarily one, since it could also be dependent on the scope, e.g. when running multiple applications in one container ;)

 

1.c.) How do you know that randomlib.jar needs some configuration? Sounds very error-prone.

​Honestly said I dont know, unless the jar uses annotations for injection and I would scan for them. But this is not an issue, it is simply the reality also a library programmer has to deal with. Concerning thath

  - the library should be implemented in a resilient way, meaning providing defaults to be used, if no config for the given keys are present. If the configuration is really mandatory it is OK for me to throw an error. That is a design decition of the library implementors not a concern of the config system.

  - we could define a location in the classpath, where we will always lookup default configuration, e.g. META-INF/config/* and the config system by default always considers this files with minimal priority. Nevertheless thinking of single jar deployments where everything is baked together this is not always working reliably, so the solution before IMO is the way to go here. 

@James
> I don't like the idea of statics.
If there is some 'factory' involved then I know no other way. Directly or indirectly (via a factory provider) is only an implementation detail.
How to avoid that?

​Statics are not per se bad, it is just a very broad scope. And it is the only way to make something globally accessible in all known of runtime scenarios in a uniform way. And also remember that the SPIs backing the statics can be more intelligent, so static access does not imply static behaviour.​

=============
Gunner,
1. On YAML support: I think the first version of Config api will not support out of box, as it can be easily extended by developers. We can consider it in the next version.

​One of the non trivial things to discuss is how to map YAML collection types to normalized properties. Tamaya basically does this using a very (probably too) simplistic approach, by just combining all levels with a '.' separator into a key. Obviously this is not working for certain scenarios, Therefore the mapping can be customized as needed AFAIK.​

2. Config testing and implementation, I agree what you they will go hand in hand with APIs. The APIs are not final until implementation and tests are ready.
==============

​We will for sure not deliver any implementation in Tamaya without testing them first, let's see if we can reuse some tests​ then for the "spec"...

Are there any other issues with the APIs? I would like to encourage the config projects, e.g. DeltaSpike config, Tamaya, etc, to implement the APIs. Thoughts?

​I will look into the APIs later intensivly. If so, we can easily add them into the next Tamaya release 0.3-incubating, which is planned in about 2 weeks to be ready. If not I will raise up my concerns ;) Thanks for the work done so far.

​

 

[Anatole Tresch]

Hi Gunnar

I will give you some insights, what we have done in Tamaya related to some of your points...

2017-01-05 9:20 GMT+01:00 Gunnar Morling <gun...@hibernate.org>:

Emily,

> 1. On YAML support: I think the first version of Config api will not support out of box, as it can be

> easily extended by developers. We can consider it in the next version.

Putting the question whether to support it out of the box at the side for a minute, how would such an extension look like? E.g. YAML allows for nested values:

    connection:
        url:

​​

jdbc:mysql://localhost:3306/db
        poolSize: 5

How does the proposed API/SPI deal with that? How would a user access the url value? Dot syntax ("connection.url") may be one idea, but is this described somewhere?

​That is exactly what currently Tamaya does, so it ends up in

connection.url=​

​

jdbc:mysql://localhost:3306/db

connection.poolSize=5

Then YAML has lists and hashes:

    protocols:
       - http
       - https

    users:
        tom: passwd

        bob: passwd

How would these be handled? ConfigSource defines the method Map<String, String> getProperties(). How would this one be implemented for lists and hashes? To me it seems not like a good match.

How would a user access lists/hashes? Currently I could only do

    config.getValue( "protocols", List.class )

resulting in ugly raw type usage and unchecked casting for the user. So you may need to foresee something like Java EE's TypeLiteral (or simply use that one):

    config.getValue( "protocols", new TypeLiteral<List<String>>{}() )

Basically this is the question of how to support collections. Tamaya has a collections extension, which handles the problem as follows:

1. Tamaya has a TypeLiteral<T> as part of the API.
2. Tamaya knows special entries, so one can add additional metadata to every key/value pair. Hereby we use the metadata notation from etcd, where metadata entries start with the same key, prefixed with a '_', so your example could be mapped to: 

protocols=http,https

_protocols.type=List

users=[tom:passwd][bob:passwd]

_users.type=Map

// Customoizing default separator syntax

#_protocols.item-separator=,

#_users.item-separator=[]

#_users.value-separator=:

 

​So the question is how we deal with/support metadata, a Pandorra's box, but an important aspect not neglectable. Currently accessing a config value returns a single String value from a ConfigSource.​ In Tamaya the (so called) PropertySources do in fact not return a String but a PropertyValue, which in fact returns a Map<String,String> of entries for the accessed key, including metadata entries. Beside collection support metadata entries are also used to track things like the origin (eg _key.origin=c:/config/GLOBAL/app.properties) of an entry (in case of error analysis) or to forward metadata from the data provider, e.g. etcd/elasticsearch TTL values. Easily it could also be used, e.g. for constraining entries for certain roles to be visible and more ...

[Werner Keil]

Gunnar,

I assume the examples are just for illustration?

Under no circumstances any sane project would put plain text passwords into config files, regardless if it uses YAML, JSON, XML or another config source.
Most real life projects I come across use SSH certificates instead, those that don't or can't at least encrypt them.

If management of passwords and secrets makes enough progress in JSR 375 I guess that's an interesting point of synergy where another Java Standard could be used and recommended for cases where passwords do end up in configuration sources ;-)

Werner

To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

[Gunnar Morling]

Sure, that's just for illustration.

The key question is how to deal with list/map style configuration values as enabled by more powerful formats such as YAML. We need to make sure the API/SPI is prepared for this, also if not supporting such format OOTB.

To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/29dba4f8-746c-42cc-8138-b642615e6314%40googlegroups.com.

[Mark Struberg]

@Anatole:

>> 1.a.) What visibility or 'scope' does such a produced configuration have?
​> That is out of scope of the configuration framework. The created instance is completely decoupled (a good thing), unless
> it is applied as current shared configuration accessible via Configuration.getInstance(). As mentioned creating a configuration
> and making it accessible are two separate concerns not to be mixed up IMO. This gives the flexibility
> to use configuration logic in any scope as needed by the current usage scenario.
​
>> 1.b.) Who is in charge to assemble the configuration and how do you ensure that there is only one?
​> There is no need that there is only one configuration. Dont constrain users what they can do with config and it can be used.
> Said that, the commonly shared instance is managed by the SPI implementation backing the Configuration's static accessor methods.,
> so this implementation defines the lifecycle anhd scope of configuration. Also here even in a Java EE context there is not necessarily one,
> since it could also be dependent on the scope, e.g. when running multiple applications in one container ;)

In sentence 1 you say that providing a 'scope' for a Configuration is not in the responsibility of a Config impl. And in the 2nd paragraph you say it (the 'commonly shared instance') gets provided by a 'static accessor method'.

Now please answer to us what the difference to the already proposed API is?
This also has a
* ConfigProvider.getConfig() -> the 'static' accessor the the shared config which gets auto-discovered
* ConfigProvider.builder() -> a builder to produce a manually assembled Config

So all you proposed functionality already exists, right?

>> 1.c.) How do you know that randomlib.jar needs some configuration? Sounds very error-prone.
​> Honestly said I dont know, unless the jar uses annotations for injection and I would scan for them.

Well, those are exactly those problems we had in real live.

And it's perfectly solved with auto-pickup.


@Gunnar:

> connection:
> url: jdbc:mysql://localhost:3306/db
> poolSize: 5
>
> How does the proposed API/SPI deal with that? How would a user access the url value?
> Dot syntax ("connection.url") may be one idea, but is this described somewhere?

That's exactly how it works in the current proposal.
But to be more concrete it's not a problem of the spec but gets mapped inside any custom YamlConfigSource.
This is kind of like XPath or JsonPointer.

> protocols:
> - http
> - https
>
> users:
> tom: passwd
> bob: passwd
>
> How would these be handled?

That's more complicated. In my original proposal I explicitly ruled it out. It is not really a problem with the String, String mapping, but what about the overriding?
What if you e.g. have a DatabaseConfigSource (with higher ordinal) and you want to amend those values?
In this case you come across all the include/exclude problems!
Does the whole List/Map gets replaced? Or is there some 'patch' mechanism in place?
If you can answer this question, then I can tell you how it can be implemented with the current interface.
In my experience it really becomes complicated and thus we left this out for 1.0.

LieGrue,
strub

[Werner Keil]

1.0?

The 1.0 release of Microprfile was already defined, thus if the proposal/incubator for config was to enter a future release it might be 1.1 or beyond.

Having a single test case that contains only methods like
/*** Rigourous Test :-)*/
public void testApp(){assertTrue( true );}

Does not feel ready for a milestone or release right now.

Eclipse (EMO/ Technology PMC and the mentors of a project) does this before a project publishes a new release, take Nebula as example since there is no release for Microprofile yet:
http://projects.eclipse.org/projects/technology.nebula/reviews/1.1.0-release-review

Leaving the way of how to order it aside (there are different approaches and opinions here on how to archive it) the actual format being YAML, XML, JSON or something like Typesafe HOCON should not matter to an API or SPI element like PropertySource, ConfigSource or whatever it is called by different solutions. Most of them have YAML support beside other formats, but the abstract base types (class or interface) do not have to worry about that.

Werner

[Gunnar Morling]

I think it actually is a problem of the spec, because the user must know how to access such nested value when getting it from the Config option. What should they specify as the property name? The API should provide a unified behaviour and experience, so that's something that needs to be defined at the spec level, it shouldn't be specific for single config sources.

Also it needs to be defined whether values for one sub-node may be contributed by different sources or not. Say you have "connection.url" in one source but "connection.poolSize" in another. Is that legal? Or is everything under "connection.*" required to be defined by a single source to avoid unexpected mixing from different sources? I don't have a strong preference either way, but it needs to be answered in the spec.

 

This is kind of like XPath or JsonPointer.

>   protocols:
>      - http
>       - https
>
>    users:
>        tom: passwd
>        bob: passwd
>
> How would these be handled?
That's more complicated. In my original proposal I explicitly ruled it out. It is not really a problem with the String, String mapping, but what about the overriding?
What if you e.g. have a DatabaseConfigSource (with higher ordinal) and you want to amend those values?
In this case you come across all the include/exclude problems!
Does the whole List/Map gets replaced? Or is there some 'patch' mechanism in place?
If you can answer this question, then I can tell you how it can be implemented with the current interface.

Let's say a config value always comes entirely from the source with the highest priority, no merging/amending across different sources. How would you implement it for list/map style properties with the current SPI?

 

In my experience it really becomes complicated and thus we left this out for 1.0.

I'm afraid that leaving such essential thing out of the picture because "it's complicated" is the safe recipe for circling us into a corner where we are stuck with an 1.0 API that fits for the "simple case" but not for others.

To provide a meaningful and versatile abstraction, we should dig a bit deeper. I mean, how can we say users should provide their own YAML source when we don't do it ourselves because it's too difficult? IMO the SPI must be vetted to be capable of this right from the beginning, to ensure it's actually useful for a wider range of applications.

LieGrue,
strub

--
You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/3fdc3672-2c49-497e-94a2-9339de23f46c%40googlegroups.com.

[James Carman]

On Thu, Jan 5, 2017 at 9:38 AM Gunnar Morling <gun...@hibernate.org> wrote:

I think it actually is a problem of the spec, because the user must know how to access such nested value when getting it from the Config option. What should they specify as the property name? The API should provide a unified behaviour and experience, so that's something that needs to be defined at the spec level, it shouldn't be specific for single config sources.

Agreed.  I was thinking about this yesterday.  Because names of properties in things like properties files should reflect some "structure" of sorts that would correspond to something else in a YAML file or JSON.  I think "dot notation" is probably best suited for this (and is the most portable), at least for purely scalar properties.  Can properties file keys contain stuff like "[0]"?  I've never tried.

 

Also it needs to be defined whether values for one sub-node may be contributed by different sources or not. Say you have "connection.url" in one source but "connection.poolSize" in another. Is that legal? Or is everything under "connection.*" required to be defined by a single source to avoid unexpected mixing from different sources? I don't have a strong preference either way, but it needs to be answered in the spec.

I think you should be able to mix and match.  An obvious example would be for database connection properties (as you're pointing out here).  You might, for instance, put all the regular stuff in a properties file, but use a more secure way to define the credentials.  This should all get munged together in the end, however.  Spring supports this and uses the ordering of the sources to decide who wins.  I think this is probably sufficient.  

[Werner Keil]

Looking at the major solutions (not inspired by or related to DeltaSpike or OWB)
- Apache Commons Config 2
- Spring
- Archaius
- Typesafe Config

Neither of them has a concept of "order" or "ordinal" on their PropertySource or ConfigurationSource constructs.

Spring uses its very old (at least since 2003) Ordered interface when it comes to a "Configuration Listener"
http://stackoverflow.com/questions/40944064/spring-boot-change-the-order-of-the-propertysource

The actual ordering happens via http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/core/env/MutablePropertySources.html container.
This is the "List / Map" style container in Spring's case.
Archaius, at least in version 2 knows something called AggregatedConfiguration with a similar ordering as Spring PropertySources.

Can't say how widely used cfg4j is in real life, but the way it does this feels similar to the https://javadocs.com/docs/com.typesafe/config/0.3.1/com/typesafe/config/ConfigMergeable.java base interface of Typesafe Config.
There every config is also mergeable, and can have a a fallback.

Interestingly, Pivotal more and more uses the entire Netflix stack under the hood. Spring Cloud refers not only Hystrix, it also seems to use Archaius in many places instead of its own "legacy" mechanisms.

So Archaius can certainly be seen as a de-facto standard in this space.

Unless a true standard effort like a JSR came out of this, do you think Netflix would be convinced to implement this API in say Archaius 3 instead of its own?

Werner

LieGrue,
strub

To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

[Emily Jiang]

As for YAML support, as Mark mentioned earlier, the parser bit is isolated in YAMLConfigSource. In the config source, the parser should be able to flatter the hierarchy. Take your example:

protocols:
>      - http
>       - https
>
>    users:
>        tom: passwd
>        bob: passwd

The key might be like: protocols.http, protocols.https. It should be in the YAMLConfigSource javadoc.


I don't see the point of disallowing users.* defined in another config source. The config sources are ordered and if a lower rank config source provides the same users.tom, it will be ignored.

If for some reason, the ConfigSource parses the file differently say "protocols", when asking for "protocols", the value will be the structure. A converter will then needed to be passed in and read the object out with http and https. Therefore, it is the implementation details for a particular ConfigSource. I am fine to add the javadoc on the ConfigSource to mandate the implementation of YAML to the flat structure of the config. Thoughts? The same thing should apply to Json files.

[Gunnar Morling]

Am Donnerstag, 5. Januar 2017 18:38:41 UTC+1 schrieb Emily Jiang:

As for YAML support, as Mark mentioned earlier, the parser bit is isolated in YAMLConfigSource. In the config source, the parser should be able to flatter the hierarchy. Take your example:
protocols:
>      - http
>       - https
>
>    users:
>        tom: passwd
>        bob: passwd

The key might be like: protocols.http, protocols.https. It should be in the YAMLConfigSource javadoc.

"protocols" is *one* configuration value of type list, containing two elements, "http" and "https". So the key would be "protocols".

Of course the YAML parsing would be done within the config source, but my question is how the config source would return these values. I don't think it can reasonably be done with the current proposal for the ConfigSource SPI which suggests to return a Map<String, String>.

 

I don't see the point of disallowing users.* defined in another config source. The config sources are ordered and if a lower rank config source provides the same users.tom, it will be ignored.

If for some reason, the ConfigSource parses the file differently say "protocols", when asking for "protocols", the value will be the structure. A converter will then needed to be passed in and read the object out with http and https. Therefore, it is the implementation details for a particular ConfigSource. I am fine to add the javadoc on the ConfigSource to mandate the implementation of YAML to the flat structure of the config. Thoughts? The same thing should apply to Json files.

I'm not sure I'm following. Maybe you can sketch an implementation of a YAML config source which allows a user to obtain a List<String> with the configured protocols when asking for "protocols"? I think it's this sort of small PoCs that we need to write in order to make sure the API/SPI fits.

[Emily Jiang]

Sorry. Misread. In the current design, the return value of protocols will the string representation of the List of http and https. Maybe the return value of Map<String, Object> might be more flexible.
Emily

[Mark Struberg]

So the last open question seems to be how to support Lists and Maps?

I remember we did discuss this in DeltaSpike and ended up to intentionally NOT support it!
And we had the same discussions again in Tamaya and with the same result (initially, not sure what happened in the meantime).

The reason is simple:
* Maps are implicitly supported anyway. user.bob and user.tom are simple keys. Just take all config values which start with user. and be done.
* Lists can be represented as comma separated value. Done.

Ok, that puts a bit more work on the user, but be honest: how often is it used in real world?
I use DeltaSpike Config in about 20 big projects. And needed the list feature for about 5 configured values in those 20 projects.

An example is e.g.

myproject.batches.somebatch.finished.emailTo=a...@comp.any,b.comp.any

Now if the ops team want's to replace this token they simply add the following to the properties table in the app db:
myproject.batches.somebatch.finished.emailTo=o...@comp.any,busine...@comp.any

With anything else than comma separated values you not only need to define a certain format, but you would also introduce the following complexity:
* make the ConfigSource interface more complicated
* introduce metadata (what a pain!)
* define include/exclude rules (serious headache for users most times)
* introduce some kind of patch rules for value overrides (but this trashes the independence of ConfigSources as you need to know exactly WHAT you gonna patch)

Is that really worth it? I'd rather say no.

You might now probably also understand why my original proposal has a few additional features, e.g.
https://github.com/struberg/javaConfig/blob/master/api/src/main/java/io/microprofile/config/Config.java#L73
<T> T convert(String value, Class<T> asType);

That way you can still have a comma separated List but use the registered Converters to fully support custom representations.

We might probably introduce a method

/**
* Resolves the value configured for the given key as comma separated list
* and convert it to an Iterable of the required asType.
*
* Escaping:
* Any ',' in a value is represented as "\,".
* Any '\' in a value is represented as "\\".
*
* @throws UnsupportedOperationException if there is no {@link Converter} registered for asType
*/
<T> Iterable<T> getListValue(String key, Class<T> asType);

That way it is
* easy for developers as we still stick with String, String for all the internal handling
* easy for the users to administer the values as it's just a comma separated list

I'm worried that introducing anything beyond that is rendering the API and usage massively more complex.

[Gunnar Morling]

Hey,

2017-01-06 10:36 GMT+01:00 Mark Struberg <markst...@gmail.com>:

So the last open question seems to be how to support Lists and Maps?

I don't really know. Have we settled on the ordinal discussion?

Personally I find the lack of type-safety discomforting. Having an approach which allows to define the value types, validation rules (legal value ranges etc.) and default values in a central place would be an essential part of the story to me. I made a proposal on this a while ago, and while we may not agree on making this the exclusive approach for the API (I still kinda think it should be), it should be at least be added on top of the unsafe layer.

I remember we did discuss this in DeltaSpike and ended up to intentionally NOT support it!
And we had the same discussions again in Tamaya and with the same result (initially, not sure what happened in the meantime).

The reason is simple:
* Maps are implicitly supported anyway. user.bob and user.tom are simple keys. Just take all config values which start with user. and be done.

How would a user do that with the current API?

 

* Lists can be represented as comma separated value. Done.

So you are suggesting that a YAML config source parses a *.yaml file and then, for list values, it assembles a comma-separated string to pass it back up? Honestly, I'd find such SPI and approach sub-par.

Ok, that puts a bit more work on the user, but be honest: how often is it used in real world?
I use DeltaSpike Config in about 20 big projects. And needed the list feature for about 5 configured values in those 20 projects.

An example is e.g.

myproject.batches.somebatch.finished.emailTo=a...@comp.any,b.comp.any

Now if the ops team want's to replace this token they simply add the following to the properties table in the app db:
myproject.batches.somebatch.finished.emailTo=o...@comp.any,busine...@comp.any

With anything else than comma separated values you not only need to define a certain format, but you would also introduce the following complexity:
* make the ConfigSource interface more complicated

I'd rather say make it suitable for other cases than the most trivial one.

 

* introduce metadata (what a pain!)

Can you elaborate?

 

* define include/exclude rules (serious headache for users most times)

I'm not sure what that means.

 

* introduce some kind of patch rules for value overrides (but this trashes the independence of ConfigSources as you need to know exactly WHAT you gonna patch)

Can you give an example? 

Is that really worth it? I'd rather say no.

You might now probably also understand why my original proposal has a few additional features, e.g.
https://github.com/struberg/javaConfig/blob/master/api/src/main/java/io/microprofile/config/Config.java#L73
<T> T convert(String value, Class<T> asType);

That way you can still have a comma separated List but use the registered Converters to fully support custom representations.

We might probably introduce a method

    /**
     * Resolves the value configured for the given key as comma separated list
     * and convert it to an Iterable of the required asType.
     *
     * Escaping:
     * Any ',' in a value is represented as "\,".
     * Any '\' in a value is represented as "\\".
     *
     * @throws UnsupportedOperationException if there is no {@link Converter} registered for asType
     */
    <T> Iterable<T> getListValue(String key, Class<T> asType);

I'd rather add getValue(String key, TypeLiteral<T> type) as suggested before, as it allows you to get values as a set, list etc.

That way it is
* easy for developers as we still stick with String, String for all the internal handling

Personally I don't find that easy, rather I think it's a poor abstraction. I think that support for properties with a cardinality greater than one needs to find explicit support in the ConfigSource SPI.

 

* easy for the users to administer the values as it's just a comma separated list

I'm worried that introducing anything beyond that is rendering the API and usage massively more complex.

I'm not sure how this is affecting API complexity. As a user, you could simply do List<String> protocols = config.get( "protocols", new TypeLiteral<List<String>{}() ).

I agree it increases complexity for the ConfigSource SPI, but I think that's needed and justified to make it actually work.

--
You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to microprofile+unsubscribe@googlegroups.com.
To post to this group, send email to microp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/microprofile/12319c9f-448a-4d4a-b42e-17664c22ba6a%40googlegroups.com.

[Werner Keil]

Hi,

On Friday, January 6, 2017 at 12:29:33 PM UTC+1, Gunnar Morling wrote:

Hey,

2017-01-06 10:36 GMT+01:00 Mark Struberg <markst...@gmail.com>:

So the last open question seems to be how to support Lists and Maps?

I don't really know. Have we settled on the ordinal discussion?

Personally I find the lack of type-safety discomforting. Having an approach which allows to define the value types, validation rules (legal value ranges etc.) and default values in a central place would be an essential part of the story to me. I made a proposal on this a while ago, and while we may not agree on making this the exclusive approach for the API (I still kinda think it should be), it should be at least be added on top of the unsafe layer.

Was that proposal annotation-based? Tamaya does that on top of the "unsafe" base API/SPI layer in an extension module.
 

I remember we did discuss this in DeltaSpike and ended up to intentionally NOT support it!
And we had the same discussions again in Tamaya and with the same result (initially, not sure what happened in the meantime).

The reason is simple:
* Maps are implicitly supported anyway. user.bob and user.tom are simple keys. Just take all config values which start with user. and be done.

How would a user do that with the current API?

Please also see Anatole's blog post, but just as much here, Tamaya still has the ordinal, but it's marked as TODO and you may consider it a technical debt or something to sort out.
There are mechanisms for collection support on the SPI level, too, e.g. PropertyValueCombinationPolicy which very roughly could be compared to elements in Spring like PropertySources or similar ones in other frameworks like Archaius.
 

In a post Java EE 8 world if you cut a solution into smaller parts (like CDI 2 tries or actual "Micro" JSRs like 361 and 363 already demonstrated) one could have a very simple small API (not so different from what Emily sketched out so far) and an optional SPI. CDI has that separation, especially the accessor class with the same name is located in the SPI. With one exception (getContext() in Configuration, it used to be in ConfigProvider before) Tamaya also separates those in a clean way. And a minimal API could be separated from an SPI module with more complex functionality.
 

--
You received this message because you are subscribed to a topic in the Google Groups "MicroProfile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/microprofile/VJEEAOsVj5E/unsubscribe.

To unsubscribe from this group and all its topics, send an email to microprofile...@googlegroups.com.

To post to this group, send email to microp...@googlegroups.com.

[Gunnar Morling]

Am Freitag, 6. Januar 2017 14:05:59 UTC+1 schrieb Werner Keil:

Hi,

On Friday, January 6, 2017 at 12:29:33 PM UTC+1, Gunnar Morling wrote:

Hey,

2017-01-06 10:36 GMT+01:00 Mark Struberg <markst...@gmail.com>:

So the last open question seems to be how to support Lists and Maps?

I don't really know. Have we settled on the ordinal discussion?

Personally I find the lack of type-safety discomforting. Having an approach which allows to define the value types, validation rules (legal value ranges etc.) and default values in a central place would be an essential part of the story to me. I made a proposal on this a while ago, and while we may not agree on making this the exclusive approach for the API (I still kinda think it should be), it should be at least be added on top of the unsafe layer.

Was that proposal annotation-based? Tamaya does that on top of the "unsafe" base API/SPI layer in an extension module.

It's based on static config type declarations, using annotations for binding config type members to keys. See the orginal message for the details: https://groups.google.com/d/msg/microprofile/JRJXHqXpHZA/ZnDb5y5MAAAJ.