How to handle lots of config flags?

[David Kowis]

I'm finding the need to configure a reasonably large number of flags to my Finatra app.

I was starting to go down the path of using Typesafe's Config library to specify a single flag -config.file and then have that handle the configuration, but I thought I'd ask if there's a "Finatra way" to do it.

Thanks!

[Steve Cosenza]

Hi David,

We currently only support flags for configuration. However, Typesafe Config seems like a great choice once the number of flags become unwieldy. If you end up with some reusable utils for working with TypeSafe Config, please submit a PR for a finatra-typesafe-config library :-)

Thanks!

Steve

--
You received this message because you are subscribed to the Google Groups "finatra-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to finatra-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Kowis]

On Tuesday, September 1, 2015 at 3:26:54 PM UTC-5, Steve Cosenza wrote:

Hi David,

We currently only support flags for configuration. However, Typesafe Config seems like a great choice once the number of flags become unwieldy. If you end up with some reusable utils for working with TypeSafe Config, please submit a PR for a finatra-typesafe-config library :-)

Makes sense. I'm thinking about trying to do things in the same way the @Flag annotation works, but perhaps with a bit more safety: @Config(classOf[String], "config.param") thing:String or @Config(classOf[Integer], "config.param2") thing.Integer.

I found the com.twitter.inject: inject-app library which contains the definition of the Flag, and it's implementation and then a FlagsModule to do the binding of strings to the value provided by that flagsMap. What I'm having trouble figuring out is where the dots are connected from that map of flags to the `flag` methods I call in the actual FinatraServer itself.

I think if I could figure that out, I could get a TypesafeConfig object in there instead, and do a bit of inspection on the passed in values to get a typesafe extraction, or failure, of the configuration parameter. That would be the best, and it seems like it'd be pretty simple, but I'm not clear on how to wire the two together.

Thanks for any advice!

 

[Fayimora Femi-Balogun]

I recently(an hour ago) started looking for a solution to this too. Managing a bunch of flags and modules is quite a pain.

Off the top of my head, I'm thinking: 

1. Create flag -app.config which has default "application". Every app needs configuration so this is probably pointless.

2. Create flag -app.env with default "development" (unless finatra already has this)

3. Configs will reside in config/ with names like application.conf, development.conf, test.conf, production.conf ... e.t.c

4. Create ConfigModule that loads the config and provides an AppConfig instance

5. AppConfig exposes 2 methods. get(path: string) and getOrEmpty(path: String). the former throws an exception if path doesn't exist

6. dev/test/stage config is loaded, based on -app.env. application.conf would be 'included' in the environment specific configs. Perhaps, development.conf can be default

7. Add ConfigModule to Seq of modules in server and @Inject AppConfig wherever you need it.

8. Profit!

@Steve does that sound reasonable? Perhaps it can be simplified somehow(i'm new to finatra)


Cheers,

Fayi

[David Kowis]

On 2015-09-02 13:42, Fayimora Femi-Balogun wrote:
> I recently(an hour ago) started looking for a solution to this too.
> Managing a bunch of flags and modules is quite a pain.
>
> Off the top of my head, I'm thinking:
>
> 1. Create flag -app.config which has default "application". Every app
> needs configuration so this is probably pointless.
>
> 2. Create flag -app.env with default "development" (unless finatra
> already has this)
>
> 3. Configs will reside in config/ with names like application.conf,
> development.conf, test.conf, production.conf ... e.t.c
>
> 4. Create ConfigModule that loads the config and provides an AppConfig
> instance
>
> 5. AppConfig exposes 2 methods. get(path: string) and getOrEmpty(path:
> String). the former throws an exception if path doesn't exist
>
> 6. dev/test/stage config is loaded, based on -app.env.
> application.conf would be 'included' in the environment specific
> configs. Perhaps, development.conf can be default
>
> 7. Add ConfigModule to Seq of modules in server and @Inject AppConfig
> wherever you need it.
>
> 8. Profit!
>
> @Steve does that sound reasonable? Perhaps it can be simplified
> somehow(i'm new to finatra)

We can probably simplify some things to work a bit better within the
typesafe config library structure. It's got an ability to inherit within
the same configuration file, so you can very easily have a single
override for a specific environment.

I think I'd like to avoid baking in the concept of environments directly
into the configuration because the application is kept simpler if we
keep out environments and let an external configuration tool wire up the
right config file for the right environment the application is deployed
into.

I was just going to have a flag that takes a path to a configuration
file, and then using that flag, configure the ConfigModule (probably
TypesafeConfigModule) and then build the typeSafe annotation to expose
the configuration from the config file and fail in the standard
TypesafeConfig ways (exceptions, I think if there's no config path
found) Defaults are handled by having an "application.conf" in your
classpath, which it will fall back to, if it's unable to find the
parameter in the passed in configuration file.

But first step is understanding how the Server actually gets those
flags, so I can do the same thing and keep any services simple, where
they don't need to get the config object itself and then get a value
from the config object. I've got all the data I need, I just need to
wire it up in the proper order, so I can use @Config(type, key) and
extract it directly into the variable. Keeps services easy to test and
easy to reason about :)

--
David

>>> For more options, visit https://groups.google.com/d/optout [1].
>
>
> Links:
> ------
> [1] https://groups.google.com/d/optout

[David Kowis]

On 2015-09-02 13:42, Fayimora Femi-Balogun wrote:

> I recently(an hour ago) started looking for a solution to this too.
> Managing a bunch of flags and modules is quite a pain.

ALso looks like most of the work has been done for me:
https://github.com/racc/typesafeconfig-guice

All I really need to do is integrate this into the load process of the
twitter server, so the value of the -config.file flag is used to load.
Piece of cake!

[Steve Cosenza]

Nice!!!

+1 to keeping environments out of code and represented in external files

Here's where the flags are retrieved, and then eventually passed into FlagsModule: https://github.com/twitter/finatra/blob/master/inject/inject-app/src/main/scala/com/twitter/inject/app/App.scala#L124

-Steve

For more options, visit https://groups.google.com/d/optout.

[Fayimora Femi-Balogun]

I agree. David, are you doing this in a sample app? It would be nice if you push it so I/we can see how it all fits :)

Cheers

---------------------

Fayimora Femi-Balogun

MSc Machine Learning Student, UCL

[David Kowis]

I'm super close to having something working, using a third-party TypesafeConfig Guice module that I found:

val configurationFile = flag("config.file", "", "Optional config file to override settings")

//Stuff to hook up the typesafe config library into guice
lazy val config: Config = {
  val specified = configurationFile()
  if (specified.nonEmpty) {
    ConfigFactory.load(specified)
  } else {
    ConfigFactory.load()
  }
}

addFrameworkModule(TypesafeConfigModule.fromConfig(config))

I must be doing something wrong, because I can't get the value of the passed in flag from my tests. It's registered as a flag, since the twitterserver doesn't bomb for me, but I'm unable to resolve the flag that's specified this early in the process. What am I missing to make the value of the flag show up for use by that TypesafeConfigModule?

[David Kowis]

Yep, only thing I'm missing is how to read a value from a flag after parsing, but before all the guice stuff has run. How do I do that?

[Steve Cosenza]

I'm taking a look now.

-Steve

[Steve Cosenza]

Can you try implementing this as a non-framework module, since ideally the integration with typesafe-config would be optionally provided throug a finatra library (e.g. finatra-typesafe-config).

e.g. 

object FinatraTypesafeConfigModule extends TwitterModule {

  val configurationFile = flag("config.file", "", "Optional config file to override settings")

  

  lazy val config: Config = {
    val specified = configurationFile()
    if (specified.nonEmpty) {
      ConfigFactory.load(specified)
    } else {
      ConfigFactory.load()
    }
  }

  override val modules = Seq(TypesafeConfigModule.fromConfig(config))
}

You'd then add a FinatraTypesafeConfigModule to your server.

-Steve

 

[David Kowis]

On 2015-09-02 16:30, Steve Cosenza wrote:
> Can you try implementing this as a non-framework module, since ideally
> the integration with typesafe-config would be optionally provided
> throug a finatra library (e.g. finatra-typesafe-config).
> e.g.
>
> object FinatraTypesafeConfigModule extends TwitterModule { val
> configurationFile = flag("config.file", "", "Optional config file to
> override settings")
>
> lazy val config: Config = {
> val specified = configurationFile()
> if (specified.nonEmpty) {
> ConfigFactory.load(specified)
> } else {
> ConfigFactory.load()
> }
> }
>
> override val modules = Seq(TypesafeConfigModule.fromConfig(config))
> }
>
> You'd then add a FinatraTypesafeConfigModule to your server.

Ah that makes sense. I'm probably missing some terminology, and not
using the right components.

Do I add it as a module in the same way to my Server? (the override val
modules bit)

If that's the case, it doesn't seem to work, as I get lots of Guice
errors that tell me the module isn't ready to go. It didn't pick up the
module in time.

[info] 1) No implementation for java.lang.String annotated with
@com.github.racc.tscg.TypesafeConfig(value=auth.password) was bound.
[info] while locating java.lang.String annotated with
@com.github.racc.tscg.TypesafeConfig(value=auth.password)
[info] for parameter 2 at
com.homedepot.olt.jobber.BasicAuthFilter.<init>(BasicAuthFilter.scala:15)
[info] while locating com.homedepot.olt.jobber.BasicAuthFilter

There's probably something else I'm missing around here... It didn't
complain about the flag, so I probably loaded the module right, but it
didn't get into the right place to be able to actually be available for
guice...

Thanks for all the help!

>> [1]

>>
>> -Steve
>>
>> On Wed, Sep 2, 2015 at 12:03 PM, David Kowis <dko...@shlrm.org>
>> wrote:
>> On 2015-09-02 13:42, Fayimora Femi-Balogun wrote:
>> I recently(an hour ago) started looking for a solution to this too.
>> Managing a bunch of flags and modules is quite a pain.
>>
>> ALso looks like most of the work has been done for me:

>> https://github.com/racc/typesafeconfig-guice [2]

>> For more options, visit https://groups.google.com/d/optout [3] [1].
>
> Links:
> ------
> [1] https://groups.google.com/d/optout [3]

>
> --
> You received this message because you are subscribed to the Google
> Groups "finatra-users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to finatra-user...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout [3].
>
>
>
> Links:
> ------
> [1]
> https://github.com/twitter/finatra/blob/master/inject/inject-app/src/main/scala/com/twitter/inject/app/App.scala#L124
> [2] https://github.com/racc/typesafeconfig-guice
> [3] https://groups.google.com/d/optout

[Steve Cosenza]

Is your branch available? If so, what test can I run to reproduce the problem?

Thanks,

Steve

--
-Steve

Sent from Gmail Mobile

[David Kowis]

On 2015-09-02 17:36, Steve Cosenza wrote:
> Is your branch available? If so, what test can I run to reproduce the
> problem?

It's for work stuff, so I'll have to get it up onto regular githubs.
It's pretty easy to do though. I'll have something up by this evening I
hope (Central time).

Thanks for the help! Having code to look at should help quite a bit.

>> https://github.com/racc/typesafeconfig-guice [2] [2]

>> For more options, visit https://groups.google.com/d/optout [3] [3]
>> [1].
>>
>> Links:
>> ------
>> [1] https://groups.google.com/d/optout [3] [3]

>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "finatra-users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to finatra-user...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout [3]
>> [3].
>>
>> Links:
>> ------
>> [1]
>>
> https://github.com/twitter/finatra/blob/master/inject/inject-app/src/main/scala/com/twitter/inject/app/App.scala#L124

>> [1]
>> [2] https://github.com/racc/typesafeconfig-guice [2]
>> [3] https://groups.google.com/d/optout [3]

[David Kowis]

On 2015-09-02 17:36, Steve Cosenza wrote:

> Is your branch available? If so, what test can I run to reproduce the
> problem?

Okay, didn't get there as quickly as I'd liked, but here's a sample
application that does mostly what I want to do.

https://github.com/dkowis/finatra-typesafe-config

It's not picking up the flag settings, so my test configuration file
that I write out is always getting the default configuration, not the
one from the specified file. Currently, sbt test fails.

Thanks!

>> https://github.com/racc/typesafeconfig-guice [2] [2]

>> For more options, visit https://groups.google.com/d/optout [3] [3]
>> [1].
>>
>> Links:
>> ------
>> [1] https://groups.google.com/d/optout [3] [3]

>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "finatra-users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to finatra-user...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout [3]
>> [3].
>>
>> Links:
>> ------
>> [1]
>>
> https://github.com/twitter/finatra/blob/master/inject/inject-app/src/main/scala/com/twitter/inject/app/App.scala#L124

>> [1]
>> [2] https://github.com/racc/typesafeconfig-guice [2]
>> [3] https://groups.google.com/d/optout [3]
>

[David Kowis]

On 2015-09-03 10:04, David Kowis wrote:
> On 2015-09-02 17:36, Steve Cosenza wrote:
>> Is your branch available? If so, what test can I run to reproduce the
>> problem?
>
> Okay, didn't get there as quickly as I'd liked, but here's a sample
> application that does mostly what I want to do.
>
> https://github.com/dkowis/finatra-typesafe-config
>
> It's not picking up the flag settings, so my test configuration file
> that I write out is always getting the default configuration, not the
> one from the specified file. Currently, sbt test fails.
>
> Thanks!

Just to update, I was hacking on this a bit more, and I may have found a
bug with the upstream typesafeGuice module that I found. I'm going to
verify that, because I think something is buggered :(

[David Kowis]

On 2015-09-03 10:48, David Kowis wrote:
> On 2015-09-03 10:04, David Kowis wrote:
>> On 2015-09-02 17:36, Steve Cosenza wrote:
>>> Is your branch available? If so, what test can I run to reproduce the
>>> problem?
>>
>> Okay, didn't get there as quickly as I'd liked, but here's a sample
>> application that does mostly what I want to do.
>>
>> https://github.com/dkowis/finatra-typesafe-config
>>
>> It's not picking up the flag settings, so my test configuration file
>> that I write out is always getting the default configuration, not the
>> one from the specified file. Currently, sbt test fails.
>>
>> Thanks!
>
> Just to update, I was hacking on this a bit more, and I may have found
> a bug with the upstream typesafeGuice module that I found. I'm going
> to verify that, because I think something is buggered :(

Nope, upstream works just fine. It's got to be something else. I
verified it within their tests. I can have multiple strings injected in
a constructor and expect the right values to come out. Something odd is
going on.

For some reason, when I specify two @TypesafeConfig annotations, only
one of them is being populated. auth.user is always an empty string,
auth.pass is whatever I specify it to be. If I don't include auth.pass
in the application.conf file, it blows up properly (no config parameter
found), If I neglect auth.user, nothing happens. Perhaps I'm missing
something obvious.

--
David

[David Kowis]

As usual, it's because I'm stupid. Back on track, sorry for all the spam
:(

>
> --
> David

[David Kowis]

On 2015-09-03 11:02, David Kowis wrote:
> On 2015-09-03 11:00, David Kowis wrote:
>> On 2015-09-03 10:48, David Kowis wrote:
>>> On 2015-09-03 10:04, David Kowis wrote:
>>>> On 2015-09-02 17:36, Steve Cosenza wrote:
>>>>> Is your branch available? If so, what test can I run to reproduce
>>>>> the
>>>>> problem?
>>>>
>>>> Okay, didn't get there as quickly as I'd liked, but here's a sample
>>>> application that does mostly what I want to do.
>>>>
>>>> https://github.com/dkowis/finatra-typesafe-config
>>>>
>>>> It's not picking up the flag settings, so my test configuration file
>>>> that I write out is always getting the default configuration, not
>>>> the
>>>> one from the specified file. Currently, sbt test fails.

Okay, last email update until you guys get to look at it, I promise[0].

Using the custom guice module and a second server ends very, very badly.
The second server starts up and the entire guice stuff has already been
loaded, so any custom modules that you want do not get loaded.

This is somewhat problematic, since the fake backend server doesn't need
any of those guice things, only the primary server.

I had to add a second http server to my tests to evidence the failure.

sbt test will barf out the guice configuration errors now as it is in
the application I'm working on.

Thanks for the help again,
David Kowis

[Steve Cosenza]

Hi David,

As you've no doubt seen, it can be tricky to get flags working with externally written Guice modules. However, I've just upgraded TwitterModule to make integrations like these easier. The fix will be available in our next release which I'm hoping to publish this weekend. You'll then be able to write the following:

object FinatraTypesafeConfigModule extends TwitterModule {
  val configFile = flag("config.file", "", "Optional config file to override settings")

  override def configure() = {
    val specified = configFile()
    val config = if (specified.nonEmpty) {
      ConfigFactory.load(specified)
    } else {
      ConfigFactory.load()
    }

    install(TypesafeConfigModule.fromConfig(config))
  }
}

Sorry for the inconvenience, and thanks again for working on this feature!

Thanks,

Steve

--
You received this message because you are subscribed to the Google Groups "finatra-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to finatra-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[David Kowis]

> On Sep 5, 2015, at 14:29, Steve Cosenza <scos...@twitter.com> wrote:
>
> Hi David,
>
> As you've no doubt seen, it can be tricky to get flags working with externally written Guice modules. However, I've just upgraded TwitterModule to make integrations like these easier. The fix will be available in our next release which I'm hoping to publish this weekend. You'll then be able to write the following:
>
> object FinatraTypesafeConfigModule extends TwitterModule {
> val configFile = flag("config.file", "", "Optional config file to override settings")
>
> override def configure() = {
> val specified = configFile()
> val config = if (specified.nonEmpty) {
> ConfigFactory.load(specified)
> } else {
> ConfigFactory.load()
> }
>
> install(TypesafeConfigModule.fromConfig(config))
> }
> }
>
> Sorry for the inconvenience, and thanks again for working on this feature!

Pretty sweet! This will be super handy, thanks!

[David Kowis]

Any news on publishing that release?

Thanks,
David Kowis

[Christopher Coco]

Hey David,

The changes were merged in this commit: https://github.com/twitter/finatra/commit/c14ff121cccfe13fb1c761f076c20593dd105151 available in master and should be in the 2.0.0. release available on maven central.

Thanks!

-c

[David Kowis]

On Tuesday, September 15, 2015 at 10:39:49 AM UTC-5, Christopher Coco wrote:

Hey David,

The changes were merged in this commit: https://github.com/twitter/finatra/commit/c14ff121cccfe13fb1c761f076c20593dd105151 available in master and should be in the 2.0.0. release available on maven central.

OOHHH. I didn't notice that there was a 2.0.0 non-milestone release. I was looking for a .M3!

Awesome, thanks!
 

[Fayimora Femi-Balogun]

Hey folks,

Having "install(TypesafeConfigModule.fromConfig(config))" certainly got rid of the guice errors(No implementation for...). However, if you package a fatjar and try to run it, the errors resurface. Any ideas why and how to fix this?