Setting database string from environmental variable?

[Drew Carey Buglione]

Hello, all!

I'm attempting to create what I believe should be a very simple Trac plugin. All I want to do is set the database string from an environmental variable.

I've been playing around with this for a bit now, and I can't figure out the way I ought to do this.

I'm under the impression that I want to do something like the following:

self.config.set('trac', 'database', os.environ.get('DATABASE_URL'))

But I'm not sure where. Or how.

Thanks for any help,
~Drew Carey Buglione

[Olemis Lang]

On 10/15/13, Drew Carey Buglione <drewcb...@gmail.com> wrote:
> Hello, all!
>

:)

> I'm attempting to create what I believe should be a very simple Trac
> plugin. All I want to do is set the database string from an environmental
> variable.
>
> I've been playing around with this for a bit now, and I can't figure out
> the way I ought to do this.
>
> I'm under the impression that I want to do something like the following:
>
> self.config.set('trac', 'database', os.environ.get('DATABASE_URL'))
>
> But I'm not sure where. Or how.
>

In theory , yes . Nevertheless DB string is supposed to be configured
in trac.ini . I'm not sure if your approach will work , but I wonder
why is it that config is not enough for you ? Maybe what you need is
not exactly a plugin but an external script ?

--
Regards,

Olemis - @olemislc

[Saint Germain]

On Tue, 15 Oct 2013 17:07:13 -0500, Olemis Lang <ole...@gmail.com>
wrote :

> > I'm attempting to create what I believe should be a very simple Trac
> > plugin. All I want to do is set the database string from an
> > environmental variable.
> >
> > I've been playing around with this for a bit now, and I can't
> > figure out the way I ought to do this.
> >
> > I'm under the impression that I want to do something like the
> > following:
> >
> > self.config.set('trac', 'database', os.environ.get('DATABASE_URL'))
> >
> > But I'm not sure where. Or how.
> >
>
> In theory , yes . Nevertheless DB string is supposed to be configured
> in trac.ini . I'm not sure if your approach will work , but I wonder
> why is it that config is not enough for you ? Maybe what you need is
> not exactly a plugin but an external script ?
>

Hello,

I have the exact same need as all my files configuration are versioned
and I would like to keep the database login/password outside the
configuration files (and the repository).

It may also help if I share the repository with others: they can use
whatever database they see fit.

Another solution would be to keep the database string in a dedicated
file (not in the repository) and import it in trac.ini.

Regards,

[Olemis Lang]

The recommended approach in this case is to use [inherit] file option
in trac.ini and configure DB in an external file (empty by default?) .
We use this in Bloodhound default installation to inherit in trac.ini
the configs specified in base.ini . Inheritance levels are unlimited
(afaict , but 2-3 levels should be enough ...)

--
Regards,

Olemis - @olemislc

Apache™ Bloodhound contributor
http://issues.apache.org/bloodhound
http://blood-hound.net

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

[Saint Germain]

On Tue, 15 Oct 2013 18:36:55 -0500, Olemis Lang <ole...@gmail.com>

Exactly what I needed, thanks !

[Drew Carey Buglione]

        I'm attempting to create what I believe should be a very simple Trac plugin. All I want to do is set the database string from an environmental variable.

        I've been playing around with this for a bit now, and I can't figure out the way I ought to do this.

        I'm under the impression that I want to do something like the following:

        self.config.set('trac', 'database', os.environ.get('DATABASE_URL'))

        But I'm not sure where. Or how.

        Thanks for any help,
        ~Drew Carey Buglione

In theory , yes . Nevertheless DB string is supposed to be configured in trac.ini . I'm not sure if  your approach will work , but I wonder why is it that config is not enough for you ? Maybe what you need is not exactly a plugin but an external script ?

Thank you for responding, Olemis.

My reason for desiring to do things this way is simple: I'm running Trac on Heroku, where there is a convention of keeping all private or environment-specific data in config variables.

I can not deploy any files to Heroku without checking them into version control, and I'd like to avoid checking my database string into version control.

Heroku provides the database string as an environmental variable, though, so if anyone has any idea how I can go about resolving my problem the way I'd like to, I'd really appreciate some pointers.

~Drew Carey Buglione

[Noah Kantrowitz]

In your procfile, just make the process started be a bash script that writes out trac.ini based on $database_url and then execs gunicorn.

--Noah

Drew Carey Buglione <drewcb...@gmail.com> wrote:

Heroku provides the database string as an environmental variable, though, so if anyone has any idea how I can go about resolving my problem th e way I'd like to, I'd really appreciate some pointers.

~Drew Carey Buglione

[Drew Carey Buglione]

Noah,

I appreciate the reply; that is a solution I had considered.

However, it feels like such a kludgy solution. Is there really no Trac-sanctioned way to write a plugin that will do what I'm asking?

~Drew

[Olemis Lang]

On 10/15/13, Drew Carey Buglione <drewcb...@gmail.com> wrote:
> Noah,
>
> I appreciate the reply; that is a solution I had considered.
>
> However, it feels like such a kludgy solution. Is there really no
> Trac-sanctioned way to write a plugin that will do what I'm asking?
>

It seems to me that this is a chicken egg situation . You need a
plugin to configure DB string based on ... in order to instantiate a
plugin you need an instance of Environment , but that object
automatically allocates a db object , and in order to do so it will
demand DB connection string ... and the circle starts all over again .

cmiiw

--
Regards,

Olemis - @olemislc

[Ethan Jucovy]

On Tue, Oct 15, 2013 at 11:44 PM, Drew Carey Buglione <drewcb...@gmail.com> wrote:

My reason for desiring to do things this way is simple: I'm running Trac on Heroku, where there is a convention of keeping all private or environment-specific data in config variables.

I can not deploy any files to Heroku without checking them into version control, and I'd like to avoid checking my database string into version control.

Heroku provides the database string as an environmental variable, though, so if anyone has any idea how I can go about resolving my problem the way I'd like to, I'd really appreciate some pointers.

There's definitely no well supported way to do this, and a plugin wouldn't really be the right approach for the reason Olemis mentioned (chicken/egg).  

Building your own WSGI script that sets up the environment and then overrides the relevant config variable before dispatching each request is probably the best way to go.  With a bit of tinkering and copying code around from trac.web.main I was able to get something that at least appears to work:

https://gist.github.com/ejucovy/7007820 

You can point gunicorn (or any other wsgi server) to it.  It serves a single Trac environment -- it assumes TRAC_ENV is present in an environment variable; it would take some more work to make it multi-environment-friendly.  If you set a TRAC_DATABASE environment variable, it will (non-persistently; just for the duration of each request) override the database connection string from the environment's config file.

It seems to work, but it's possible that it would run into problems (unclosed database connections?) that aren't apparent at first glance.

BTW, I've been thinking of moving some of my Trac sites over to Heroku, but I was worried that Trac's assumptions about filesystem access for environments would break under Heroku's model.  I guess this is a topic for another thread, but I'd love to hear more about how you're setting up each Trac environment in Heroku and managing/versioncontrolling/deploying the environment directories themselves.

Best,

Ethan

[Olemis Lang]

On 10/16/13, Ethan Jucovy <ethan....@gmail.com> wrote:
> On Tue, Oct 15, 2013 at 11:44 PM, Drew Carey Buglione <
> drewcb...@gmail.com> wrote:
>
>> My reason for desiring to do things this way is simple: I'm running Trac
>> on Heroku, where there is a convention of keeping all private or
>> environment-specific data in config variables.
>>
>> I can not deploy any files to Heroku without checking them into version
>> control, and I'd like to avoid checking my database string into version
>> control.
>>
>> Heroku provides the database string as an environmental variable, though,
>> so if anyone has any idea how I can go about resolving my problem the way
>> I'd like to, I'd really appreciate some pointers.
>>
>
> There's definitely no well supported way to do this, and a plugin wouldn't
> really be the right approach for the reason Olemis mentioned (chicken/egg).
>

Considering that there's a real use case for this , question is : is
it a good idea to request for this enhancement by creating a ticket in
t.e.o issue tracker ?

[...]

--
Regards,

Olemis - @olemislc

[Ethan Jucovy]

> There's definitely no well supported way to do this, and a plugin wouldn't
> really be the right approach for the reason Olemis mentioned (chicken/egg).
>

Considering that there's a real use case for this , question is : is
it a good idea to request for this enhancement by creating a ticket in
t.e.o issue tracker ?

Before I file a ticket, I'm actually wondering whether this database override is sufficient for Trac on Heroku.  Since Trac (and plugins) are free to make persistent changes to the environment's trac.ini file and calling config.save(), what will happen to that file on a Heroku environment, since it will then have changes that are not committed to the repository that's managing it?

I wonder if the entire Configuration implementation needs to be swappable -- so that e.g. a Heroku-friendly implementation could get and set *all* settings directly from os.environ (which is persistent on Heroku).  From a quick grep, it looks like it wouldn't be very hard to make this swappable, since almost all usage of Configuration is through `env.config` instead of instantiating a `Configuration` directly.

But maybe this isn't necessary, or maybe Drew's already figured out another approach that works for managing the Trac environment's config file.

-Ethan

[Olemis Lang]

On 10/16/13, Ethan Jucovy <ethan....@gmail.com> wrote:

>> > There's definitely no well supported way to do this, and a plugin
>> wouldn't
>> > really be the right approach for the reason Olemis mentioned
>> (chicken/egg).
>> >
>>
>> Considering that there's a real use case for this , question is : is
>> it a good idea to request for this enhancement by creating a ticket in
>> t.e.o issue tracker ?
>
>
> Before I file a ticket, I'm actually wondering whether this database
> override is sufficient for Trac on Heroku. Since Trac (and plugins) are
> free to make persistent changes to the environment's trac.ini file and
> calling config.save(), what will happen to that file on a Heroku
> environment, since it will then have changes that are not committed to the
> repository that's managing it?
>
> I wonder if the entire Configuration implementation needs to be swappable
> -- so that e.g. a Heroku-friendly implementation could get and set *all*
> settings directly from os.environ (which is persistent on Heroku).
>
> From a
> quick grep, it looks like it wouldn't be very hard to make this swappable,
> since almost all usage of Configuration is through `env.config` instead of
> instantiating a `Configuration` directly.
>

Well , it's possible to replace Configuration objects with whatever
compatible class you could develop . Indeed product configuration in
Apache™ Bloodhound is stored in the database (see multiproduct.config
module ...) . In that case I initially proposed a mechanism to make
config implementations swappable , but the complexity was not really
considered useful for our use cases , especially considering that we
had other priorities ...

[Drew Carey Buglione]

So, I gave up on doing things within Trac, and combined the two approaches suggested earlier. My setup is documented here: https://github.com/drewbug/heroku-trac#protect-database-string

This even works when developing locally with foreman, which I'm happy about.

It's probably a good idea for me to switch to gunicorn, rather than tracd, but I was aiming for the simplest functional product, not necessarily the fastest.

Next up will be my adventures with AccountManagerPlugin, which I'm hoping is easy.

As for changes to the configuration file not persisting: I'm not too worried at the moment, but I'm not exactly able to come up with an idea on what the worst-case scenario would be. Ideas?

If I run in to any more trouble, I'm sure I'll be back.

Thanks for all the help, everyone.

[Ethan Jucovy]

On Wed, Oct 16, 2013 at 3:34 PM, Drew Carey Buglione <drewcb...@gmail.com> wrote:

So, I gave up on doing things within Trac, and combined the two approaches suggested earlier. My setup is documented here: https://github.com/drewbug/heroku-trac#protect-database-string

This is really cool, thanks for sharing it. 

 

As for changes to the configuration file not persisting: I'm not too worried at the moment, but I'm not exactly able to come up with an idea on what the worst-case scenario would be. Ideas?

Well .. changing which plugins/components are active through the web interface is probably the biggest // most likely problem.  If you do change that through the web UI and then Heroku decides to wipe out your changes, I guess you could conceivably end up with a broken site, or at least one that does not behave at all how you expect it to.

A couple of other (probably minor) things that can be changed TTW and therefore might reset unexpectedly would be the project name, url, default timezone and language, and logging configuration.

The other thing that seems both likely and major (at least from my own usage) would be custom ticket fields or ticket workflow changes, but those are both only editable through the web via plugins (CustomFieldAdminPlugin and TracWorkflowAdminPlugin) so if you don't have those installed you're probably pretty OK.  

Similarly if you have IniAdminPlugin installed all bets are off.

And of course it's hard to pin down exactly what through-the-web edits will result in config changes once you get into plugin-land.

[Drew Carey Buglione]

So, I gave up on doing things within Trac, and combined the two approaches suggested earlier. My setup is documented here: https://github.com/drewbug/heroku-trac#protect-database-string

This is really cool, thanks for sharing it. 

Definitely, thanks for taking a look!
 

As for changes to the configuration file not persisting: I'm not too worried at the moment, but I'm not exactly able to come up with an idea on what the worst-case scenario would be. Ideas?

Well .. changing which plugins/components are active through the web interface is probably the biggest // most likely problem.  If you do change that through the web UI and then Heroku decides to wipe out your changes, I guess you could conceivably end up with a broken site, or at least one that does not behave at all how you expect it to.

 

A couple of other (probably minor) things that can be changed TTW and therefore might reset unexpectedly would be the project name, url, default timezone and language, and logging configuration.

The other thing that seems both likely and major (at least from my own usage) would be custom ticket fields or ticket workflow changes, but those are both only editable through the web via plugins (CustomFieldAdminPlugin and TracWorkflowAdminPlugin) so if you don't have those installed you're probably pretty OK.  

Similarly if you have IniAdminPlugin installed all bets are off.

And of course it's hard to pin down exactly what through-the-web edits will result in config changes once you get into plugin-land.

Hmm. It sounds like most of these are settings that won't be changing all that often, so I'm fine with just manually editing the trac.ini file when the needs arise.

How feasible do you think it would be to make a plugin that intercepts config changes and, instead of saving them, shows a page with an embedded diff-style list of changes that need to be made to trac.ini? Maybe call it EphemeralConfigPlugin ;)

~Drew Carey Buglione

[Olemis Lang]

On 10/16/13, Drew Carey Buglione <drewcb...@gmail.com> wrote:
>
[...]

>
>> A couple of other (probably minor) things that can be changed TTW and
>> therefore might reset unexpectedly would be the project name, url, default
>>
>> timezone and language, and logging configuration.
>>
>> The other thing that seems both likely and major (at least from my own
>> usage) would be custom ticket fields or ticket workflow changes, but those
>>
>> are both only editable through the web via plugins (CustomFieldAdminPlugin
>>
>> and TracWorkflowAdminPlugin) so if you don't have those installed you're
>> probably pretty OK.
>>
>> Similarly if you have IniAdminPlugin installed all bets are off.
>>
>> And of course it's hard to pin down exactly what through-the-web edits
>> will result in config changes once you get into plugin-land.
>>
>
> Hmm. It sounds like most of these are settings that won't be changing all

> that often, so I'm fine with just manually editing the *trac.ini* file when

>
> the needs arise.
>
> How feasible do you think it would be to make a plugin that intercepts
> config changes and, instead of saving them, shows a page with an embedded

> diff <https://en.wikipedia.org/wiki/Diff#Context_format>-style list of
> changes that need to be made to *trac.ini*? Maybe call it *
> EphemeralConfigPlugin* ;)
>

I'm guessing that it will be lot easier to write a Configuration
object storing config data in the DB and inject it into instances of
trac.env.Environment in place of standard . If you need a starting
point to get this done , see Bloodhound's [1]_

.. [1] http://issues.apache.org/bloodhound/browser/trunk/bloodhound_multiproduct/multiproduct/config.py

--
Regards,

Olemis - @olemislc

[Ethan Jucovy]

On Wed, Oct 16, 2013 at 11:22 AM, Ethan Jucovy <ethan....@gmail.com> wrote:

I wonder if the entire Configuration implementation needs to be swappable

I submitted a ticket for this: http://trac.edgewall.org/ticket/11339

I'll try to put together a proof-of-concept patch unless Olemis beats me to it.  ;-) 

-Ethan