problem when importing astropy from a webserver

[DVD PS]

Hello all!
-------------

Below there's a problem I've been having all this day, at the end it's how I managed to solve it momentarily.  In any case I think it's still worth to send it to the developers for hints, or for future reference to other users.

--------------
I've run into a problem when trying to deploy some software to be executed as a webservice.  This software uses couple of functionalities from astropy, the votables and the units modules.

I've found that the problem has been found before and reported in github: https://github.com/astropy/astropy/pull/536

In there it's mentioned as "first import of astropy", but in my case the user that calls the script, www-data in my case, does not have a home directory to save a config file.  I've set the variable "XDG_CONFIG_HOME" in /etc/apache2/envvars to a writeable directory (in this case is /var/www).  If I execute the script by itself from the www-data user works fine, it does not show that error message at all. Even if I check the /var/www directory after any execution I get the file created:

drwxr-xr-x  3 www-data www-data 4.0K Aug  1 10:08 .astropy

But even doing so my script keeps failing with the same error.

I've compile a very simple example on how to reproduce the error I get and included the error in the gist to:
https://gist.github.com/dpshelio/6131758

I would like to know if there's a way to avoid astropy to create/need such config file, as I don't see I need it for the use I'm going to provide.

------------- The solution

I've just found in the documentation: https://astropy.readthedocs.org/en/v0.2.4/configs/index.html

that when the XDG_CONFIG_HOME is defined we need to have a $XDG_CONFIG_HOME/astropy directory this is a bit contradictory if I don't have any file and I run python shell from that user and import astropy.  This later method creates a .astropy (with a dot) directory.

In any case, since I wouldn't have direct access on the real machine to the www-data user, is there a way we can avoid this without messing with that user?

Thanks a lot!!
David

[Michael Droettboom]

XDG_CONFIG_HOME as a concept comes from the XDG base directory standard from the Linux Standards Base:

http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

I agree that it is somewhat confusing that the default is to use ~/.astropy, or $XDG_CONFIG_HOME/astropy if XDG_CONFIG_HOME is defined.  This comes from the fact that we don't fully support the XDG standard, which states that the default should be ~/.config/astropy in the absence of XDG_CONFIG_HOME.  But we went a different way because we didn't want to tell Linux and Mac users something different about where to find their config files.

In any event, the issue you're running into is related to the log file, not a config file, which is stored in ~/.astropy/cache (by default) or $XDG_CACHE_HOME/astropy if XDG_CACHE_HOME is defined.  So if you need to move it from the default location, you need to set XDG_CACHE_HOME in addition to XDG_CONFIG_HOME.

While hopefully that workaround will work for you, it's possible that we could consider all of this required configuration as a bug.  If we can't write to the config directory, we should proceed with defaults, and if we can't write to the cache directory, it might be possible to use a /tmp directory.  I vaguely recall some discussion about this in the past, but looking at the code it doesn't appear that we do anything like this at the moment.

Cheers,

Mike

[Erik Bray]

Indeed, I thought we had an open issue for Astropy to just chill out
and (at most) emit a warning message if it has nowhere sensible to
write its config file and/or log. There is an issue. But I can't
find anything of the sort now, so maybe we should rethink this a bit
and open an issue for it.

Erik

[DVD PS]

Hi Michael, Erik,

Thanks for your replies.

XDG_CONFIG_HOME as a concept comes from the XDG base directory standard from the Linux Standards Base:

http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

I agree that it is somewhat confusing that the default is to use ~/.astropy, or $XDG_CONFIG_HOME/astropy if XDG_CONFIG_HOME is defined.  This comes from the fact that we don't fully support the XDG standard, which states that the default should be ~/.config/astropy in the absence of XDG_CONFIG_HOME.  But we went a different way because we didn't want to tell Linux and Mac users something different about where to find their config files.

In any event, the issue you're running into is related to the log file, not a config file, which is stored in ~/.astropy/cache (by default) or $XDG_CACHE_HOME/astropy if XDG_CACHE_HOME is defined.  So if you need to move it from the default location, you need to set XDG_CACHE_HOME in addition to XDG_CONFIG_HOME.

While hopefully that workaround will work for you, it's possible that we could consider all of this required configuration as a bug.  If we can't write to the config directory, we should proceed with defaults, and if we can't write to the cache directory, it might be possible to use a /tmp directory.  I vaguely recall some discussion about this in the past, but looking at the code it doesn't appear that we do anything like this at the moment.

The thing is that these workaround are not a really good solution for me.  The worked on the virtual machine where I test the webservice I'm developing, but I can foresee problems when I tell the sys-admin to make these changes into the system.  Even, if it's fine, it could be a source of problems in the future.

The 2 solutions I've come across is

1. Define the XDG_CONFIG_HOME for the www-data user and create the $XDG_CONFIG_HOME/astropy directory manually.
2. Add www-data to the list of sudo users to execute python as my personal user (I have not tested that one yet).

Both of these approaches needs the sysadmin to "touch" system configuration files which could bring future problems for other users and services.  All this it's because I need a config file which I don't think I really need for these automated services. So, would it be possible to remove that dependency/feature? could we set some flags to remove the need of the config file?

Thanks,

David

[DVD PS]

 Erik,

Indeed, I thought we had an open issue for Astropy to just chill out
and (at most) emit a warning message if it has nowhere sensible to
write its config file and/or log.  There is an issue.  But I can't
find anything of the sort now, so maybe we should rethink this a bit
and open an issue for it.

Do you mean this one: https://github.com/astropy/astropy/pull/536 ??
This does not solved my problem till I read properly the documentation and I created the $XDG_CONFIG_HOME/astropy directory.
But this, in my case, is not a very good option.

Thanks,
David

[Erik Bray]

On Fri, Aug 2, 2013 at 7:48 AM, DVD PS <dps....@gmail.com> wrote:
> Erik,
>
>
>> Indeed, I thought we had an open issue for Astropy to just chill out
>> and (at most) emit a warning message if it has nowhere sensible to
>> write its config file and/or log. There is an issue. But I can't
>> find anything of the sort now, so maybe we should rethink this a bit
>> and open an issue for it.
>
>
> Do you mean this one: https://github.com/astropy/astropy/pull/536 ??

No, I mean that's related I guess but that's not what I was thinking
of. I thought there was an issue specifically to use the defaults of a
config file can't be found or created.

> This does not solved my problem till I read properly the documentation and I
> created the $XDG_CONFIG_HOME/astropy directory.
> But this, in my case, is not a very good option.

I don't see why it's so bad--it's more or less the "right" way. The
objection that in the case of the webserver it requires a sysadmin to
create a directory under /var/www doesn't make sense to me since
presumably you already have that privilege if you're administering a
webserver. The reason that in your first use case it created
/var/www/.astropy is that /var/www is what the $HOME environment
variable was set to for your webserver.

That said, I agree there should be an option to just not touch the
filesystem at all if it isn't necessary to. I kind of hate it when
Python packages try to do anything on the filesystem at import time
beyond looking for other Python modules to import. But I think we
need to strike a balance here. The log and config files are useful
for ordinary users, and it has to be simple and more or less automatic
for those to be created in the normal use case. If $HOME/.astropy
can't be written, or if $XDG_CONFIG_HOME/astropy doesn't exist or
can't be created/written to then a config file shouldn't be used.

Erik

[DVD PS]

Hi Erik,

 

> This does not solved my problem till I read properly the documentation and I
> created the $XDG_CONFIG_HOME/astropy directory.
> But this, in my case, is not a very good option.

I don't see why it's so bad--it's more or less the "right" way.  The
objection that in the case of the webserver it requires a sysadmin to
create a directory under /var/www doesn't make sense to me since
presumably you already have that privilege if you're administering a
webserver.  The reason that in your first use case it created
/var/www/.astropy is that /var/www is what the $HOME environment
variable was set to for your webserver.

Not really, we are not the only users to that server, and either the options I have found could modify the behaviour to other services.  Probably, the less intrusive would be to define the $XDG_CONFIG_HOME (or create the astropy directory if the variables has been set already).
 

That said, I agree there should be an option to just not touch the
filesystem at all if it isn't necessary to. I kind of hate it when
Python packages try to do anything on the filesystem at import time
beyond looking for other Python modules to import.  But I think we
need to strike a balance here.  The log and config files are useful
for ordinary users, and it has to be simple and more or less automatic
for those to be created in the normal use case.  If $HOME/.astropy
can't be written, or if $XDG_CONFIG_HOME/astropy doesn't exist or
can't be created/written to then a config file shouldn't be used.

I agree with this, but I was wondering whether I was missing something.  Maybe the "webservice" use case could be used in the documentation as an example.  What do you think?  I could do a PR for that if it seems ok.  This may be redundant with the information already available, but I think it will help to future users. 

Thanks! and Have a good week!
David

[Erik Bray]

On Mon, Aug 5, 2013 at 8:39 AM, DVD PS <dps....@gmail.com> wrote:
> Hi Erik,
>
>>
>> > This does not solved my problem till I read properly the documentation
>> > and I
>> > created the $XDG_CONFIG_HOME/astropy directory.
>> > But this, in my case, is not a very good option.
>>
>> I don't see why it's so bad--it's more or less the "right" way. The
>> objection that in the case of the webserver it requires a sysadmin to
>> create a directory under /var/www doesn't make sense to me since
>> presumably you already have that privilege if you're administering a
>> webserver. The reason that in your first use case it created
>> /var/www/.astropy is that /var/www is what the $HOME environment
>> variable was set to for your webserver.
>
>
> Not really, we are not the only users to that server, and either the options
> I have found could modify the behaviour to other services. Probably, the
> less intrusive would be to define the $XDG_CONFIG_HOME (or create the
> astropy directory if the variables has been set already).

You can set environment variables on a process launched by a CGI
script without affecting other users of the web server. In your
specific example you were using shell_exec which makes this
particularly straightfoward since it forks a new shell to run your
program in. Though I'm pretty sure the webserver is doing this
anyways whenever it runs a CGI script. Just set the environment
variable in the script, not in /etc/apache/envvars.

>>
>> That said, I agree there should be an option to just not touch the
>> filesystem at all if it isn't necessary to. I kind of hate it when
>> Python packages try to do anything on the filesystem at import time
>> beyond looking for other Python modules to import. But I think we
>> need to strike a balance here. The log and config files are useful
>> for ordinary users, and it has to be simple and more or less automatic
>> for those to be created in the normal use case. If $HOME/.astropy
>> can't be written, or if $XDG_CONFIG_HOME/astropy doesn't exist or
>> can't be created/written to then a config file shouldn't be used.
>
>
> I agree with this, but I was wondering whether I was missing something.
> Maybe the "webservice" use case could be used in the documentation as an
> example. What do you think? I could do a PR for that if it seems ok. This
> may be redundant with the information already available, but I think it will
> help to future users.

I think this would be worthwhile, yes. matplotlib used to have
problems with this too, where it was annoying to use it in a web
server. I know at one time I was maintaining my own Debian package of
it that made some fixes so I could use it more easily in the webserver
appliance I was working on at the time. Astropy should be easier to
use in this context too, and in the meantime I would welcome better
documentation on the issue.

Erik

[Tom Aldcroft]

Along the lines of providing a solution for future users, I ran into a maddening process hang when trying to import astropy.table in a web application running under Apache + mod_wsgi.  After a fair bit of digging I finally discovered the real culprit to be `numpy.finfo(float)` in astropy/units/utils.py [1].  This more quickly led to a useful solution in terms of the Apache / mod_wsgi configuration [2]: 

   WSGIApplicationGroup %{GLOBAL}

This seems to be working thus far.  I also needed to set the XDG_CONFIG_HOME (and XDG_CACHE_HOME) env vars as noted, but I just did this in the settings.py of my Django project.

I wonder if this might warrant mention somewhere in the astropy docs, i.e. using astropy in a web server.

- Tom

[1]: https://github.com/astropy/astropy/blob/44376d65b9e8b381149e124cb9517f263b325308/astropy/units/utils.py#L26

[2]: https://github.com/numpy/numpy/issues/3837

 

--
You received this message because you are subscribed to the Google Groups "astropy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astropy-dev...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.