Issues creating files in app workspace in production

[sarva...@nasa.gov]

All,

I have an app that interacts with an external database to show VIC/DSSAT model output. The issue only exists in a production instance. I am using ogr2ogr utility to export a shapefile from the database to a temp directory in the app workspace and then I am uploading a zipped shapefile to a geoserver. The workflow works without a hitch in development, but in production because www-data generates the temp directory, ogr2ogr is unable to write the shapefile to the directory. I have saved files and ran processes in a temp directory on production before, but this time around nothing seems to be doing the trick. I initially had a problem with libgeos and gdal dependencies, I am still unable to export from the database even after fixing that issue. I have even given 777 permission to the /tmp folder and that didn't seem to make a difference. I have also tried to give 777 permission to the temp directory inside app_workspaces in production, still no luck. Any ideas on how to get past the permission issues without giving full privileges to www-data? Thank you!

https://github.com/SERVIR/DARWIN-Viewer/blob/master/tethysapp/darwin_viewer/model.py: The calculate_yield function fails to generate a shapefile in production.

Sarva

[swainn]

Sounds perplexing. What linux distro are you running on? It sounds symptomatic of an selinux problem. If I recall, you have to grant permissions to directories for users explicitly or something along those lines. Drew has some experience with RedHat selinux.

I'm also a little confused. Are you trying to have it write the file to /tmp directory or in the app workspace? 

The other confusing thing is that if it is called from within the app, it should be executed by the user running the web application, which should be www-data. So if the www-data user has write access to that directory, it should be fine. Again, it may be an selinux issue though. You have to tell selinux to allow that user to have write permissions to that directory.

[sarva...@nasa.gov]

It's been awhile since I have looked at the issues. Upon further inspection the www-data user is having a hard time running a subprocess through the app. The following subprocess is failing through the app, but it works just fine if you run it in the shell. https://github.com/SERVIR/DARWIN-Viewer/blob/master/tethysapp/darwin_viewer/model.py#L258

The same issues persists if you try to run an external python script within the app. If there is a command line utility, the www-data user is unable to execute the executable. Facing the same issues here: https://github.com/spence97/nasaaccess2.0/blob/master/tethysapp-nasaaccess2/tethysapp/nasaaccess2/model.py#L40. This is probably frowned upon, but I even added www-data to the sudo group, still no luck.

Has anyone successfully ran an executable through python subprocess in an app? Thanks!

[Zhiyu (Drew) Li]

We simply disabled selinux on centos as it gave us different weird issues several times. Not sure this would help your case though.

Thanks

Drew

--
You received this message because you are subscribed to the Google Groups "Tethys Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tethysplatfor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tethysplatform/bd6d79fa-ab79-4014-a1b1-0bec4b21241c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sarva Pulla]

I don't think it will help our case. But have you ever executed gdal command line or any other command line tool within an app in production, or something similar to the examples posted before? The issue is figuring out how to enable the www-data user to run executables or more specifically gdal through a subprocess in production. Might have to change something in nginx or uwsgi but haven't come across anyone even trying to do this...

[Zhiyu (Drew) Li]

I vaguely remember the gis app Shawn wrote calls external gdal executable to handle projection or something. I need to search for the code.

On Wed, Oct 24, 2018, 14:04 Sarva Pulla <sarv...@gmail.com> wrote:

I don't think it will help our case. But have you ever executed gdal command line or any other command line tool within an app in production, or something similar to the examples posted before? The issue is figuring out how to enable the www-data user to run executables or more specifically gdal through a subprocess in production. Might have to change something in nginx or uwsgi but haven't come across anyone even trying to do this...

--

You received this message because you are subscribed to the Google Groups "Tethys Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tethysplatfor...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tethysplatform/5aae5ce2-c862-439b-ab7b-ea38d3c9d9f5%40googlegroups.com.

[sarva...@nasa.gov]

Finally got around the issue. Giving the absolute path to the gdal executable and then changing the miniconda environment ownership to www-data fixed the issue. The resolved code looks like the following:

'''/home/ubuntu/tethys/miniconda/envs/tethys/bin/ogr2ogr -overwrite -f \"ESRI Shapefile\" {export_path}/{pgtable_name}.shp PG:"host={host} user={username} dbname={db} password={password}" -sql "{pg_sql_select}"'''

Do the following to change the ownership of the conda environment:

sudo chown www-data:www-data -R /home/ubuntu/tethys/miniconda/envs/tethys/

Thanks Nathan and Scott!