Graham help! Need with python requests module running in mod_wsgi environment

[Νίκος Βέργος]

Graham i know you helped me a lot but my script cannot work if i don;t het this oe sorted out and i'am trying for hours allready.

Please do help.

I have 3 wsgi scripts listening on 3 locations. What i'm trying to run an wsgi script from within another wsgi script with the following statement.

page = 'clientele'
pdata = requests.get( 'http://superhost.gr/' + page )
pdata = pdata.text + counter

page = the location of another wsgi app.

The error i'am getting when for i.e i try to load http://superhost.gr/clientele

mod_wsgi (pid=7152): Exception occurred processing WSGI script '/home/nikos/public_html/app.py' 
OSError: Apache/mod_wsgi failed to write response data: Broken pipe. 

The other script by itself executes normally but NOT from within my app.py script.

Any ideas on how to execute a wsgi app (b) from within a wsgi app(a) and store the response as html data?

[Graham Dumpleton]

On 10 Sep 2018, at 11:52 am, Νίκος Βέργος <nikos.at....@gmail.com> wrote:

Graham i know you helped me a lot but my script cannot work if i don;t het this oe sorted out and i'am trying for hours allready.

Please do help.

I have 3 wsgi scripts listening on 3 locations. What i'm trying to run an wsgi script from within another wsgi script with the following statement.

page = 'clientele'
pdata = requests.get( 'http://superhost.gr/' + page )
pdata = pdata.text + counter

Have you tried to add print() statements into your code, when it is running and what variables are at each point? If you use print(), what you print will appear in the Apache error log so you can see it. As per example in requests docs at:

http://docs.python-requests.org/en/master/

you don't even appear to check the response status code. Check such things, print out the values and see what it is getting.

page = the location of another wsgi app.

The error i'am getting when for i.e i try to load http://superhost.gr/clientele

mod_wsgi (pid=7152): Exception occurred processing WSGI script '/home/nikos/public_html/app.py' 
OSError: Apache/mod_wsgi failed to write response data: Broken pipe. 

This specific message from mod_wsgi indicates that the HTTP client making the request dropped the connection before the server could write the response.

So possibly an error is occurring in app.py if that is what is calling it.

Have you enabled Flask debug so that errors are shown in the browser for you. If you don't, Flask will just return a 500 error and may not actually log anything about the error. You may need to set up Flask logging so that it logs details of exceptions to the error log.

http://flask.pocoo.org/docs/1.0/logging/#logging

The other script by itself executes normally but NOT from within my app.py script.

Any ideas on how to execute a wsgi app (b) from within a wsgi app(a) and store the response as html data?

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Νικόλαος Κούρας]

1) How do i enable flask so that errors are shown in the browser for m, so i dont have to tail -f error_log all the time?

When i was using the flask development server i had this

app.run( host='superhost.gr', port=5000, debug=True )

Now i have commented this out becuas eitsw being run by mod_wsgi and its sending everythign to the error_log text file.

2) Regarding this code which not behalfing as it should

@app.route( '/<page>' )

@app.route( '/' )

def index( page=None ):

# read environment

page = request.args.get('page', 'index.html')

When i try via browser:

'http://superhost.gr/' page variable equals to 'index.html'

'http://superhost.gr/something' page variable still equals to 'index.html'

How is this possbile under mod_wsgi ?

You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/lZDr6zKcyUg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com.

[Graham Dumpleton]

On 10 Sep 2018, at 8:34 pm, Νικόλαος Κούρας <ni...@superhost.gr> wrote:

1) How do i enable flask so that errors are shown in the browser for m, so i dont have to tail -f error_log all the time?

When i was using the flask development server i had this

app.run( host='superhost.gr', port=5000, debug=True )

Now i have commented this out becuas eitsw being run by mod_wsgi and its sending everythign to the error_log text file.

You use:

    app.debug = True

See:

    http://flask.pocoo.org/docs/1.0/api/?highlight=flask%20debug#flask.Flask.debug

2) Regarding this code which not behalfing as it should

@app.route( '/<page>' )

@app.route( '/' )

def index( page=None ):

# read environment

page = request.args.get('page', 'index.html')

When i try via browser:

'http://superhost.gr/' page variable equals to 'index.html'

'http://superhost.gr/something' page variable still equals to 'index.html'

How is this possbile under mod_wsgi ?

You shouldn't get index.html in either case if doing things correctly.

The issue may because if I recollect correctly, you had all your .py files for the WSGI application under your DocumentRoot directory. You should not do that as it can be insecure and wrong configuration elsewhere can lead to someone being able to download your source code. Having things in their could also cause issues with Apache trying to remap URLs to index files.

Create a directory separate to your DocumentRoot directory and move the .py files into it. Then change the WSGIScriptAlias directives to reference the new location. You will also need to create a new Directory block for the new directory and grant access to being able to serve files from that directory.

[Νικόλαος Κούρας]

Like this you mean? ialso moved static/ files inside the new wsgi/ directory because even with the directive in the alias home=/home/nikos/public_html templates could not be found

====================================

DocumentRoot /home/nikos/public_html

<Directory /home/nikos/public_html>
    Require all granted
</Directory>

<Directory /home/nikos/public_html/wsgi>
    Require all granted
</Directory>


WSGIDaemonProcess app user=nikos group=nikos
WSGIScriptAliasMatch ^/(?!phpmyadmin) /home/nikos/public_html/wsgi/app.py process-group=app application-group=%{GLOBAL}

WSGIDaemonProcess clientele user=nikos group=nikos
WSGIScriptAlias /clientele /home/nikos/public_html/wsgi/clientele.py process-group=clientele application-group=%{GLOBAL}

WSGIDaemonProcess downloads user=nikos group=nikos
WSGIScriptAlias /downloads /home/nikos/public_html/wsgi/downloads.py process-group=downloads application-group=%{GLOBAL}

====================================

Still the ERROR remains the same after moving to wsgi/

[Graham Dumpleton]

It should not be anywhere under /home/nikos/public_html, not even a sub directory.

So use a parallel directory such as:

    /home/nikos/wsgi_apps

When testing use:

    curl http://superhost.gr/

Don't trust your browser, as it may have cached that accessing http://superhost.gr/ should redirect to http://superhost.gr/index.html

If curl works, you will need to clear your browser cache history for http://superhost.gr/

[Νικόλαος Κούρας]

[root@superhost public_html]# curl http://superhost.gr
index.html

[root@superhost public_html]# curl http://superhost.gr/clientele
index.html

As it shows even with an argument passed to the domain '/' 	page = request.args.get('page', 'index.html') returns the latter

Also:

application = Flask(__name__)
app = application
app.debug = True

Wont display error in browser but still in the error_log text file

[Graham Dumpleton]

Okay, no I clicked as to what you are trying to do.

If you want the path segment, don't use:

    page = request.args.get('page', 'index.html')

Just use the argument given to the function.

When you use request.args you are looking at query string parameters.

    http://superhost.gr?page=xxx.html

They are different things.

[Νικόλαος Κούρας]

/home/nikos/wsgi/ i places my apps and static remains at public_html Configuration now is:

DocumentRoot /home/nikos/public_html

<Directory /home/nikos/public_html>
    Require all granted
</Directory>

<Directory /home/nikos/wsgi>
    Require all granted
</Directory>


WSGIDaemonProcess app user=nikos group=nikos home=/home/nikos/public_html
WSGIScriptAliasMatch ^/(?!phpmyadmin) /home/nikos/public_html/wsgi/app.py process-group=app application-group=%{GLOBAL}

WSGIDaemonProcess clientele user=nikos group=nikos home=/home/nikos/public_html
WSGIScriptAlias /clientele /home/nikos/public_html/wsgi/clientele.py process-group=clientele application-group=%{GLOBAL}

WSGIDaemonProcess downloads user=nikos group=nikos home=/home/nikos/public_html
WSGIScriptAlias /downloads /home/nikos/public_html/wsgi/downloads.py process-group=downloads application-group=%{GLOBAL}

=========================

I haven't understood your last mail.

@app.route( '/' )
@app.route( '/<page>' )
def index( page=None ):

	# read environment
	pdata = ''
	page = request.args.get('page', 'index.html')

'page' variable is supposed to hold the URL parameter givn in browser as 'http://superhost.gr/something' no?!

It should instead be page = request.args.get(page, 'index.html') you mean?!

But why?!

[Graham Dumpleton]

On 10 Sep 2018, at 10:08 pm, Νικόλαος Κούρας <ni...@superhost.gr> wrote:

/home/nikos/wsgi/ i places my apps and static remains at public_html Configuration now is:

DocumentRoot /home/nikos/public_html

<Directory /home/nikos/public_html>
    Require all granted
</Directory>

<Directory /home/nikos/wsgi>
    Require all granted
</Directory>


WSGIDaemonProcess app user=nikos group=nikos home=/home/nikos/public_html
WSGIScriptAliasMatch ^/(?!phpmyadmin) /home/nikos/public_html/wsgi/app.py process-group=app application-group=%{GLOBAL}

WSGIDaemonProcess clientele user=nikos group=nikos home=/home/nikos/public_html
WSGIScriptAlias /clientele /home/nikos/public_html/wsgi/clientele.py process-group=clientele application-group=%{GLOBAL}

WSGIDaemonProcess downloads user=nikos group=nikos home=/home/nikos/public_html
WSGIScriptAlias /downloads /home/nikos/public_html/wsgi/downloads.py process-group=downloads application-group=%{GLOBAL}

=========================

I haven't understood your last mail.

@app.route( '/' )
@app.route( '/<page>' )
def index( page=None ):

	# read environment
	pdata = ''
	page = request.args.get('page', 'index.html')

You are overriding the 'page' argument given to the function, which is what you want, with the value of the 'page' query string parameter, which is a completely different thing, but because it isn't passed, falls back to using 'index.html'.

Comment out the line:

    page = request.args.get('page', 'index.html')

Do not use that.

[Νικόλαος Κούρας]

You are right!

please can you explain to me the difference between and argument and a query parameter?

Oh my God, i had almost a week this issue which you helped me resolve!

Thank you !!!

[Graham Dumpleton]

There are lots of resources out there you can read about them.

    https://en.wikipedia.org/wiki/Query_string

You might want to find a basic tutorial which explains to you how HTTP requests work as they should explain this and much more.

I presume you have gone through the Flask tutorial at least.

[Νικόλαος Κούρας]

I think i have understood the differnce:

url argument => http://superhost.gr/page

whcih we grab by its name from the view function definition

url query parameter => http://superhost.gr/?page=something

which we grab as 'request.args.get('page')

Correct ?!

[Graham Dumpleton]

Yes, although I don't know that they are universally known as 'url argument', but close enough.

[Νικόλαος Κούρας]

One *very* last thing please(you have been more than helpful !!):

Having this:

WSGIDaemonProcess clientele user=nikos group=nikos home=/home/nikos/wsgi

WSGIScriptAlias /clientele /home/nikos/wsgi/clientele.py process-group=clientele application-group=%{GLOBAL}

I cannot see to get a proper response when my app.py tries to load http://superhost.gr/clientele

if page in ('clientele', 'downloads'):

try:

r = requests.get( 'http://superhost.gr/' + page )

if r.ok:

pdata = r.text + counter

else:

pdata = r.status_code

except:

pdata = "Error retrieving webpage!"

Just this one please and iam done asking you!!

[Graham Dumpleton]

On 10 Sep 2018, at 11:16 pm, Νικόλαος Κούρας <ni...@superhost.gr> wrote:

One *very* last thing please(you have been more than helpful !!):

Having this:

WSGIDaemonProcess clientele user=nikos group=nikos home=/home/nikos/wsgi

WSGIScriptAlias /clientele /home/nikos/wsgi/clientele.py process-group=clientele application-group=%{GLOBAL}

I cannot see to get a proper response when my app.py tries to load http://superhost.gr/clientele

Impossible to see without knowing what you expect to see, what you are seeing or what errors you get. A few comments though.

if page in ('clientele', 'downloads'):

try:

r = requests.get( 'http://superhost.gr/' + page )

if r.ok:

pdata = r.text + counter

What is counter?

If counter is not defined yet this will fail.

If counter is an integer value, this will fail as you can't add strings and integers together.

else:

pdata = r.status_code

Status code is also likely an integer. So in some cases you are setting pdata to be a string and other times integer.

As have said before, add print() statements into your code to verify where it gets to and check values of things.

except:

You should never use naked except. Use:

    except Exception:

By using a naked except you would capture certain exceptions you shouldn't related to application exit. Ie., SystemExit, exception.

pdata = "Error retrieving webpage!"

Just this one please and iam done asking you!!

I am done for the night now.

[Νικόλαος Κούρας]

First of all have a good night's sleep!! :-)

The idea behind my code is to add a counter at the end of data of each html file and wsgi_mod file running:

Counter is just adding that value bases on previous calculations.

The problem is that every time i try to load 

superhost.gr/clientele 

or

superhost.gr/download

which i already have in httpd.conf an open mod_wsgi process for those 2 python apps

So every time app.py tries via request object to get the response of another wsgi app its keeps waiting and waiting for a long time until i ctrl-c

How will i be able to open via request method a wsgi app and return it in the form of html data?

=======================================================

@app.route( '/' )

@app.route( '/<page>' )

def index( page='index.html' ):

# read environment

pdata = ''

if ('.html', 'clientele', 'downloads') in page:

init()

database( page )

cur.execute( '''SELECT hits FROM pages WHERE url = %s''', page )

pagehit = cur.fetchone()[0]

# pick random line from the quote text file & random mp3 from the the music folder

quote = random.choice( list( open( '/home/nikos/public_html/data/private/quotes.txt', encoding='utf-8' ) ) )

music = random.choice( os.listdir( '/home/nikos/wsgi/static/mp3/' ) )

# add counter to html template and start rendering

counter = '''<table bgcolor=black    bordercolor=orangered>

<td><font size=3 color=lime> Αριθμός Επισκεπτών: </font></td> <td><a href="%s"><font size=3 color=plum> %s </font></a></td>

</table>

''' % ( url_for('log', page=page), pagehit )

if page in ('clientele', 'downloads'):

try:

r = requests.get( 'http://superhost.gr/' + page )

if r.ok:

pdata = r.text + counter

except Exception:

sys.exit(0)

elif page.endswith( '.html' ):

pdata = render_template( page, quote=quote, music=music, page=page, pagehit=pagehit ) + counter

return pdata

[Graham Dumpleton]

Did you add any print() statements to your code as I suggested to see what code gets run when you make requests?

On 11 Sep 2018, at 12:23 am, Νικόλαος Κούρας <ni...@superhost.gr> wrote:

First of all have a good night's sleep!! :-)

The idea behind my code is to add a counter at the end of data of each html file and wsgi_mod file running:

Counter is just adding that value bases on previous calculations.

The problem is that every time i try to load 

superhost.gr/clientele 

or

superhost.gr/download

which i already have in httpd.conf an open mod_wsgi process for those 2 python apps

If you are expecting code below in app.py to be invoked when accessing those URLs, it will not. The WSGIScriptAlias for /clientele and /download will intercept those requests and send them direct to the WSGI applications in the separate .pt files.

How are the clientele.py and download.py different/coded?

Sounds like you should only have WSGIScriptAlias for / pointing at app.py and not the others. But then, why even need the HTTP request back onto itself. I sort of think your understanding of how web applications in Python work with WSGI are a bit off.

Why aren't you doing everything in app.py file alone, with multiple separate routes for different functions?

Have you done the Flask tutorial all the way through?

Some other comments below.

So every time app.py tries via request object to get the response of another wsgi app its keeps waiting and waiting for a long time until i ctrl-c

How will i be able to open via request method a wsgi app and return it in the form of html data?

=======================================================

@app.route( '/' )

@app.route( '/<page>' )

def index( page='index.html' ):

# read environment

pdata = ''

if ('.html', 'clientele', 'downloads') in page:

This is bad approach as 'in' operator in this case will match substrings of page and not exact match.

init()

database( page )

cur.execute( '''SELECT hits FROM pages WHERE url = %s''', page )

I believe this is subject to SQL injection attacks. Should never format SQL queries using input provided by a user.

pagehit = cur.fetchone()[0]

Really should have error handling around any database stuff.

# pick random line from the quote text file & random mp3 from the the music folder

quote = random.choice( list( open( '/home/nikos/public_html/data/private/quotes.txt', encoding='utf-8' ) ) )

This is inefficient as reading the file every time and never closing the file object created, thus is relying on garbage collection for closing the file, which is not good.

music = random.choice( os.listdir( '/home/nikos/wsgi/static/mp3/' ) )

# add counter to html template and start rendering

counter = '''<table bgcolor=black    bordercolor=orangered>

<td><font size=3 color=lime> Αριθμός Επισκεπτών: </font></td> <td><a href="%s"><font size=3 color=plum> %s </font></a></td>

</table>

''' % ( url_for('log', page=page), pagehit )

if page in ('clientele', 'downloads'):

try:

r = requests.get( 'http://superhost.gr/' + page )

if r.ok:

pdata = r.text + counter

except Exception:

sys.exit(0)

elif page.endswith( '.html' ):

pdata = render_template( page, quote=quote, music=music, page=page, pagehit=pagehit ) + counter

return pdata

Except for possible misunderstanding around needing multiple WSGIScriptAlias directives, and having separate .py files for multiple WSGI applications, this all comes down to understanding how to use Flask, which is where other forums are better place to ask.

[Νίκος Βέργος]

I have tried removing WSGIScriptAlias for /clientele and /download enirely as well as putting them beneath '/' alias but i'm still getting the same waiting and waiting when i try for example to invoke superhost.gr/downloads request from within.py

I understand the logic of python we apps.

Before moving to flask and mod_wsgi i was using the same approach for my 3 script when i was using cgi mode.

Let's say i have 10 mod_wsgi applications than need to be opened from within 'app.py' via requests.get and then take the html output of those and add my counter data to that, how do you suggest i woud do it?

Its not practical to take all of the ohter sapps code and place it into app.py code.

[Graham Dumpleton]

Did you add print() statements in your code yet to help you understand when code is executing? I keep asking that and you keep ignoring it and show no evidence you have done so. You need to do this so you can debug it yourself.

As far as I can tell, your code is hanging because it is possibly going into a loop, as the request handler is requesting back onto itself. If you added the debug statements, whether this is occurring would be obvious.

At this point, I am sorry, but I can't help you any further. Please go use StackOverflow or some other mailing list or forum that can help you with general web programming. I don't have the time for this anymore, you have already sucked up way too much of my time over this. We solved the mod_wsgi configuration issues some time back. This list is for mod_wsgi and not general web programming help.

[Νικόλαος Κούρας]

Graham okey i'll think i can use this but i will move this to a python forum.

r = subprocess.check_output( 'curl http://superhost.gr/' + page )

about mod_wsgi:

i moved the .py files to /home/nikos/wsgi folder and decided to leave static/ and templates/ to public_html

The error_log complains that it cannot find the templates.

WSGIDaemonProcess clientele user=nikos group=nikos home=/home/nikos/wsgi

Also you told me to put

application = Flask(__name__)
app = application

app.debug = True

for seeing errors in browsers but this didnt not work, as  i only see internal server eror and have to look via console

You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/lZDr6zKcyUg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com.

[Graham Dumpleton]

Last time. From the Flask documentation:

    http://flask.pocoo.org/docs/1.0/api/?highlight=testing#flask.Flask.testing

Set app.testing instead of app.debug and exceptions should propagate back to mod_wsgi and it will log it. You really should though read the Flask documentation on how to have it log exceptions itself.

And make sure you use those print() statements I keep mentioning to add you own log message to track things though. So long as don't want to setup logging properly, and add your own, you will be in the dark.

Good luck.

Graham

[Νικόλαος Κούρας]

Having this:

app.debug = testing 

keeps provides me "internal server error", while the error_log is the one who readly displays the error messages.

in WSGIDaemonProcess clientele user=nikos group=nikos home=/home/nikos/wsgi

where does mod_wsgi tries to locate for other static files i.e. images in the same folder or in public_html

app.debug = testing keeps ptoviding me internal server error, while the error_log is the one who readly displays the error messages.

in WSGIDaemonProcess clientele user=nikos group=nikos home=/home/nikos/wsgi

where does mod_wsgi tries to locate for other static files i.e. images in the same folder or in public_html

because my wsgi apps are in wsgi folder while my static files are inside the public_html folder.

[Graham Dumpleton]

Read my email properly and read the docs.

That is not what I told you set.

--

[Νικόλαος Κούρας]

You said that instead app.debug =True i should set app.testing = True

You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/lZDr6zKcyUg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com.

[Νίκος Βέργος]

Τη Τρίτη, 11 Σεπτεμβρίου 2018 - 11:58:50 π.μ. UTC+3, ο χρήστης Graham Dumpleton έγραψε:

Last time. From the Flask documentation:

    http://flask.pocoo.org/docs/1.0/api/?highlight=testing#flask.Flask.testing

Set app.testing instead of app.debug and exceptions should propagate back to mod_wsgi and it will log it. You really should though read the Flask documentation on how to have it log exceptions itself.

And make sure you use those print() statements I keep mentioning to add you own log message to track things though. So long as don't want to setup logging properly, and add your own, you will be in the dark.

Good luck.

Goo news! I chnages the Logic and i created a module wi all required fucnton that did the adding counter process i wanted! it works!! And i do test variables with print.

I cant understand from the link you gave me how to enable error messages in browser though. It is nto clear at all to me.