Init scripts

[Kevin Campbell]

In order to make running the processes as persistent daemons easier, there are some scripts in the codebase now.

service/api
service/backend
service/frontend
service/proxy

These are standard init script format and will take (start | stop | restart | status) as a parameter. Process ids will be written into the 'run' directory I've added.
I'm sure there could be some improvements to layout here, but it's a starting point.

K

[Heather Piwowar]

These are great!  Trying to start frontend now.  What does it do that is different than api.py?  Is it for the webapp?  It is looking for frontend.py?

Thanks, 

Heather

K

--
You received this message because you are subscribed to the Google Groups "total-impact-dev" group.
To post to this group, send email to total-im...@googlegroups.com.
To unsubscribe from this group, send email to total-impact-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/total-impact-dev?hl=en.

[Kevin Campbell]

Heather,

Frontend is just a placeholder, I'll need to complete that. It should really be in total-impact-webapp.

K

[Heather Piwowar]

Cool, good to know.

Is all of your stuff in at this point: it should run?  I'm getting a 500 error on functional_tests (see below).

Ideas?  Thoughts on how you'd start debugging this?  

Seems like maybe the proxy isn't running properly on my machine.

This page works

http://localhost:8080/test

but this one 

http://www.localhost.com:8080/http://datadryad.org/solr/search/select/?q=dc.identifier:10.5061/dryad.7898&fl=dc.identifier.uri,dc.title

returns a 500.  Is that what you'd expect?

Heather

Heather-Piwowars-MacBook-Pro:total-impact-core hpiwowar$ git checkout master

Switched to branch 'master'

Heather-Piwowars-MacBook-Pro:total-impact-core hpiwowar$ git pull

Already up-to-date.

Heather-Piwowars-MacBook-Pro:total-impact-core hpiwowar$ service/backend start

Starting backend server: Started

Heather-Piwowars-MacBook-Pro:total-impact-core hpiwowar$ service/api start

Starting api server: Started

Heather-Piwowars-MacBook-Pro:total-impact-core hpiwowar$ service/proxy start

Starting proxy server: Started

Heather-Piwowars-MacBook-Pro:total-impact-core hpiwowar$ extras/providers_check.py 

Checking Dryad provider

Checking Wikipedia provider

Checking Github provider

Checks complete, no data inconsistencies were found

Heather-Piwowars-MacBook-Pro:total-impact-core hpiwowar$ extras/functional_test.py 

http://localhost:5001/item/doi/10.5061/dryad.7898

http://localhost:5001/item/doi/10.1371/journal.pcbi.1000361

http://localhost:5001/item/github/egonw/gtd

Traceback (most recent call last):

  File "extras/functional_test.py", line 141, in <module>

    itemdata = ti.request_item_result(itemid[item_type][idx])

  File "extras/functional_test.py", line 36, in request_item_result

    response = urllib2.urlopen(req)

  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 124, in urlopen

    return _opener.open(url, data, timeout)

  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 389, in open

    response = meth(req, response)

  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 502, in http_response

    'http', request, response, code, msg, hdrs)

  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 427, in error

    return self._call_chain(*args)

  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 361, in _call_chain

    result = func(*args)

  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib2.py", line 510, in http_error_default

    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

urllib2.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR

[Heather Piwowar]

whoops, copy and paste error.  localhost breaks too, not just localhost.com :)

http://localhost:8080/http://datadryad.org/solr/search/select/?q=dc.identifier:10.5061/dryad.7898&fl=dc.identifier.uri,dc.title

http://localhost:8080/https://api.github.com/users/egonw/repos

both return 500s.

Heather

[Kevin Campbell]

Heather,

The functional tests are working fine for me, and that item/url should be covered by the proxy.

Can you check your proxy port and code version? I'm on the latest trunk. The port in the codebase is 5001 (hardcoded)

K

[Heather Piwowar]

fresh clone, running with all defaults.

It seems that proxy points to testenv.cfg which includes a proxy setting on port 8081

whereas the providers_test_proxy defaults to port 8080?

Or maybe this isn't a proxy issue at all?  If I make sure the PROXY config setting isn't set (by avoiding testenv.cfg for example) then functional_tests should work without a proxy, is that right?

My api on http://localhost:5001/ seems to be working fine. Functional_tests makes the first few calls ok then fails.  

http://localhost:5001/item/doi/10.5061/dryad.7898

http://localhost:5001/item/doi/10.1371/journal.pcbi.1000361

http://localhost:5001/item/github/egonw/gtd

http://localhost:5001/item/3a205d2fbdde4c42af4e1c03940f624a

Traceback (most recent call last):

  File "extras/functional_test.py", line 142, in <module>

    itemdata = ti.request_item_result(itemid[item_type][idx])

  File "extras/functional_test.py", line 37, in request_item_result

    response = urllib2.urlopen(req)

...

urllib2.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR

Maybe it is a problem with my couch config or something??  Looking.

Heather

[Kevin Campbell]

Sorry, yes the proxy is defaulting to 8080.

The functional tests don't honour the PROXY settings in the config. There's certainly some consistency changes needed done here, which I'll try and improve. I am still working on these startup scripts and daemon settings, so there are pieces needing ironed out.

Can you check the details in logs/total-impact.log? Also, try running api in non-daemonized mode, which should show the error details on stdout.

K

[Heather Piwowar]

I just stopped and started memcached and it works now.

Heather

[Heather Piwowar]

Cool. 

Yup, to be a bit more clear, load_test and functional_test and provider_check work now when I do this:

clean clone

start couch

start memcached

python totalimpact/backend.py

python totalimpact/api.py

hehe this is just what you said should work a few hours ago :)  I lost my footing when some of the daemon changes in backend were broken and then I forgot to go back to first principles when I caught a glimpse of the new shiny scripts.

Main thing: it works now!  I'm onward with the merge!

Kevin, thanks again for all of this infrastructure.

Heather