Fwd: Build failed in Jenkins: total-impact #58

[Jason Priem]

Jenkins is going to keep breaking until we either

1. upload secrets.py to the server (not sure of creds needed to do this)

2. use env var technique (also needs login to jenkins server, plus heather not crazy about env vars) or

3. take the amateur-hour approach and check secrets.py into source control.

I'm in favor of #3. We're not dealing with national security here, we can fix it later.

j

---------- Forwarded message ----------
From: Jenkins on orchid.weegie.net <jen...@weegie.net>
Date: Tue, May 15, 2012 at 9:12 PM
Subject: Build failed in Jenkins: total-impact #58
To: k...@weegie.net, hpiw...@gmail.com, j...@jasonpriem.org


See <http://orchid.vm.bytemark.co.uk:8500/job/total-impact/58/changes>

Changes:

[jp] adds secrets.py to .gitignore

[jp] fixes provenance_url display in template...no longer looks in static_meta

------------------------------------------
[...truncated 816 lines...]
 File "/usr/lib/pymodules/python2.6/nose/case.py", line 364, in setUp
   try_run(self.inst, ('setup', 'setUp'))
 File "/usr/lib/pymodules/python2.6/nose/util.py", line 487, in try_run
   return func()
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/test/unit_tests/providers/test_mendeley.py",> line 26, in setUp
   ProviderTestCase.setUp(self)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/test/unit_tests/providers/common.py",> line 82, in setUp
   self.provider = ProviderFactory.get_provider(self.provider_name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/provider.py",> line 14, in get_provider
   provider_module = importlib.import_module('totalimpact.providers.'+provider_name)
 File "/usr/lib/pymodules/python2.6/importlib/__init__.py", line 37, in import_module
   __import__(name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/mendeley.py",> line 2, in <module>
   from totalimpact.providers.secrets import Mendeley_key
ImportError: No module named secrets

======================================================================
ERROR: test_09_get_providers (test.unit_tests.providers.test_provider.Test_Provider)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/test/unit_tests/providers/test_provider.py",> line 94, in test_09_get_providers
   providers = ProviderFactory.get_providers(self.provider_configs)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/provider.py",> line 25, in get_providers
   prov = ProviderFactory.get_provider(provider_name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/provider.py",> line 14, in get_provider
   provider_module = importlib.import_module('totalimpact.providers.'+provider_name)
 File "/usr/lib/pymodules/python2.6/importlib/__init__.py", line 37, in import_module
   __import__(name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/mendeley.py",> line 2, in <module>
   from totalimpact.providers.secrets import Mendeley_key
ImportError: No module named secrets

======================================================================
ERROR: test_item_post_urldecodes (test.unit_tests.test_api.TestItem)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/test/unit_tests/test_api.py",> line 151, in test_item_post_urldecodes
   saved_item = json.loads(resp.data)
 File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
   return _default_decoder.decode(s)
 File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
   obj, end = self.raw_decode(s, idx=_w(s, 0).end())
 File "/usr/lib/python2.6/json/decoder.py", line 338, in raw_decode
   raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
-------------------- >> begin captured logging << --------------------
totalimpact.api: ERROR: Exception on /item/9a25542a0a8b49de92bce3c533744821 [GET]
Traceback (most recent call last):
 File "/usr/lib/python2.6/dist-packages/flask/app.py", line 861, in wsgi_app
   rv = self.dispatch_request()
 File "/usr/lib/python2.6/dist-packages/flask/app.py", line 696, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/api.py",> line 180, in item
   item_dict = make_item_dict(tiid)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/api.py",> line 154, in make_item_dict
   app.config["PROVIDERS"])
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/models.py",> line 182, in get
   provider = provider_maker(provider_name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/provider.py",> line 14, in get_provider
   provider_module = importlib.import_module('totalimpact.providers.'+provider_name)
 File "/usr/lib/pymodules/python2.6/importlib/__init__.py", line 37, in import_module
   __import__(name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/mendeley.py",> line 2, in <module>
   from totalimpact.providers.secrets import Mendeley_key
ImportError: No module named secrets
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: test.unit_tests.test_models.TestItemFactory.test_adds_provenance_url
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/lib/pymodules/python2.6/nose/case.py", line 183, in runTest
   self.test(*self.arg)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/test/unit_tests/test_models.py",> line 274, in test_adds_provenance_url
   default_settings.PROVIDERS)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/models.py",> line 182, in get
   provider = provider_maker(provider_name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/provider.py",> line 14, in get_provider
   provider_module = importlib.import_module('totalimpact.providers.'+provider_name)
 File "/usr/lib/pymodules/python2.6/importlib/__init__.py", line 37, in import_module
   __import__(name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/mendeley.py",> line 2, in <module>
   from totalimpact.providers.secrets import Mendeley_key
ImportError: No module named secrets

======================================================================
ERROR: test.unit_tests.test_models.TestItemFactory.test_adds_static_meta
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/lib/pymodules/python2.6/nose/case.py", line 183, in runTest
   self.test(*self.arg)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/test/unit_tests/test_models.py",> line 263, in test_adds_static_meta
   default_settings.PROVIDERS)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/models.py",> line 182, in get
   provider = provider_maker(provider_name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/provider.py",> line 14, in get_provider
   provider_module = importlib.import_module('totalimpact.providers.'+provider_name)
 File "/usr/lib/pymodules/python2.6/importlib/__init__.py", line 37, in import_module
   __import__(name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/mendeley.py",> line 2, in <module>
   from totalimpact.providers.secrets import Mendeley_key
ImportError: No module named secrets

======================================================================
ERROR: test.unit_tests.test_models.TestItemFactory.test_get
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/lib/pymodules/python2.6/nose/case.py", line 183, in runTest
   self.test(*self.arg)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/test/unit_tests/test_models.py",> line 227, in test_get
   default_settings.PROVIDERS)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/models.py",> line 182, in get
   provider = provider_maker(provider_name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/provider.py",> line 14, in get_provider
   provider_module = importlib.import_module('totalimpact.providers.'+provider_name)
 File "/usr/lib/pymodules/python2.6/importlib/__init__.py", line 37, in import_module
   __import__(name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/mendeley.py",> line 2, in <module>
   from totalimpact.providers.secrets import Mendeley_key
ImportError: No module named secrets

======================================================================
ERROR: test.unit_tests.test_models.TestItemFactory.test_get_creates_metrics
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/lib/pymodules/python2.6/nose/case.py", line 183, in runTest
   self.test(*self.arg)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/test/unit_tests/test_models.py",> line 244, in test_get_creates_metrics
   default_settings.PROVIDERS)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/models.py",> line 182, in get
   provider = provider_maker(provider_name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/provider.py",> line 14, in get_provider
   provider_module = importlib.import_module('totalimpact.providers.'+provider_name)
 File "/usr/lib/pymodules/python2.6/importlib/__init__.py", line 37, in import_module
   __import__(name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/mendeley.py",> line 2, in <module>
   from totalimpact.providers.secrets import Mendeley_key
ImportError: No module named secrets

======================================================================
FAIL: test_item_get_success_fakeid (test.unit_tests.test_api.TestItem)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/test/unit_tests/test_api.py",> line 138, in test_item_get_success_fakeid
   assert_equals(response.status_code, 200)
AssertionError: 500 != 200
-------------------- >> begin captured stdout << ---------------------
"def1884e6015404e9d077c65faddbfb5"
<Response streamed [500 INTERNAL SERVER ERROR]>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>


--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
totalimpact.api: ERROR: Exception on /item/def1884e6015404e9d077c65faddbfb5 [GET]
Traceback (most recent call last):
 File "/usr/lib/python2.6/dist-packages/flask/app.py", line 861, in wsgi_app
   rv = self.dispatch_request()
 File "/usr/lib/python2.6/dist-packages/flask/app.py", line 696, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/api.py",> line 180, in item
   item_dict = make_item_dict(tiid)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/api.py",> line 154, in make_item_dict
   app.config["PROVIDERS"])
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/models.py",> line 182, in get
   provider = provider_maker(provider_name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/provider.py",> line 14, in get_provider
   provider_module = importlib.import_module('totalimpact.providers.'+provider_name)
 File "/usr/lib/pymodules/python2.6/importlib/__init__.py", line 37, in import_module
   __import__(name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/mendeley.py",> line 2, in <module>
   from totalimpact.providers.secrets import Mendeley_key
ImportError: No module named secrets
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_returns_html_view (test.unit_tests.test_api.TestItem)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/test/unit_tests/test_api.py",> line 187, in test_returns_html_view
   assert returned.find('h4') is not None
AssertionError:
-------------------- >> begin captured stdout << ---------------------
('thisisauuid', '2-445b40684722689cccef56438066ac88')

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
totalimpact.api: ERROR: Exception on /item/thisisauuid.html [GET]
Traceback (most recent call last):
 File "/usr/lib/python2.6/dist-packages/flask/app.py", line 861, in wsgi_app
   rv = self.dispatch_request()
 File "/usr/lib/python2.6/dist-packages/flask/app.py", line 696, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/api.py",> line 180, in item
   item_dict = make_item_dict(tiid)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/api.py",> line 154, in make_item_dict
   app.config["PROVIDERS"])
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/models.py",> line 182, in get
   provider = provider_maker(provider_name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/provider.py",> line 14, in get_provider
   provider_module = importlib.import_module('totalimpact.providers.'+provider_name)
 File "/usr/lib/pymodules/python2.6/importlib/__init__.py", line 37, in import_module
   __import__(name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/mendeley.py",> line 2, in <module>
   from totalimpact.providers.secrets import Mendeley_key
ImportError: No module named secrets
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_returns_json_default (test.unit_tests.test_api.TestItem)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/test/unit_tests/test_api.py",> line 170, in test_returns_json_default
   assert_equals(response.headers[0][1], 'application/json')
AssertionError: 'text/html' != 'application/json'
-------------------- >> begin captured stdout << ---------------------
('thisisauuid', '3-6060f81cd466794f04bf7a4c326b09a5')

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
totalimpact.api: ERROR: Exception on /item/thisisauuid [GET]
Traceback (most recent call last):
 File "/usr/lib/python2.6/dist-packages/flask/app.py", line 861, in wsgi_app
   rv = self.dispatch_request()
 File "/usr/lib/python2.6/dist-packages/flask/app.py", line 696, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/api.py",> line 180, in item
   item_dict = make_item_dict(tiid)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/api.py",> line 154, in make_item_dict
   app.config["PROVIDERS"])
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/models.py",> line 182, in get
   provider = provider_maker(provider_name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/provider.py",> line 14, in get_provider
   provider_module = importlib.import_module('totalimpact.providers.'+provider_name)
 File "/usr/lib/pymodules/python2.6/importlib/__init__.py", line 37, in import_module
   __import__(name)
 File "<http://orchid.vm.bytemark.co.uk:8500/job/total-impact/ws/totalimpact/providers/mendeley.py",> line 2, in <module>
   from totalimpact.providers.secrets import Mendeley_key
ImportError: No module named secrets
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 190 tests in 76.291s

FAILED (SKIP=15, errors=35, failures=3)
Build step 'Execute shell' marked build as failure

--
Jason Priem
UNC Royster Scholar
School of Information and Library Science
University of North Carolina at Chapel Hill

[Kevin Campbell]

On Wed, May 16, 2012 at 2:17 AM, Jason Priem <j...@jasonpriem.org> wrote:

Jenkins is going to keep breaking until we either

1. upload secrets.py to the server (not sure of creds needed to do this)

2. use env var technique (also needs login to jenkins server, plus heather not crazy about env vars) or

3. take the amateur-hour approach and check secrets.py into source control.

Using env vars to refer to a local settings is pretty standard for handling multiple environments. I think it would be more consistent to have config/jenkins-env.conf (not in version control) which is put in place by Jenkins and run with the env variable set.

K

[Kevin Campbell]

We will be getting URL calls made similar to the following:

  http://api.mendeley.com/oapi/documents/details/10.1371%252Fjournal.pcbi.1000361?type=doi&consumer_key=

We should define a test consumer key (lets say 'testkey') used for all providers, for when using with the test proxy. This would then give:

  config/jenkins-env.conf  (has production keys set, uses live services)
  config/testenv.conf         (has all keys set to 'testkey', uses the proxy)

Or if you would rather arrange configs differently, something analogous to give the two environments.

If you are adding more providers, provider logic or tests, we should be adding the relevant snippets to the proxy as we go. I know that's more work, but I think it's more than worthwhile to have this maintained. We can either hardcode in the testkey to the proxy script, or change those to regexes if that's preferred.

Regards,
Kevin

[Jason Priem]

Let's just make this simple for now: leave the sample config checked into source control, make the real config by hand. upload the real config by hand to the jenkins and production servers. for the test runs, no harm in using the real secret passwords...the proxy shouldn't care what api keys you're using...it's not like we're going to require ourselves to authenticate.

I think this is the simplest thing that can possibly work, and only costs us the trouble of hand-uploading new secrets when we get them. To be honest, this is going to be a rare enough occurrence that I'm fine with it.

agreed, we should be adding canned responses for new providers has we add them, for sure.

--
You received this message because you are subscribed to the Google Groups "total-impact-dev" group.
To post to this group, send email to total-im...@googlegroups.com.
To unsubscribe from this group, send email to total-impact-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/total-impact-dev?hl=en.

[Heather Piwowar]

Agree with this approach for dealing with secret keys for now.

Yup, I'm adding canned provider pages and response checks as I add new providers.  I'm only adding the checks to alt_funtional_tests for now because we'll want to refactor the checks so that they can be shared with load_tests etc.

Heather