elasticsearch issues after devstack lms halted (Eucalyptus.2)

219 views
Skip to first unread message

Zachary Trabookis

unread,
Dec 13, 2016, 10:17:04 AM12/13/16
to General Open edX discussion
I setup an OPENEDX_RELEASE="open-release/eucalyptus.2" using these instructions http://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-eucalyptus.master/installation/analytics/install_analytics.html. After starting the LMS devstack and halting the virtual machine while that application was running I had to following these directions to unlock MongoDB.

sudo rm /edx/var/mongo/mongodb/mongod.lock
sudo mongod -repair --config /etc/mongod.conf
sudo chown -R mongodb:mongodb /edx/var/mongo/.
sudo service mongod start

After restarting the LMS I came across this issue with elasticsearch. Now I also noticed the http://localhost:8000/courses page shows this error "There was an error, try searching again". I've closed out the running LMS application and halted the VM and restarted it with no luck. How can I get past these elasticsearch issues below.

edxapp@precise64:~/edx-platform$ paver devstack lms
---> pavelib.servers.devstack
---> pavelib.prereqs.install_prereqs
---> pavelib.prereqs.install_node_prereqs
Node prereqs unchanged, skipping...
---> pavelib.prereqs.install_python_prereqs
---> pavelib.prereqs.uninstall_python_packages
Python uninstalls unchanged, skipping...
Python prereqs unchanged, skipping...
python manage.py cms --settings=devstack reindex_course --setup
/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/require/helpers.py:4: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
  from django.utils.importlib import import_module

2016-12-13 09:57:46,803 WARNING 3304 [py.warnings] importlib.py:9 - /edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/require/helpers.py:4: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
  from django.utils.importlib import import_module

2016-12-13 09:57:50,460 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (1): localhost
2016-12-13 09:57:50,461 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (3 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,461 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (2): localhost
2016-12-13 09:57:50,462 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (2 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,462 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (3): localhost
2016-12-13 09:57:50,463 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (1 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,463 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (4): localhost
2016-12-13 09:57:50,464 WARNING 3304 [elasticsearch] base.py:68 - HEAD http://localhost:9200/courseware_index [status:N/A request:0.005s]
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 46, in perform_request
    response = self.pool.urlopen(method, url, body, **kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 591, in urlopen
    raise MaxRetryError(self, url, e)
MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
2016-12-13 09:57:50,469 INFO 3304 [elasticsearch] base.py:70 - > None
2016-12-13 09:57:50,470 WARNING 3304 [elasticsearch] connection_pool.py:141 - Connection <Urllib3HttpConnection: http://localhost:9200> has failed for 1 times in a row, putting on 60 second timeout.
2016-12-13 09:57:50,470 INFO 3304 [elasticsearch] connection_pool.py:185 - Resurrecting connection <Urllib3HttpConnection: http://localhost:9200> (force=True).
2016-12-13 09:57:50,471 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (5): localhost
2016-12-13 09:57:50,472 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (3 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,472 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (6): localhost
2016-12-13 09:57:50,472 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (2 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,473 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (7): localhost
2016-12-13 09:57:50,473 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (1 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,474 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (8): localhost
2016-12-13 09:57:50,474 WARNING 3304 [elasticsearch] base.py:68 - HEAD http://localhost:9200/courseware_index [status:N/A request:0.003s]
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 46, in perform_request
    response = self.pool.urlopen(method, url, body, **kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 591, in urlopen
    raise MaxRetryError(self, url, e)
MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
2016-12-13 09:57:50,477 INFO 3304 [elasticsearch] base.py:70 - > None
2016-12-13 09:57:50,477 WARNING 3304 [elasticsearch] connection_pool.py:141 - Connection <Urllib3HttpConnection: http://localhost:9200> has failed for 2 times in a row, putting on 120 second timeout.
2016-12-13 09:57:50,478 INFO 3304 [elasticsearch] connection_pool.py:185 - Resurrecting connection <Urllib3HttpConnection: http://localhost:9200> (force=True).
2016-12-13 09:57:50,478 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (9): localhost
2016-12-13 09:57:50,479 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (3 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,479 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (10): localhost
2016-12-13 09:57:50,479 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (2 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,480 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (11): localhost
2016-12-13 09:57:50,480 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (1 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,481 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (12): localhost
2016-12-13 09:57:50,481 WARNING 3304 [elasticsearch] base.py:68 - HEAD http://localhost:9200/courseware_index [status:N/A request:0.003s]
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 46, in perform_request
    response = self.pool.urlopen(method, url, body, **kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 591, in urlopen
    raise MaxRetryError(self, url, e)
MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
2016-12-13 09:57:50,483 INFO 3304 [elasticsearch] base.py:70 - > None
2016-12-13 09:57:50,484 WARNING 3304 [elasticsearch] connection_pool.py:141 - Connection <Urllib3HttpConnection: http://localhost:9200> has failed for 3 times in a row, putting on 240 second timeout.
2016-12-13 09:57:50,484 INFO 3304 [elasticsearch] connection_pool.py:185 - Resurrecting connection <Urllib3HttpConnection: http://localhost:9200> (force=True).
2016-12-13 09:57:50,484 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (13): localhost
2016-12-13 09:57:50,485 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (3 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,485 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (14): localhost
2016-12-13 09:57:50,486 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (2 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,486 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (15): localhost
2016-12-13 09:57:50,487 WARNING 3304 [urllib3.connectionpool] connectionpool.py:603 - Retrying (1 attempts remain) after connection broken by 'error(111, 'Connection refused')': /courseware_index
2016-12-13 09:57:50,487 INFO 3304 [urllib3.connectionpool] connectionpool.py:257 - Starting new HTTP connection (16): localhost
2016-12-13 09:57:50,487 WARNING 3304 [elasticsearch] base.py:68 - HEAD http://localhost:9200/courseware_index [status:N/A request:0.003s]
Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 46, in perform_request
    response = self.pool.urlopen(method, url, body, **kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 607, in urlopen
    release_conn=release_conn, **response_kw)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 591, in urlopen
    raise MaxRetryError(self, url, e)
MaxRetryError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
2016-12-13 09:57:50,490 INFO 3304 [elasticsearch] base.py:70 - > None
2016-12-13 09:57:50,490 WARNING 3304 [elasticsearch] connection_pool.py:141 - Connection <Urllib3HttpConnection: http://localhost:9200> has failed for 4 times in a row, putting on 480 second timeout.
2016-12-13 09:57:50,491 ERROR 3304 [root] reindex_course.py:86 - Search Engine error - ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.error'>: [Errno 111] Connection refused)) caused by: MaxRetryError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.error'>: [Errno 111] Connection refused))
Traceback (most recent call last):
  File "/edx/app/edxapp/edx-platform/cms/djangoapps/contentstore/management/commands/reindex_course.py", line 84, in handle
    searcher = SearchEngine.get_search_engine(index_name)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/search/search_engine_base.py", line 50, in get_search_engine
    return search_engine_class(index=index) if search_engine_class else None
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/search/elastic.py", line 275, in __init__
    if not self._es.indices.exists(index=self.index_name):
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 70, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 149, in exists
    self.transport.perform_request('HEAD', _make_path(index), params=params)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 274, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore)
  File "/edx/app/edxapp/venvs/edxapp/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 51, in perform_request
    raise ConnectionError('N/A', str(e), e)
ConnectionError: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.error'>: [Errno 111] Connection refused)) caused by: MaxRetryError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /courseware_index (Caused by <class 'socket.error'>: [Errno 111] Connection refused))
---> pavelib.assets.update_assets
python manage.py lms --settings=devstack preprocess_assets lms/static/sass/*.scss lms/static/themed_sass
                Finished preprocessing lms assets.
xmodule_assets common/static/xmodule
                Finished processing xmodule assets.
mkdir_p path('common/static/common/js/vendor')
/bin/cp -rf node_modules/jquery/dist/jquery.js common/static/common/js/vendor
/bin/cp -rf node_modules/jquery-migrate/dist/jquery-migrate.js common/static/common/js/vendor
/bin/cp -rf node_modules/jquery.scrollto/jquery.scrollTo.js common/static/common/js/vendor
/bin/cp -rf node_modules/underscore/underscore.js common/static/common/js/vendor
/bin/cp -rf node_modules/underscore.string/dist/underscore.string.js common/static/common/js/vendor
/bin/cp -rf node_modules/picturefill/dist/picturefill.js common/static/common/js/vendor
/bin/cp -rf node_modules/backbone/backbone.js common/static/common/js/vendor
/bin/cp -rf node_modules/edx-ui-toolkit/node_modules/backbone.paginator/lib/backbone.paginator.js common/static/common/js/vendor
---> pavelib.assets.compile_coffeescript
node_modules/.bin/coffee --compile `find /edx/app/edxapp/edx-platform/lms /edx/app/edxapp/edx-platform/cms /edx/app/edxapp/edx-platform/common -type f -name "*.coffee"`
python manage.py lms --settings=devstack compile_sass lms 
Skipping theme asset compilation: enable theming to process themed assets
---> pavelib.assets.compile_sass
                Started compiling Sass:
Finished compiling 'common' sass.
Started compiling 'lms' Sass for 'system'.
Finished compiling 'lms' Sass for 'system'.
                Finished compiling Sass:
>> common/static/sass -> common/static/css in 0:00:00.018948s
>> lms/static/sass -> lms/static/css in 0:00:50.096556s
>> lms/static/certificates/sass -> lms/static/certificates/css in 0:00:12.936575s

Successful compilations:
--- "common" sass files.
--- lms sass for system.

---> pavelib.assets.watch_assets
find /edx/app/edxapp/edx-platform/lms /edx/app/edxapp/edx-platform/cms /edx/app/edxapp/edx-platform/common -type f -name "*.coffee"
Starting asset watcher...
Enter CTL-C to end
Performing system checks...

System check identified no issues (0 silenced).
December 13, 2016 - 09:59:19
Django version 1.8.13, using settings 'lms.envs.devstack'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.



Zachary Trabookis

unread,
Dec 14, 2016, 8:27:45 AM12/14/16
to General Open edX discussion
This is what my NAT network Port Forwarding section looks like with port 9200.


I'm guessing I'm getting this error because Elasticsearch site is not running at port 9200?




Zach

Zachary Trabookis

unread,
Dec 14, 2016, 9:50:49 AM12/14/16
to General Open edX discussion
Update: 

# From https://openedx.slack.com/archives/general/p1481721224000199

I ran `sudo service elasticsearch start` but it came back with
```* Starting ElasticSearch Server                                                                                                                                                                                            [ OK ] 
```

but when I check the status `sudo service elasticsearch status` it says
``` * elasticsearch is not running
```

[9:38]  
@tobz I checked the log files located in `/var/log/elasticsearch/*` but couldn't find anything relevant to an error. I'm going to revisit the `/etc/init.d/elasticsearch` script to see if it exists early for something.

[9:41]  
@tobz Actually I was looking at the wrong log file. When you look at `/etc/default/elasticsearch` it sets the log path to `LOG_DIR=/edx/var/log/elasticsearch`. I found some more relevant logs.


2016-12-14 14:46:52,635][WARN ][common.jna               ] Unknown mlockall error 0
[2016-12-14 14:46:52,703][INFO ][node                     ] [Wild Child] version[0.90.13], pid[8898], build[249c9c5/2014-03-25T15:27:12Z]
[2016-12-14 14:46:52,703][INFO ][node                     ] [Wild Child] initializing ...
[2016-12-14 14:46:52,708][INFO ][plugins                  ] [Wild Child] loaded [], sites []
[2016-12-14 14:46:54,214][WARN ][transport                ] [Wild Child] Registered two transport handlers for action discovery/zen/multicast, handlers: org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Multicast$
[2016-12-14 14:46:54,485][WARN ][transport                ] [Wild Child] Registered two transport handlers for action discovery/zen/multicast, handlers: org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Multicast$

< A bunch of these warnings >

[2016-12-14 14:46:54,839][WARN ][transport                ] [Wild Child] Registered two transport handlers for action discovery/zen/multicast, handlers: org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$MulticastPingResponseRequestHandler@20d3e3b1, org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$MulticastPingResponseRequestHandler@4389d21
[2016-12-14 14:46:54,856][ERROR][bootstrap                ] {0.90.13}: Initialization Failed ...
1) IllegalStateException[This is a proxy used to support circular references involving constructors. The object we're proxying is not constructed yet. Please wait until after injection has completed to use this object.]2) ElasticSearchIllegalArgumentException[Failed to resolve address for [[]]
        StringIndexOutOfBoundsException[String index out of range: -2]

Zach



On Tuesday, December 13, 2016 at 10:17:04 AM UTC-5, Zachary Trabookis wrote:

Zachary Trabookis

unread,
Dec 14, 2016, 3:24:04 PM12/14/16
to General Open edX discussion
I gave up and instead am provisioning devstack again to see if that fixes elasticsearch.

vagrant up && vagrant provision



On Tuesday, December 13, 2016 at 10:17:04 AM UTC-5, Zachary Trabookis wrote:

Dhiraj Panjwani

unread,
Jan 3, 2017, 1:53:58 PM1/3/17
to General Open edX discussion
Were you able to nail down the issue?

I have been struggling with this all day but haven't been able to solve it. Seems like Elasticsearch isn't able to initialize. The service isn't running either. I tried starting it manually but it instantly shuts down automatically.

Zachary Trabookis

unread,
Jan 10, 2017, 3:14:46 PM1/10/17
to General Open edX discussion
No I gave up. Ran into same issue that you did. Look at my vagrant commands above your last comment to see what I did to resolve this. Reprovisioned devstack.

Zach
Reply all
Reply to author
Forward
0 new messages