Dashboard: Internal server error

[Heather Klish]

Hi,

I'm having an issue where no matter what we click on in the dashboard, it reverts back to http://server/elasticsearch.  We see the following on the screen.

Archivematica

• Transfer
• Ingest
• Archival storage
• Preservation planning
• Access
• Administration

Internal server error

Sorry! Something wrong happened. Please, contact with your administrator.

I've checked and ElasticSearch is running.

In dashboard.log it looks like we have a lot of errors that look like this:

ProtocolError: ('Connection aborted.', HTTPException('got more than 100 headers',))

WARNING   2017-02-16 20:23:01  elasticsearch:base:log_request_fail:82:  PUT http://localhost:9200/aips/_mapping/aip [status:N/A request:0.045s]

Traceback (most recent call last):

  File "/usr/lib/python2.7/archivematica/dashboard/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 78, in perform_request

    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)

  File "/usr/lib/python2.7/archivematica/dashboard/lib/python2.7/site-packages/urllib3/connectionpool.py", line 640, in urlopen

    _stacktrace=sys.exc_info()[2])

  File "/usr/lib/python2.7/archivematica/dashboard/lib/python2.7/site-packages/urllib3/util/retry.py", line 238, in increment

    raise six.reraise(type(error), error, _stacktrace)

  File "/usr/lib/python2.7/archivematica/dashboard/lib/python2.7/site-packages/urllib3/connectionpool.py", line 595, in urlopen

    chunked=chunked)

  File "/usr/lib/python2.7/archivematica/dashboard/lib/python2.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request

    httplib_response = conn.getresponse(buffering=True)

  File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse

    response.begin()

  File "/usr/lib64/python2.7/httplib.py", line 476, in begin

    self.msg = HTTPMessage(self.fp, 0)

  File "/usr/lib64/python2.7/mimetools.py", line 25, in __init__

    rfc822.Message.__init__(self, fp, seekable)

  File "/usr/lib64/python2.7/rfc822.py", line 108, in __init__

    self.readheaders()

  File "/usr/lib64/python2.7/httplib.py", line 308, in readheaders

    raise HTTPException("got more than %d headers" % _MAXHEADERS)

I really have no idea where to start with troubleshooting!  Does anyone have any ideas?

Thank you!

Heather Klish

Systems Librarian

Tufts University

heathe...@tufts.edu

[Heather Klish]

We're using v. 1.5.1  on RHEL 7.  I've tried setting the DEBUG flag to True in /usr/share/archivematica/dashboard/settings/common.py

but that doesn't produce any additional information.

Heather

[Heather Klish]

Can anyone help me with this?   I'm stuck and really don't have any idea what to do beyond what I've already done.

Heather

On Thursday, February 16, 2017 at 4:37:41 PM UTC-5, Heather Klish wrote:

[Holly Becker]

Hi Heather,

That is a very odd problem.  It looks like ElasticSearch is returning over 100 HTTP headers when Archivematica tries to set up the ElasticSearch index structure.  We could modify Python to handle that many headers, but I would rather know why this is happening.

A few questions for context:

• What version of Archivematica are you using?
• How was Archivematica installed? Package install, Ansible, etc?
• Was this a fresh install or an upgrade?

If you're comfortable on the command line, can you post the output of these commands?  Text inside the backticks (e.g. `text` ) is the command to run on the machine with Archivematica installed.

What version of ElasticSearch are you running?  `curl -v -X GET 'localhost:9200'`
Was the AIP index created successfully? `curl -v -X GET localhost:9200/aips/_mapping/aip/` (this one may have a large output)
Is there any more information in the dashboard debug log at /var/log/archivematica/dashboard/dashboard.debug.log ?

Retrying the install on a new machine may also fix the problem, since this is an odd and unusual state to be in.

Thanks,
Holly

[Heather Klish]

Hi Holly,

• We're using v. 1.5.1  on RHEL 7.
  

• It's a fresh install.  I used the packages.  I followed the instructions here:  https://www.archivematica.org/en/docs/archivematica-1.5/admin-manual/installation/installation/#centos
• I ran the commands.  Here's what I got:

Version of ElasticSearch:

* About to connect() to localhost port 9200 (#0)

*   Trying ::1...

* Connected to localhost (::1) port 9200 (#0)

> GET / HTTP/1.1

> User-Agent: curl/7.29.0

> Host: localhost:9200

> Accept: */*

> 

< HTTP/1.1 200 OK

< content-type: application/json; charset=UTF-8

< content-length: 327

< 

{

  "name" : "OAodTDd",

  "cluster_name" : "elasticsearch",

  "cluster_uuid" : "VIYRMoIIRPSBISVFb22ZMQ",

  "version" : {

    "number" : "5.0.0",

    "build_hash" : "253032b",

    "build_date" : "2016-10-26T04:37:51.531Z",

    "build_snapshot" : false,

    "lucene_version" : "6.2.0"

  },

  "tagline" : "You Know, for Search"

}

* Connection #0 to host localhost left intact

AIP Index    (this is perhaps where my problem is)

* About to connect() to localhost port 9200 (#0)

*   Trying ::1...

* Connected to localhost (::1) port 9200 (#0)

> GET /aips/_mappin/aip/ HTTP/1.1

> User-Agent: curl/7.29.0

> Host: localhost:9200

> Accept: */*

> 

< HTTP/1.1 404 Not Found

< content-type: application/json; charset=UTF-8

< content-length: 61

< 

* Connection #0 to host localhost left intact

{"_index":"aips","_type":"_mappin","_id":"aip","found":false}

I've attached a file with more info from the log.  

Thank you so much!!!

Heather

[Qing Zou]

Hi Heather,

Oops, your ES version is too high. You need to find an older version (1.7).

Best,

Jason

[Heather Klish]

Thank you!