Error while uploading files on pithos web /synnefo v0.18.1

[Martin Njau]

Hi,


I am in the process of building a new synnefo cluster but i am encoutering some issues with pithos

I am getting below errors on the web console while i try upload files on pithos.

Chunked promise failed Object { jqXHR={...}, textStatus="error", errorThrown="INTERNAL SERVER ERROR"}

snf-ui-...f4e9.js (line 4)

Chunked upload failed. Retrying with XHR.

snf-ui-...f4e9.js (line 4)

Chunked promise failed Object { jqXHR={...}, textStatus="error", errorThrown="INTERNAL SERVER ERROR"}

snf-ui-...f4e9.js (line 4)

Chunked upload failed. Retrying with XHR.

snf-ui-...f4e9.js (line 4)

Chunked promise failed Object { jqXHR={...}, textStatus="error", errorThrown="INTERNAL SERVER ERROR"}

snf-ui-...f4e9.js (line 4)

Chunked upload failed. Retrying with XHR.

snf-ui-...f4e9.js (line 4)

I highly appreciate any leads.


Thanks

Martin

[Martin Njau]

Below are the logs that i get when i try upload any file.

# tail -f /var/log/synnefo/gunicorn.log

2017-02-15 14:07:23,178 [14849] django.request __init__ [ERROR] Internal Server Error: /pithos/object-store/v1/488f8b20-f871-4465-8803-5bbeb3ccf151/pithos/DOC-20161128-WA0011.pdf

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/snf_django/lib/api/__init__.py", line 115, in wrapper

    response = func(request, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/pithos/api/util.py", line 1094, in wrapper

    response = func(request, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/pithos/api/functions.py", line 959, in object_write

    content_type, hashmap, checksum, 'pithos', meta, True, permissions

  File "/usr/lib/python2.7/dist-packages/pithos/backends/modular.py", line 158, in wrapper

    result = func(self, *args, **kw)

  File "/usr/lib/python2.7/dist-packages/pithos/backends/modular.py", line 1614, in update_object_hashmap

    missing = self.store.block_search(map_)

  File "/usr/lib/python2.7/dist-packages/pithos/backends/lib/hashfiler/store.py", line 67, in block_search

    return self.blocker.block_ping(map)

  File "/usr/lib/python2.7/dist-packages/pithos/backends/lib/hashfiler/blocker.py", line 38, in block_ping

    return self.archip_blocker.block_ping(hashes)

  File "/usr/lib/python2.7/dist-packages/pithos/backends/lib/hashfiler/archipelagoblocker.py", line 99, in block_ping

    if h not in notfound and not self._check_rear_block(h):

  File "/usr/lib/python2.7/dist-packages/pithos/backends/lib/hashfiler/archipelagoblocker.py", line 73, in _check_rear_block

    ioctx = self.ioctx_pool.pool_get()

  File "/usr/lib/python2.7/dist-packages/objpool/__init__.py", line 176, in pool_get

    obj = self._pool_create()

  File "/usr/lib/python2.7/dist-packages/pithos/workers/glue.py", line 82, in _pool_create

    self.cnt)

  File "/usr/lib/python2.7/dist-packages/archipelago/common.py", line 894, in __init__

    ctx = segment.join()

  File "/usr/lib/python2.7/dist-packages/archipelago/common.py", line 559, in join

    cast(0, cb_null_ptrtype))

  File "/usr/lib/python2.7/dist-packages/archipelago/common.py", line 551, in errcheck

    % (config['SEGMENT_NAME'], os.strerror(e)))

Error: Cannot join segment 'archipelago': Permission denied

2017-02-15 14:07:25,253 [14848] django.request __init__ [ERROR] Internal Server Error: /pithos/object-store/v1/488f8b20-f871-4465-8803-5bbeb3ccf151/pithos/DOC-20161128-WA0011.pdf

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/snf_django/lib/api/__init__.py", line 115, in wrapper

    response = func(request, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/pithos/api/util.py", line 1094, in wrapper

    response = func(request, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/pithos/api/functions.py", line 994, in object_write_form

    if 'X-Object-Data' not in request.FILES:

  File "/usr/lib/python2.7/dist-packages/django/core/handlers/wsgi.py", line 196, in _get_files

    self._load_post_and_files()

  File "/usr/lib/python2.7/dist-packages/django/http/__init__.py", line 368, in _load_post_and_files

    self._post, self._files = self.parse_file_upload(self.META, data)

  File "/usr/lib/python2.7/dist-packages/django/http/__init__.py", line 328, in parse_file_upload

    return parser.parse()

  File "/usr/lib/python2.7/dist-packages/django/http/multipartparser.py", line 138, in parse

    self.handle_file_complete(old_field_name, counters)

  File "/usr/lib/python2.7/dist-packages/django/http/multipartparser.py", line 241, in handle_file_complete

    file_obj = handler.file_complete(counters[i])

  File "/usr/lib/python2.7/dist-packages/pithos/api/util.py", line 803, in file_complete

    self.put_data(l)

  File "/usr/lib/python2.7/dist-packages/pithos/api/util.py", line 781, in put_data

    self.file.hashmap.append(self.backend.put_block(block))

  File "/usr/lib/python2.7/dist-packages/pithos/backends/modular.py", line 2084, in put_block

    return binascii.hexlify(self.store.block_put(data))

  File "/usr/lib/python2.7/dist-packages/pithos/backends/lib/hashfiler/store.py", line 59, in block_put

    hashes, absent = self.blocker.block_stor((data,))

  File "/usr/lib/python2.7/dist-packages/pithos/backends/lib/hashfiler/blocker.py", line 56, in block_stor

    (hashes, missing) = self.archip_blocker.block_stor(blocklist)

  File "/usr/lib/python2.7/dist-packages/pithos/backends/lib/hashfiler/archipelagoblocker.py", line 174, in block_stor

    self._check_rear_block(h)]

  File "/usr/lib/python2.7/dist-packages/pithos/backends/lib/hashfiler/archipelagoblocker.py", line 73, in _check_rear_block

    ioctx = self.ioctx_pool.pool_get()

  File "/usr/lib/python2.7/dist-packages/objpool/__init__.py", line 176, in pool_get

    obj = self._pool_create()

  File "/usr/lib/python2.7/dist-packages/pithos/workers/glue.py", line 82, in _pool_create

    self.cnt)

  File "/usr/lib/python2.7/dist-packages/archipelago/common.py", line 894, in __init__

    ctx = segment.join()

  File "/usr/lib/python2.7/dist-packages/archipelago/common.py", line 559, in join

    cast(0, cb_null_ptrtype))

  File "/usr/lib/python2.7/dist-packages/archipelago/common.py", line 551, in errcheck

    % (config['SEGMENT_NAME'], os.strerror(e)))

Error: Cannot join segment 'archipelago': Permission denied

[Filippos Giannakos]

Hello Martin,

This looks like that the user/group configuration is not correct. Since Pithos integrates with
Archipelago over a shared memory segment, both processes should run as the same group
to both have permissions on the shared segment. More specifically archipelago should be run
as archipelago/synnefo while gunicorn as synnefo/synnefo. Both these settings should be
configured on the configuration files (‘/etc/archipelago/archipelago.conf’, ‘/etc/gunicorn/synnefo’
respectively).

So check and properly configure the values and then restart the worker using:

# service gunicorn stop
# archipelago stop
# rm /dev/shm/posixfd/* <— to clean up any pipes using wrong permissions
# archipelago start
# service gunicorn stop

Kind Regards,
Filippos

> --
> You received this message because you are subscribed to the Google Groups "Synnefo" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to synnefo+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Martin Njau]

Hi Filippos,

Thanks for the quick assist

This worked like charm

Highly appreciate your assistance.

Regards,

Martin

[Brian Candler]

Incidentally, I've seen a very similar error:

 File "/usr/lib/python2.7/dist-packages/pithos/workers/glue.py", line 82, in _pool_create
    self.cnt)
  File "/usr/lib/python2.7/dist-packages/archipelago/common.py", line 894, in __init__
    ctx = segment.join()
  File "/usr/lib/python2.7/dist-packages/archipelago/common.py", line 559, in join
    cast(0, cb_null_ptrtype))
  File "/usr/lib/python2.7/dist-packages/archipelago/common.py", line 551, in errcheck
    % (config['SEGMENT_NAME'], os.strerror(e)))

Error: Cannot join segment 'archipelago': Invalid argument

In this case the cause turned out to be very simple: archipelago wasn't running. "/etc/init.d/archipelago start" was sufficient to fix it. (The node had been rebooted but archipelago hadn't been set to auto start)

It would be nice if a more helpful error message could be given in this case.