configuring for Swift?

[Randy Rue]

Hi Again All, more newbie questions!

For testing I'm trying to configure for our SwiftStack cluster on the
back end, first using the builtin pytest.

Having trouble finding the right syntax for ~/.s3ql/.authinfo2.

We're using v1 authentication.

Should the storage URL in the config file be just the cluster name and
container i.e.
swift://cluster.domain.com/container/prefix

or the swift "auth" URL i.e.
swift://cluster.domain.com/auth/v1.0/container/prefix

?

I've tried both with no luck, I get to:
tests/t4_fuse.py::TestFuse::test FAILED

and a skip for t1_backends.py:
SKIP [1] t1_backends.py:362: got empty parameter set ['backend']

that I'm guessing means it couldn't connect to a backend with those auth
settings?

[Nikolaus Rath]

On Nov 15 2016, Randy Rue <rand...@gmail.com> wrote:
> Hi Again All, more newbie questions!
>
> For testing I'm trying to configure for our SwiftStack cluster on the
> back end, first using the builtin pytest.

I thought you wanted to use S3QL with Amazon S3?

>
> Having trouble finding the right syntax for ~/.s3ql/.authinfo2.
>
> We're using v1 authentication.
>
> Should the storage URL in the config file be just the cluster name and
> container i.e.
> swift://cluster.domain.com/container/prefix
>
> or the swift "auth" URL i.e.
> swift://cluster.domain.com/auth/v1.0/container/prefix

The first one.

> I've tried both with no luck, I get to:
> tests/t4_fuse.py::TestFuse::test FAILED
>
> and a skip for t1_backends.py:
> SKIP [1] t1_backends.py:362: got empty parameter set ['backend']
>
> that I'm guessing means it couldn't connect to a backend with those
> auth settings?

No, it means that you haven't told the test runner that it should
connect to this container. It would be a little inconvenient if it
picked a random service from your authinfo2 file and filled it with test
data, right?

For testing, add an entry like this:

[swift-test]
test-fs: swift://cluster.domain.com/container/prefix_for_testing
backend-login: bla
backend-password: foo

For regular usage (or non-automated testing), use something like:

[my-swift-account]
storage-url: swift://cluster.domain.com/
backend-login: bla
backend-password: foo


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Randy Rue]

OK, I've done some RTFM'ing and also cruised this group back to about 2010 and have made some progress but am still flailing.

Bear with me, especially in the format of this post, I've see that you prefer a specific style of replying but can't find the link you posted on the style guide right now. Is there a way to put that information here in any google-group-equivalent form of a "sticky" post?

To answer your question about our backend choices, we're considering this for use against both S3 and/or swift (SwiftStack). When I get our swift backend working I'm happy to post a cleaned working example, from searching here I see a lot of folks describing what doesn't work but not so much about their solutions.

Hi Nikolaus,

Should I be working on getting the pytest to complete without errors? If I can't make it finish with a test backend configured, does that indicate problems that would stop me in real use or is it testing for things I don't necessarily need to worry about?

Should I be concerned that pytest often fails at different points, even with no config changes between runs? For example, it often crumps with an error:

FileNotFoundError: [Errno 2] No such file or directory: 'mock_server.log'

And back to the actual config issues:

My current .authinfo2:

[swift-test]

test-fs: swift://cluster.domain.com/container/test_prefix

backend-login: bar

backend-password: foo

[cluster]

storage-url: swift://cluster.domain.com/

backend-login: bar

backend-password: foo

questions:

* does the "[cluster]" line need to match anything or is it just a marker for that entry? Or when the app looks for authinfo, is it looking for the storage-url and finding the credentials that match?

* Can I also put a prefix in the "prod" storage-url? Your docs suggest so but you don't show it in your example.

pytest now (usually) fails at the t4_fuse.py test, complete output:

tests/t4_fuse.py::TestFuse::test FAILED

====================================================================== short test summary info ======================================================================

SKIP [1] t1_backends.py:362: got empty parameter set ['backend'], function test_issue114 at /root/s3ql-2.21/tests/t1_backends.py:362

============================================================================= FAILURES ==============================================================================

___________________________________________________________________________ TestFuse.test ___________________________________________________________________________

Traceback (most recent call last):

  File "/root/s3ql-2.21/tests/t4_fuse.py", line 166, in test

    self.mount()

  File "/root/s3ql-2.21/tests/t4_fuse.py", line 101, in mount

    retry(30, poll)

  File "/root/s3ql-2.21/tests/common.py", line 73, in retry

    ret = fn(*a, **kw)

  File "/root/s3ql-2.21/tests/t4_fuse.py", line 100, in poll

    assert self.mount_process.poll() is None

AssertionError: assert 1 is None

 +  where 1 = <bound method Popen.poll of <subprocess.Popen object at 0x7f20bed01ac8>>()

 +    where <bound method Popen.poll of <subprocess.Popen object at 0x7f20bed01ac8>> = <subprocess.Popen object at 0x7f20bed01ac8>.poll

 +      where <subprocess.Popen object at 0x7f20bed01ac8> = <t4_fuse.TestFuse object at 0x7f20becf5908>.mount_process

----------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------

Please store the following master key in a safe location. It allows

decryption of the S3QL file system in case the storage objects holding

this information get corrupted:

---BEGIN MASTER KEY---

lm7D NIOV KFeM WCua z8OB ZygQ l2hV Ndz8 09Vo DYn7 p94=

---END MASTER KEY---

----------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------

WARNING: Maximum object sizes less than 1 MiB will degrade performance.

ERROR: Uncaught top-level exception:

Traceback (most recent call last):

  File "/root/s3ql-2.21/bin/mount.s3ql", line 26, in <module>

    s3ql.mount.main(sys.argv[1:])

  File "/root/s3ql-2.21/src/s3ql/mount.py", line 204, in main

    sd_notify('READY=1')

  File "systemd/daemon.pyx", line 41, in systemd.daemon.notify (systemd/daemon.c:41)

TypeError: state must be an instance of Notigication

====================================================================== pytest-warning summary =======================================================================

WC1 /root/s3ql-2.21/tests/t2_block_cache.py cannot collect test class 'TestBackendPool' because it has a __init__ constructor

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

================================================ 1 failed, 245 passed, 1 skipped, 1 pytest-warnings in 45.48 seconds ================================================

[root@foo s3ql-2.21]#

Thanks again for your help with this!

Randy

[Nikolaus Rath]

On Nov 16 2016, Randy Rue <rand...@gmail.com> wrote:
> Bear with me, especially in the format of this post, I've see that you
> prefer a specific style of replying but can't find the link you posted on
> the style guide right now. Is there a way to put that information here in
> any google-group-equivalent form of a "sticky" post?

https://www.google.com/search?q=how+to+quote+in+email, first hit.

[Nikolaus Rath]

On Nov 16 2016, Randy Rue <rand...@gmail.com> wrote:

> Should I be working on getting the pytest to complete without errors?

Probably. But so far you haven't actually reported any errors (just
skipped tests, which are fine), so I'm not sure why the question is
relevant.

> If I can't make it finish with a test backend configured, does that
> indicate problems that would stop me in real use or is it testing for
> things I don't necessarily need to worry about?

Depends on what you mean with "can't make it finish" and what error
message (if any) you get.

> Should I be concerned that pytest often fails at different points,

Yes, very much so. But to get help, you'd need to provide the full error
messages.

> And back to the actual config issues:
>
> My current .authinfo2:
> [swift-test]
> test-fs: swift://cluster.domain.com/container/test_prefix
> backend-login: bar
> backend-password: foo
>
> [cluster]
> storage-url: swift://cluster.domain.com/
> backend-login: bar
> backend-password: foo
>
> questions:
> * does the "[cluster]" line need to match anything or is it just a marker
> for that entry?

That's just a label. From http://www.rath.org/s3ql-docs/authinfo.html:

| The authentication file consists of sections, led by a [section]
| header and followed by name: value entries. The section headers
| themselves are not used by S3QL but have to be unique within the file.

> Or when the app looks for authinfo, is it looking for the
> storage-url and finding the credentials that match?

Yes. From the same link:

| storage-url: Specifies the storage url to which this section
| applies. If a storage url starts with the value of this entry, the
| section is considered applicable.

> * Can I also put a prefix in the "prod" storage-url? Your docs suggest so
> but you don't show it in your example.

Yes. The docs are authoritative.

> -----------------------------------------------------------------------
> Captured stderr call
> ------------------------------------------------------------------------
> WARNING: Maximum object sizes less than 1 MiB will degrade performance.
> ERROR: Uncaught top-level exception:
> Traceback (most recent call last):
> File "/root/s3ql-2.21/bin/mount.s3ql", line 26, in <module>
> s3ql.mount.main(sys.argv[1:])
> File "/root/s3ql-2.21/src/s3ql/mount.py", line 204, in main
> sd_notify('READY=1')
> File "systemd/daemon.pyx", line 41, in systemd.daemon.notify
> (systemd/daemon.c:41)
> TypeError: state must be an instance of Notigication

Hmm. Apparently systemd changed the calling convention in a backwards
incompatible way. I suspect this may be a bug in systemd, but I'll have
to check. As a workaround, replace

try:
from systemd.daemon import notify as sd_notify
except ImportError:
sd_notify = None

with just

sd_notify = None

at the beginning of src/s3ql/mount.py

[Nikolaus Rath]

On Nov 16 2016, Nikolaus Rath <Niko...@rath.org> wrote:
>> -----------------------------------------------------------------------
>> Captured stderr call
>> ------------------------------------------------------------------------
>> WARNING: Maximum object sizes less than 1 MiB will degrade performance.
>> ERROR: Uncaught top-level exception:
>> Traceback (most recent call last):
>> File "/root/s3ql-2.21/bin/mount.s3ql", line 26, in <module>
>> s3ql.mount.main(sys.argv[1:])
>> File "/root/s3ql-2.21/src/s3ql/mount.py", line 204, in main
>> sd_notify('READY=1')
>> File "systemd/daemon.pyx", line 41, in systemd.daemon.notify
>> (systemd/daemon.c:41)
>> TypeError: state must be an instance of Notigication
>
> Hmm. Apparently systemd changed the calling convention in a backwards
> incompatible way. I suspect this may be a bug in systemd, but I'll have
> to check.

Actually, I think you've got some third-party python-systemd module
installed. You should be using the one that comes with systemd (from
https://github.com/systemd/python-systemd/).

It looks like you have https://github.com/mosquito/python-systemd, which
is incompatible with the "real" systemd module. The fact that the author
chose to give it the same name is not a good sign.

[Randy Rue]

Working config steps for a SwiftStack backend (so far)

* Create a new user "foo" (pw "bar") on the cluster with undelete off and auto-sharding on

* Create a container "container"

* ~/.s3ql/.authinfo2:

-------

[swift-test]

test-fs: swift://cluster.domain.com/container/test_prefix

backend-login: foo

backend-password: bar

[cluster]

storage-url: swift://cluster.domain.com/container/

backend-login: foo

backend-password: bar

----

* note the trailing slash on the storage-url, don't know if it's necessary but whatever you enter for any call using this auth file must match or you'll be asked for credentials

* "python3 -m pytest -rs /root/s3ql-2.21/tests/"  (or wherever you built it) now completes largely without trouble (occasional errors "No such file or directory: 'mock_server.log'")

mkdir /s3ql

mkfs.s3ql --authfile /root/.s3ql/.authinfo2 -L S3QL swift://cluster.domain.com/container/    (TRAILING SLASH)

mount.s3ql --authfile /root/.s3ql/.authinfo2 --nfs swift://cluster.domain.com/container/ /s3ql    (TRAILING SLASH)

Life is good!

Randy

[Randy Rue]

Interesting. This is a brand-new CentOS_7 pretty minimal install, the
only things I've installed since then are the packages/etc listed at
https://bitbucket.org/nikratio/s3ql/wiki/installation_centos. I even had
to "yum install fuse" as that wasn't included in the OS install and your
wiki page lists fuse-devel but not fuse

[Randy Rue]

Excellent. I've made it from RTFM to LMGTFY. I'm making good progress!
And learned that in Thunderbird if I select the text I want to quote and
hit Reply, it automagically includes only that text in the reply,
formatted as you see it below.
Life is good!

Randy

On 11/16/2016 9:07 AM, Nikolaus Rath wrote:
> https://www.google.com/search?q=how+to+quote+in+email, first hit.

[Randy Rue]

A little more digging shows that the "mosquito" version is actually a wrapper that seems to try to improve error handling but requires some different calls to sd_notify, details at https://github.com/mosquito/python-systemd

So at this point it appears my choices are to modify your mount.py to skip sd_notify, or to modify it to use the modified sd_notify?

[Nikolaus Rath]

On Nov 18 2016, Randy Rue <rand...@gmail.com> wrote:
> A little more digging shows that the "mosquito" version is actually a
> wrapper that seems to try to improve error handling but requires some
> different calls to sd_notify, details
> at https://github.com/mosquito/python-systemd

Yes. That is a terrible idea, and I would not run any software written
by someone who does not recognize this.

> So at this point it appears my choices are to modify your mount.py to skip
> sd_notify, or to modify it to use the modified sd_notify?

Your only reasonable course of action is to use the systemd python
module that comes with systemd rather than some obviously braindead
GitHub code.

[Randy Rue]

Looks like at some point pip installed "system" when what I needed was "system-python." Looking at my bash history on the box, the only call to pip that was possibly vague was:

pip3.4 install defusedxml requests dugong llfuse pytest pytest-catchlog git+https://github.com/systemd/python-systemd.git#egg=system

What's the git+https call at the end do?

Anyway. Now if I open a python3 session and import sd_notify and call it with 'READY=1' passed I get a "False" return instead of a misspelled error.

Life is good.

" Your only reasonable course of action is to use the systemd python module that comes with systemd rather than some obviously braindead GitHub code."

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus