configure s3ql with swift

[amit kertis]

hi all

i am trying to configure s3ql to work with swift\

i install s3ql on my ubuntu 16by ussing the apt get (sudo apt-get install -y s3ql) 

so i have S3QL 2.15

i set the authinfo2 as follow

ccloud@amit-kertes-s3ql:~$ cat .s3ql/authinfo2

[swift]

backend-login: xxx

backend-password: xxx

storage-url: swiftks://objectstore-3.eu-nl-1.cloud.sap/eu-nl-1:amits3ql

i run this command 

mkfs.s3ql --debug  swiftks://objectstore-3.eu-nl-1.cloud.sap:443/eu-nl-1:amits3ql

i am getting this prompt 

Enter backend login:

Enter backend passphrase:

what i need to put on those ??

i try user & pass and its not working 

any thing i try even just enter i getting this error

2019-12-09 15:38:30.894 14560 MainThread s3ql.backends.common.get_ssl_context: Reading default CA certificates.

2019-12-09 15:38:30.895 14560 MainThread s3ql.backends.swift._do_request: started with 'GET', '/', None, {'limit': 1}, None, None

2019-12-09 15:38:30.895 14560 MainThread s3ql.backends.swift._do_request: no active connection, calling _get_conn()

2019-12-09 15:38:30.896 14560 MainThread s3ql.backends.swiftks._get_conn: started

2019-12-09 15:38:30.909 14560 MainThread root.excepthook: Uncaught top-level exception:

Traceback (most recent call last):

  File "/usr/bin/mkfs.s3ql", line 9, in <module>

    load_entry_point('s3ql==2.15', 'console_scripts', 'mkfs.s3ql')()

  File "/usr/lib/s3ql/s3ql/mkfs.py", line 96, in main

    plain_backend = get_backend(options, raw=True)

  File "/usr/lib/s3ql/s3ql/common.py", line 260, in get_backend

    getattr(options, 'compress', ('lzma', 2)), raw)()

  File "/usr/lib/s3ql/s3ql/common.py", line 335, in get_backend_factory

    backend_options)

  File "/usr/lib/s3ql/s3ql/backends/swiftks.py", line 26, in __init__

    super().__init__(storage_url, login, password, options)

  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 68, in __init__

    self._container_exists()

  File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped

    return method(*a, **kw)

  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 83, in _container_exists

    self._do_request('GET', '/', query_string={'limit': 1 })

  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 210, in _do_request

    self.conn =  self._get_conn()

  File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped

    return method(*a, **kw)

  File "/usr/lib/s3ql/s3ql/backends/swiftks.py", line 98, in _get_conn

    raise HTTPError(resp.status, resp.reason, resp.headers)

s3ql.backends.s3c.HTTPError: 400 Bad Request

please help 

i need to find solution ASAP

what i am doing wrong ???

[amit kertis]

i try to use ubunto 18 and install s3ql 2.26

and i get the same probelm 

ccloud@amitkertes-s3ql-5:~$ mkfs.s3ql --version

S3QL 2.26

ccloud@amitkertes-s3ql-5:~$ mkfs.s3ql --authfile ~/.s3ql/authinfo2  swiftks://objectstore-3.eu-nl-1.cloud.sap/eu-nl-1:amits3ql

ERROR: Uncaught top-level exception:

Traceback (most recent call last):

  File "/usr/bin/mkfs.s3ql", line 11, in <module>

    load_entry_point('s3ql==2.26', 'console_scripts', 'mkfs.s3ql')()

  File "/usr/lib/s3ql/s3ql/mkfs.py", line 95, in main

    plain_backend = get_backend(options, raw=True)

  File "/usr/lib/s3ql/s3ql/common.py", line 260, in get_backend

    getattr(options, 'compress', ('lzma', 2)), raw)()

  File "/usr/lib/s3ql/s3ql/common.py", line 335, in get_backend_factory

    backend_options)

  File "/usr/lib/s3ql/s3ql/backends/swiftks.py", line 26, in __init__

    super().__init__(storage_url, login, password, options)

  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 72, in __init__

    self._container_exists()

  File "/usr/lib/s3ql/s3ql/backends/common.py", line 108, in wrapped

    return method(*a, **kw)

  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 87, in _container_exists

    self._do_request('GET', '/', query_string={'limit': 1 })

  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 231, in _do_request

    self.conn =  self._get_conn()

  File "/usr/lib/s3ql/s3ql/backends/common.py", line 108, in wrapped

    return method(*a, **kw)

  File "/usr/lib/s3ql/s3ql/backends/swiftks.py", line 98, in _get_conn

    raise HTTPError(resp.status, resp.reason, resp.headers)

s3ql.backends.s3c.HTTPError: 400 Bad Request

please help

[amit kertis]

i know why i get this 

Enter backend login:

Enter backend passphrase:

what i need to put on those ??

the object url wasnt the same in the command and the autofile 

but i still getting this error 

can someone tell me why and what it say 

and its will be great of how i can solve it :)

On Monday, December 9, 2019 at 5:44:24 PM UTC+2, amit kertis wrote:

[amit kertis]

i know why i get this 

Enter backend login:

Enter backend passphrase:

the object url wasn't the same in the command and the autofile 

but i still getting this error 

can someone tell me why and what it say 

and its will be great of how i can solve it :

On Monday, December 9, 2019 at 5:44:24 PM UTC+2, amit kertis wrote:

[amit kertis]

On Monday, December 9, 2019 at 5:44:24 PM UTC+2, amit kertis wrote:

[amit kertis]

hi ALL

i still dont have any solution for this problem 

i still getting the same error and i didnt find any solution or a direction for this issue

i think its something related to user&pass

but i dont know

it will be great if someone can give me a hint for this issue 

i am using version 2.26 on ubuntu 18 

and this is the output i get one i run the mkfs command 

mkfs.s3ql --authfile .s3ql/authinfo2 --debug --plain swiftks://objectstore-3.eu-nl-1.cloud.sap:443/eu-nl-1:amitkertis

2019-12-15 08:27:49.101 20244 DEBUG    MainThread s3ql.backends.common.get_ssl_context: Reading default CA certificates.

2019-12-15 08:27:49.101 20244 DEBUG    MainThread s3ql.backends.swift._do_request: started with 'GET', '/', None, {'limit': 1}, None, None

2019-12-15 08:27:49.101 20244 DEBUG    MainThread s3ql.backends.swift._do_request: no active connection, calling _get_conn()

2019-12-15 08:27:49.102 20244 DEBUG    MainThread s3ql.backends.swiftks._get_conn: started

2019-12-15 08:27:49.115 20244 ERROR    MainThread root.excepthook: Uncaught top-level exception:

Traceback (most recent call last):

  File "/usr/bin/mkfs.s3ql", line 11, in <module>

    load_entry_point('s3ql==2.26', 'console_scripts', 'mkfs.s3ql')()

  File "/usr/lib/s3ql/s3ql/mkfs.py", line 95, in main

    plain_backend = get_backend(options, raw=True)

  File "/usr/lib/s3ql/s3ql/common.py", line 260, in get_backend

    getattr(options, 'compress', ('lzma', 2)), raw)()

  File "/usr/lib/s3ql/s3ql/common.py", line 335, in get_backend_factory

    backend_options)

  File "/usr/lib/s3ql/s3ql/backends/swiftks.py", line 26, in __init__

    super().__init__(storage_url, login, password, options)

  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 72, in __init__

    self._container_exists()

  File "/usr/lib/s3ql/s3ql/backends/common.py", line 108, in wrapped

    return method(*a, **kw)

  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 87, in _container_exists

    self._do_request('GET', '/', query_string={'limit': 1 })

  File "/usr/lib/s3ql/s3ql/backends/swift.py", line 231, in _do_request

    self.conn =  self._get_conn()

  File "/usr/lib/s3ql/s3ql/backends/common.py", line 108, in wrapped

    return method(*a, **kw)

  File "/usr/lib/s3ql/s3ql/backends/swiftks.py", line 98, in _get_conn

    raise HTTPError(resp.status, resp.reason, resp.headers)

s3ql.backends.s3c.HTTPError: 400 Bad Request

please advised

Amit

On Monday, December 9, 2019 at 5:44:24 PM UTC+2, amit kertis wrote:

[Shannon Dealy]

Hi Amit,

I haven't used the swift backend, but the one thing I do notice is that your
storage-url from the authinfo2 file does not match the one specified
on the command line:

storage-url:
swiftks://objectstore-3.eu-nl-1.cloud.sap/eu-nl-1:amits3ql
vs
swiftks://objectstore-3.eu-nl-1.cloud.sap:443/eu-nl-1:amitkertis

specifically, the authinfo2 storage-url does not include the port number and
suffix at the ends don't match: amits3ql vs amitkertis

FWIW

Best regards,

Shannon C. Dealy | DeaTech Research Inc.
de...@deatech.com | Biotechnology Development Services
Telephone USA: +1 541-929-4089 | USA and the Netherlands
Netherlands: +31 85 208 5570 | www.deatech.com

On Sun, 15 Dec 2019, amit kertis wrote:

> hi ALL
>
> i still dont have any solution for this problem
> i still getting the same error and i didnt find any solution or a direction
> for this issue
>
> i think its something related to user&pass
> but i dont know
>
> it will be great if someone can give me a hint for this issue
>

> *i am using version 2.26 on ubuntu 18 *

>
> and this is the output i get one i run the mkfs command
>
>

> *mkfs.s3ql --authfile .s3ql/authinfo2 --debug --plain
> swiftks://objectstore-3.eu-nl-1.cloud.sap:443/eu-nl-1:amitkertis*

> *please advised*
>
> *Amit*

>
>
>
>
>
>
> On Monday, December 9, 2019 at 5:44:24 PM UTC+2, amit kertis wrote:
>>
>> hi all
>>
>> i am trying to configure s3ql to work with swift\
>>
>> i install s3ql on my ubuntu 16by ussing the apt get (sudo apt-get install
>> -y s3ql)
>>
>> so i have S3QL 2.15
>>
>> i set the authinfo2 as follow
>>
>> ccloud@amit-kertes-s3ql:~$ cat .s3ql/authinfo2
>> [swift]
>> backend-login: xxx
>> backend-password: xxx
>> storage-url: swiftks://objectstore-3.eu-nl-1.cloud.sap/eu-nl-1:amits3ql
>>
>>
>> i run this command
>>
>> mkfs.s3ql --debug
>> swiftks://objectstore-3.eu-nl-1.cloud.sap:443/eu-nl-1:amits3ql
>>
>> i am getting this prompt
>>
>> Enter backend login:
>> Enter backend passphrase:
>>

>> *what i need to put on those ??*

>> *please help *
>> *i need to find solution ASAP*
>>
>> *what i am doing wrong ???*
>>
>>
>>
>>
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups "s3ql" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+uns...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/3ccb7d49-6ea9-45c6-b71c-9a9fdac470a9%40googlegroups.com.
>

[amit kertis]

i try both swift container one i put it public but with no good result

so even when the url are identical i am getting the same error 

thanks for the response :)

Amit

On Monday, December 9, 2019 at 5:44:24 PM UTC+2, amit kertis wrote:

[Daniel Jagszent]

Hello Amit,

[swift]

backend-login: xxx

backend-password: xxx

storage-url: swiftks://objectstore-3.eu-nl-1.cloud.sap/eu-nl-1:amits3ql

what version of OpenStack Keystone (authentication mechanism) do you use? The URL you are using tries to use Keystone v2 (switfks:// prefix) but the host name (objectstore-3.eu-nl-1.cloud.sap) looks like the actual host name where your container lives. You must provide the Keystone server here.
That's normally something like auth.cloud.sap.

What credentials did you get to log into your Swift Container? Did you get user name, tenant name and password? Then this is Keystone v2 – the only version you can use, since you are using a very outdated version of S3QL.
https://www.rath.org/s3ql-docs/backends.html#openstack-swift tells you how to combine tenant and user into backend-login.

[...]  When using Keystone authentication, the backend password is your regular OpenStack password and the backend login combines you OpenStack username and tenant name in the form <tenant>:<user>. If no tenant is required, the OpenStack username alone may be used as backend login. [...]

If you got user name, project id, domain id and password, then you need to use Keystone v3 – and for this you need a newer version of S3QL: https://www.rath.org/s3ql-docs/installation.html

[amit kertis]

thank you @Daniel Jagszent sorry for my late response but i was OOO

regarding the version i am using 2.6 i try to install the new version of s3fs  but i got into a block end 

with all the dependency that is needed ,

its will be great if u can send me a details user guide of how to instal the new version with all the dependency 

step by step command 

i use the apt-get so i got the 2.6 version 

i will try to look into your comment and see if the changes will workout 

but i think that install new version this is what i need to do so i will try once again also this 

On Monday, December 9, 2019 at 5:44:24 PM UTC+2, amit kertis wrote:

[amit kertis]

thank you @Daniel Jagszent sorry for my late response but i was OOO

regarding the version i am using 2.6 i try to install the new version of S3QL  but i got into a block end 

with all the dependency that is needed , every time i got different error when i install the dependency 

and in the end i was not able to install s3ql so i use the apt-get so igot the 2.6 version 

its will be great if u can send me a details user guide of how to instal the new version with all the dependency 

step by step command 

i will try to look into your comment and see if the changes will workout 

but i think that install new version this is what i need to do so i will try once again also this 

On Monday, December 9, 2019 at 5:44:24 PM UTC+2, amit kertis wrote:

[amit kertis]

ok i install version 3.2 but still have some issue 

i will open new ticket for this topic