LeoFS ListObjects error: Unable to connect to endpoint when using Amazon S3 CPP SDK

[gourish salgaonkar]

Hello,

I am using Amazon AWS CPPS3 SDK to connect to a LeoFS server and list the objects from a bucket.

I am able to list the bucket, however the code fails if I try to list the objects.

The error which I get is as follows

ListObjects error:  Unable to connect to endpoint

The following is the code which I have used. any parameter in the ClientConfiguration?

        Aws::SDKOptions options;

        options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Info;

        Aws::InitAPI(options);

        {

            Aws::Client::ClientConfiguration opts;

            opts.scheme = Aws::Http::Scheme::HTTP;

            opts.region = Aws::String("");

            opts.endpointOverride =  "13.70.0.324:8080";;

            opts.verifySSL = false;

            opts.connectTimeoutMs = 60000;

            opts.requestTimeoutMs = 10000;

            Aws::S3::S3Client s3_client(Aws::Auth::AWSCredentials(access_key_id, secret_access_key),

                opts);

            auto outcome = s3_client.ListBuckets();

            if (outcome.IsSuccess()) {

                Aws::String SourceImageBucket("sourceimages");

                Aws::String DestnImageBucket("test");

                std::cout << "Your S3 buckets:" << std::endl;

                Aws::Vector<Aws::S3::Model::Bucket> bucket_list =

                    outcome.GetResult().GetBuckets();

                Aws::Vector<Aws::S3::Model::Bucket>::const_iterator source_bucket;

                Aws::Vector<Aws::S3::Model::Bucket>::const_iterator destin_bucket;

                for(Aws::Vector<Aws::S3::Model::Bucket>::const_iterator it = bucket_list.begin();

                    it != bucket_list.end(); it++)

                {

                    auto bname =  it->GetName();

                    if (SourceImageBucket.compare(bname) == 0)

                    {

                        source_bucket = it;

                        std::cout << "Source Bucket : " <<

                            source_bucket->GetName() << std::endl;

                    }

                    if (DestnImageBucket.compare(bname) == 0)

                    {

                        destin_bucket = it;

                        std::cout << "Destination Bucket : " <<

                            destin_bucket->GetName() << std::endl;

                    }

                }

                Aws::S3::Model::ListObjectsRequest objects_request;

                objects_request.WithBucket(DestnImageBucket);

                // The following command fails

                auto list_objects_outcome = s3_client.ListObjects(objects_request);

                if (list_objects_outcome.IsSuccess()) {

                    Aws::Vector<Aws::S3::Model::Object> object_list =

                        list_objects_outcome.GetResult().GetContents();

                    for (auto const &s3_object : object_list) {

                        std::cout << "* " << s3_object.GetKey() << std::endl;

                    }

                }

                else {

                    std::cout << "ListObjects error: " <<

                        list_objects_outcome.GetError().GetExceptionName() << " " <<

                        list_objects_outcome.GetError().GetMessage() << std::endl;

                }

            }

            else {

                std::cout << "ListBuckets error: "

                    << outcome.GetError().GetExceptionName() << " - "

                    << outcome.GetError().GetMessage() << std::endl;

            }

        }

        Aws::ShutdownAPI(options);

[yoshiyuki kanno]

Hi,

Sorry for the long delay.

In your code,

> opts.endpointOverride = "13.70.0.324:8080"

This line could be problem because AWS CPP3 SDK has used Sub-Domain
style access when accessing objects under a bucket.
Please refer https://leo-project.github.io/leofs/faq/client/ for more detail.

That said,
Say you have a bucket named `foo` then
with your current code, SDK try to access `foo.13.70.0.324:8080` so
it's obviously unreachable.

Instead,
1. Give a name to the host running leo_gateway like `leo_gw` then SDK
get to access `foo.leo_gw:8080`.
2. Make those hosts (leo_gw, foo.leo_gw) resolvable from your machine
running a client through /etc/hosts or DNS.
(If you are going to use multiple buckets then registering a wildcard
domain `*.leo_gw` might make you less work in future)

I hope you find those helpful.
Also feel free to ask me if you face any problem.

Best,
Kanno.

> --
> You received this message because you are subscribed to the Google Groups
> "LeoProject.LeoFS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to leoproject_leo...@googlegroups.com.
> To post to this group, send email to leoproje...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/leoproject_leofs/b7e3fd86-2357-447d-a95e-7cc780284579%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Yoshiyuki Kanno
LeoFS Committer(http://www.leofs.org)
--------------------------------------------------
Stoic Corp.
URL: http://www.stoic.co.jp/
E-mail: yoshiyu...@stoic.co.jp

[gourish salgaonkar]

Hello Kanno san,

Sorry for my late reply.

I did the following changes to the code at the line of S3Client constructor, as follows.

 Aws::S3::S3Client s3_client(Aws::Auth::AWSCredentials(access_key_id, secret_access_key),

 opts, false, false);

Due to which AWS CPP3 SDK uses Path Style Access for accessing the object.

As in the constructor definition as follow

S3Client (const Auth::AWSCredentials &credentials, const Client::ClientConfiguration &clientConfiguration=Client::ClientConfiguration(), bool signPayloads=false, bool useVirtualAdressing=true)

The last parameter useVirtualAdressing = false indicates to use Path Style Access.

With this change the original cod which I have mentioned is working.  I am however not able to get the code working for  Sub-Domain Style Access  .  Is there any performance issue if Path Style Access is used.  I am also working on making the code work for Sub-Domain style access.

Thanks,

Gourish Salgaonkar

 

[yoshiyuki kanno]

Hi,

Great to hear that now you have the code worked with Path Style Access(PSA).

>Is there any performance issue if Path Style Access is used

Nothing so you don't need to be worried about using PSA.
However since there are some other clients that don't have PSA but
only Sub-Domain One, I'd recommend you to get the code work for
Sub-Domain one if you are going to access LeoFS through other clients.

Let me know if you face any trouble.

Best,
Kanno.

> --
> You received this message because you are subscribed to the Google Groups
> "LeoProject.LeoFS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to leoproject_leo...@googlegroups.com.
> To post to this group, send email to leoproje...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/leoproject_leofs/05ed59e5-ffa3-4fd5-a366-e7b00cf18166%40googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.



--
Yoshiyuki Kanno

LeoFS Committer(http://leo-project.net/leofs/index.html)