jclouds beginner- can i put a file within a folder (which is within a container)

[Arvind IK Chari]

Hello,

I am using blobstore to store a blob.

Now I am able to store the blob within a container, correctly.

What I want to do is, create a folder within the container, and then store a blob within that folder.

Is this possible? With ref to http://javadocs.jclouds.cloudbees.net/org/jclouds/blobstore/BlobStore.html it refers to "putBlob" but there only a container and the blob are specified as parameters, viz. that can be used to store a blob directly at root level within a container.

Regards,

Arvind.

[Vijay Kiran]

Arvind,

Would you please consider *not* creating a new thread for every question - because your questions seem to be related.

In the link you provided you can see that there's method called: createContainerInLocation or createDirectory which lets you create a new container or Folder. 

Please do read documentation here http://www.jclouds.org/documentation/userguide/blobstore-guide 

To create a blob in a folder you can use BlobStoreUtils#newBlob(blobstore, path) - again see the documentation in the above link under "Blobstore (Synchronous)"

HTH,

./Vijay

[Arvind IK Chari]

Hello Vijay,

Ok. sorry... I wont create new threads for every doubt of mine...

Regarding current issue, I understand how to create a container(createContainerInLocation), and how to create a folder within a container("createDirectory")-- but how do I store a file within a folder. 

I would appreciate it if you could give me one example of storing a blob within a folder.

Thanks,
Arvind.

 

[Vijay Kiran]

I'm not sure if you are using Clojure or Java .. but here's clojure code on how to do it with aws-s3:

This will create a new folder called test in the specified container and will create a file called hello in the test folder.

  (let [username "ACCESS_KEY"

        password "SECRET_KEY"

        container "SOME_COTNAINER_NAME"

        bstore (blobstore/blobstore "aws-s3" username password)]

    (blobstore/create-directory bstore container "test/")

    (blobstore/put-blob bstore container (blobstore/blob "test/hello" :payload "testing")))

[Andrew Phillips]

> I would appreciate it if you could give me one example of storing a blob
> within a folder.

Try

https://github.com/jclouds/jclouds-cloud-storage-workshop/blob/master/exercise4/src/main/java/org/jclouds/labs/blobstore/exercise4/MyDropboxClient.java

Hope that helps!

ap

[Arvind IK Chari]

Hello Andrew,

Thank you...the code you pointed me to works perfectly...

I do have one more request for you- how do I specify a folder from which a blob has to be retrieved? Because I cant see how to use blobbuilder with "getBlob". Basically, I want to retrieve a blob stored within specified folder in a container.

Regards,

Arvind.

[Arvind IK Chari]

Hello Andrew,

I have an update for you-- I ran the command given below (to upload file to cloud storage) after looking at the example you had pointed me to--

 public void uploadFile(String directory, File file) throws IOException {

    System.out.println("Uploading file to directory " +  directory);

         {

            bscontext.getBlobStore().putBlob(CONTAINER_NAME, bscontext.getBlobStore().blobBuilder(

                    format("%s/%s", directory, file.getName())).payload(file).build());

             

        }

    }

However, I am getting the following error (repeatedly)-- 

Jun 20, 2012 1:32:32 PM org.jclouds.logging.jdk.JDKLogger logWarn

WARNING: Cannot retry after server error, command has exceeded retry limit 5: [method=CloudFilesAsyncClient.putObject, request=PUT https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_83ff3b65-1f2e-49cc-84f4-49ef35658418/sparkcrawler//crawler HTTP/1.1]

Exception in thread "main" org.jclouds.http.HttpResponseException: java.io.FileNotFoundException: e:\crawler (Access is denied) connecting to PUT https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_83ff3b65-1f2e-49cc-84f4-49ef35658418/sparkcrawler//crawler HTTP/1.1

at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:173)

at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:130)

If you want I can provide the entire error message. I  checked that the file which has to be uploaded, is there in specified path, and the folder within which I want to insert data (into my Rackspace CloudFiles US account) is also there...I also tried inserting file directly into the top level container, even then I am getting the same error.

What am I doing wrong here? 

Thank you for your help,

Sincerely,

Arvind.

[Andrew Phillips]

The "/" approach you are using should indeed allow you to specify the
folder. As to the error you are getting, it looks like this

java.io.FileNotFoundException: e:\crawler (Access is denied)
connecting to PUT

might be the problem: your storage URL is being mapped to the *root*
of the file system, which is probably due to the "//" in

https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_83ff3b65-1f2e-49cc-84f4-49ef35658418/sparkcrawler//crawler

Could you try

https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_83ff3b65-1f2e-49cc-84f4-49ef35658418/sparkcrawler/crawler

(note the "/" rather than "//" between "sparkcrawler" and "crawler")..?

Thanks

ap

[Arvind IK Chari]

Hi Andrew

I checked the code again- what you said is correct- but also, the main reason for the error was that the file name was being specified incorrectly -the "Crawler" that you saw is the name of folder from where the file is to be uploaded to cloud storage- I used a different variable for the file's name, and instead of file name, folder name was being passed as parameter .

Thank you very much for your help.

Also, do let me know if you can help me with ref to:- downloading a file from specific folder in cloud storage container/ recursive listing / listing only files/listing only folders-- some more usage examples would be nice :)

Regards,

Arvind.

[Andrew Phillips]

> Also, do let me know if you can help me with ref to:- downloading a file
> from specific folder in cloud storage container/ recursive listing /
> listing only files/listing only folders-- some more usage examples would be
> nice :)

Downloading a file from a specific folder should simply require you to
construct the correctly blob location (something like
"folder/subfolder/blob-name").

As regards listing only files or folders, that is a little tricky
because many of the blobstores you are talking to do not actually
*have* a concept of folders. They store blobs in a flat namespace, so
what appears in jclouds as a "folder" is just another blob. jclouds
uses the location to effectively simulate the appearance of files in a
folder structure.

Which blobstores were you planning to use? Is maintaining portability
of your code across different blobstores a priority?

Regards

ap

[Arvind IK Chari]

Hello,

(1) For uploading file to cloud storage, we use options for "blobbuilder" (something like format("%s/%s), directory,filename)

How do we use this for getblob? Am i correct in thinking that blobbuilder can be used for only "putblob"?

(2) Regarding recursive listing/listing of only files or folders- I am working on a web crawler that can extract and analyse data from any publicly available website- the result can be stored on a cloud storage provider as specified by the end user. Hence I would want to be able to use same code across as many cloud storage providers, as is possible

Sincerely,

Arvind.

 

[Andrew Phillips]

> (1) For uploading file to cloud storage, we use options for "blobbuilder"
> (something like format("%s/%s), directory,filename)
> How do we use this for getblob? Am i correct in thinking that blobbuilder
> can be used for only "putblob"?

Personally, I have always consider BlobBuilder as a helper for
*building* (i.e. making) blobs. For retrieving blobs I simply use one
of the BlobStore.getBlob [1, 2] variants.

> (2) Regarding recursive listing/listing of only files or folders- I am
> working on a web crawler that can extract and analyse data from any
> publicly available website- the result can be stored on a cloud storage
> provider as specified by the end user. Hence I would want to be able to use
> same code across as many cloud storage providers, as is possible

First off, you should be able to use the ListContainerOptions [3] in
conjunction with BlobStore.list [4] to find all contents within a
certain directory.

One thing you could do is use a tag or other piece of user metadata to
distinguish "folder" and "non-folder" blobs. Then you could filter the
result output to show only folders or files, and you would only need
to query the metadata to do this.

In fact, from what I recall, the "tag as folder" trick is what jclouds
does internally for some providers to simulate folders. But if you
choose your own tagging scheme you should have something that works
across providers.

Hope this helps!

ap

[1]
http://javadocs.jclouds.cloudbees.net/org/jclouds/blobstore/AsyncBlobStore.html#getBlob(java.lang.String,
java.lang.String)
[2]
http://javadocs.jclouds.cloudbees.net/org/jclouds/blobstore/AsyncBlobStore.html#getBlob(java.lang.String, java.lang.String,
org.jclouds.blobstore.options.GetOptions)
[3]
http://javadocs.jclouds.cloudbees.net/org/jclouds/blobstore/options/ListContainerOptions.html
[4]
http://javadocs.jclouds.cloudbees.net/org/jclouds/blobstore/AsyncBlobStore.html#list(java.lang.String,
org.jclouds.blobstore.options.ListContainerOptions)