How to write to a blob using an output stream

[Kevin D]

I am trying to use jcloud to write files from my system into a cloud file storage.   In this case I am sending the file through a templating engine and then the result gets sent out to the blob.    I would prefer to get an output stream from the blob and just send my file and the output stream into the template engine and let it do the processing.   So far I have not seen a way to grab an output stream for the blob, it only allows for passing an input stream for the copy.   Any ideas on this?

There is another way to do it and that would be to write the results to a temporary file and then out, but I was trying to avoid this.   

[Adrian Cole]

here's one way. note that it doesn't work on all providers:

      PayloadBlobBuilder blobBuilder = view.getBlobStore().blobBuilder("streaming").payload(

               new StreamingPayload(new WriteTo() {

                  @Override

                  public void writeTo(OutputStream outstream) throws IOException {

                     outstream.write("foo".getBytes());

                  }

               }));

On Fri, Mar 8, 2013 at 4:06 PM, Kevin D <kevi...@yahoo.com> wrote:

I am trying to use jcloud to write files from my system into a cloud file storage.   In this case I am sending the file through a templating engine and then the result gets sent out to the blob.    I would prefer to get an output stream from the blob and just send my file and the output stream into the template engine and let it do the processing.   So far I have not seen a way to grab an output stream for the blob, it only allows for passing an input stream for the copy.   Any ideas on this?

There is another way to do it and that would be to write the results to a temporary file and then out, but I was trying to avoid this.   

--
You received this message because you are subscribed to the Google Groups "jclouds" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jclouds+u...@googlegroups.com.
To post to this group, send email to jcl...@googlegroups.com.
Visit this group at http://groups.google.com/group/jclouds?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Kevin D]

This only gives me extremely limited access to the output stream.   I was hoping to get an OutputStream object that I could pass into another method so that method write to the output stream.

Thanks though.

[Andrew Phillips]

Hi Kevin

Maybe just taking a small step back...when do you want/need to result
of the templating process to be written to the blob? In a streaming
fashion, i.e. whenever the templating engine decides to output some
more bytes? Or can the result be buffered (assuming the sizes are
manageable and bounded) before the entire result is written to the blob?

In the former case, I think you would indeed need to use a blobstore
that supports streaming. As regards more control over the output
stream: do you also want/need to control *when* the templating engine
is called? Or is it acceptable if it's called "just in time" from
WriteTo.writeTo (nice candidate for a lambda there ;-))?

As far as I understand, it is indeed the case that you can't get a
"direct" output stream for a blob and simply "shove bytes" into there
as and when they arrive. You either need to assemble the payload in
advance, e.g. in a buffer (for non-streaming blobstores) or be ready
to *provide* the data when called (Adrian's WriteTo example).

With a bit of luck, someone will correct me and point out a third
option that also works ;-)

Regards

ap

[Kevin D]

I don't have a problem if the results are buffered assuming that the sizes are manageable.

Here is one option I thought I would throw out there.   Has anyone tried used PipedInput/Output streams along with Async blob storage to accomplish what I am asking for?   Tell me what you think of the following code.

final BlobStoreContext context = ContextBuilder.newBuilder("cloudfiles-us")

          .credentials(args[0], args[1])

          .buildView(BlobStoreContext.class);

 

java.io.PipedInputStream in = new java.io.PipedInputStream();

java.io.PipedOutputStream out = new java.io.PipedOutputStream( in );

          SwiftObject object = swift.getAsyncApi().newSwiftObject();

object.getInfo().setName("testasyc.java");

object.setPayload(in);

final ListenableFuture<String> futureETag = swift.getAsyncApi().putObject( CONTAINER_NAME, object );

Thread thread = new Thread( new Runnable() {

     public void run()

     {

           try {

                          // start the upload 

                futureETag.get();

                context.close();

           }

           catch ( Exception e ) {

              throw new RuntimeException();

           }

      }

      });

      thread.start();

                // write to the output stream 

      for ( int i = 0; i < 100; i++ ) {

         out.write( ("Test " + i).getBytes() );

      }

      out.flush();

      out.close();

                // wait until we are done writing 

      futureETag.get();

 

[Andrew Phillips]

I was going to suggest piped streams too but was a bit wary about the
potential threading issues. I haven't tried the kind of code you're
suggesting but it's certainly worth an experiment, I'd say.

I think you'd need a streaming payload, though: with this code, I
think there's a risk that jclouds will create an incomplete payload
based on the current state of "in" before you've had a chance to pipe
all the content to the output stream.

Note that there are two futureETag.get() statements in there, and it
would also be worth investigating what the visibility guarantees of
ListenableFuture are: are any changes to the future that happen due to
the call in the "get" thread guaranteed to be visible from the main
thread?

ap

PS: You also risk an unclosed context if the "get" in the thread blows
up - the "close" should probably be in a "finally" block. It probably
shouldn't be in there at all, actually, because you risk closing the
context before working with the future in the main thread later - not
sure if that's safe.

[Niraj Tolia]

Kevin,

This is not exactly what was asked for but you could also use a Guava InputSupplier wrapped in an InputStreamSupplierPayload. The InputSupplier's getInput() method can invoke whatever you do to generate the actual object and, unless you calculate MD5s, this method would only be called when you do the actual putBlob().

HTH,

Niraj

--
You received this message because you are subscribed to the Google Groups "jclouds" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jclouds+unsubscribe@googlegroups.com.