BlobStore - occational timeout or 400 error on upload - rackspace

198 views
Skip to first unread message

Kevin D

unread,
Mar 21, 2013, 5:02:00 PM3/21/13
to jcl...@googlegroups.com
I am trying to upload a ton of files into the blob store and have noticed a couple of problems and don't know if this is normal behavior.

As I am uploading 100 files most take less than a second but I have noticed that usually 1-5 of those 100 files will take particularly long (file size is small).   Usually 60+ seconds.  Similar problems to this were even more pronounced when I was using inputstreams to stream the files instead of giving jCloud direct File objects to sync (I changed due to replay ability errors).   When I did inputstreams I would get a lot more http timeout errors.

Would you consider this expected behavior and if not how do I debug this?  Call rackspace and open a ticket?

In addition to this I have noticed that even more rare I get a error 400 on one of the 100 files.

<title>Error response</title>
</head>
<body>
<h1>Error response</h1>
<p>Error code 400.
<p>Message: Bad request syntax ('0').
<p>Error code explanation: 400 = Bad request syntax or unsupported method.
</body>


Anybody experience similar behavior?

Everett Toews

unread,
Mar 22, 2013, 8:59:37 AM3/22/13
to <jclouds@googlegroups.com>, Andrew Gaul
I know Andrew Gaul was running into somewhat similar issues.

Andrew, did you come to any conclusions on it?

Regards,
Everett

--
You received this message because you are subscribed to the Google Groups "jclouds" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jclouds+u...@googlegroups.com.
To post to this group, send email to jcl...@googlegroups.com.
Visit this group at http://groups.google.com/group/jclouds?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Kevin D

unread,
Mar 22, 2013, 5:02:19 PM3/22/13
to jcl...@googlegroups.com, Andrew Gaul
For an example here is some code that illistrates the problem almost every time.   It is expecting jclouds 1.5.5, you pass in your username and api key as parameters and its expecting a test.txt file in the same directory.   When I ran it the test.txt file was a 45K file.   I have made it multi threaded as that shows the problem the fastest, but I have still seen the problem when switching the thread count down to 1.   As for the 400 error, I get it very rarely.

import org.jclouds.ContextBuilder;
import org.jclouds.blobstore.BlobStore;
import org.jclouds.blobstore.BlobStoreContext;
import org.jclouds.openstack.swift.CommonSwiftAsyncClient;
import org.jclouds.openstack.swift.CommonSwiftClient;
import org.jclouds.rest.RestContext;
import org.jclouds.blobstore.domain.Blob;
import org.jclouds.blobstore.domain.BlobBuilder;
import org.jclouds.blobstore.domain.BlobMetadata;

import java.lang.System;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;


import java.io.*;
import java.util.*;

public class TestFileUpload
{

public static String CONTAINER_NAME = "TEST_CONTAINER";
public static String CLOUD_FILENAME = "TEST_";

private static BlobStoreContext context = null;
private static BlobStore storage = null;
private static RestContext<CommonSwiftClient, CommonSwiftAsyncClient> swift = null;


public static void main(String[] args) throws Exception
{
String userName = args[0];
String apiKey = args[1];


context = ContextBuilder.newBuilder("cloudfiles-us")
.credentials(userName, apiKey)
.buildView(BlobStoreContext.class);
try {
storage = context.getBlobStore();
swift = context.unwrap();

ExecutorService executor = Executors.newFixedThreadPool(5);
for ( int i = 0; i < 200; i++ ) {
final int myI = i;
final TestFileUpload myUpload = new TestFileUpload();

Future<?> future = executor.submit(new Runnable()
{
@Override
public void run()
{
try {
myUpload.storeIt(myI);
}
catch (Exception e) {
System.out.println("ERROR :" + myI + " msg:" + e.getMessage());
}
}
});
}

executor.shutdown();
while ( !executor.awaitTermination(5, TimeUnit.SECONDS)) {
System.out.println("WAITING SHUTDOWN");
}


}
finally {
context.close();
}

}

private void storeIt( int i )
{
// make sure the container is there

storage.createContainerInLocation(null, CONTAINER_NAME);

File file = new File("test.txt");

String cloudFileName = CLOUD_FILENAME + i;

Blob blob = storage.blobBuilder(cloudFileName).payload(file).build();

long timeBefore = System.currentTimeMillis();
storage.putBlob(CONTAINER_NAME, blob);
long timeAfter = System.currentTimeMillis();

System.out.println("SYNCED " + i + " - TIME " + ((timeAfter - timeBefore)/1000) + " seconds" );
}

}

Everett Toews

unread,
Mar 23, 2013, 10:02:19 AM3/23/13
to <jclouds@googlegroups.com>, Andrew Gaul
Hi Kevin,

This is great. Now can you please enable logging and send us your output?

Checkout Logging.java [1] for an example of how to log and you can read more about logging at Logging in jclouds [2].

There will be a lot of output so please put the log file in a pastie [3] and just share the link with us !!! 

Thanks,
Everett

Adrian Cole

unread,
Mar 23, 2013, 10:47:55 AM3/23/13
to jcl...@googlegroups.com
Kevin,

There are known issues in versions <1.6 wrt thread starvation, which
could lead to certain bulk commands taking arbitrarily longer than
others. We are releasing 1.6.0-rc.2 this weekend. Watch for it and
give it a shot!

-A

Everett Toews

unread,
Mar 23, 2013, 12:02:33 PM3/23/13
to <jclouds@googlegroups.com>
This will also contain the fix for the HttpResponseException when writing to Blob with a InputStream [1] issue. See PR 1413 [2]. Let us know if that works for you too.

Everett

Everett Toews

unread,
Mar 23, 2013, 12:03:56 PM3/23/13
to <jclouds@googlegroups.com>
Adrian,

It looks like he's managing threads himself here. Is there an easier way to let jclouds manage (and limit if necessary) the threads?

Thanks,
Everett

On Mar 23, 2013, at 9:47 AM, Adrian Cole wrote:

Ignasi

unread,
Mar 23, 2013, 12:16:29 PM3/23/13
to jcl...@googlegroups.com

I don't know if it is appropriate in this use case, but the jclouds user executor (which is the one used to perform the operations against the target apis) is configured via properties when creating the context, and it can be accessed from context.getUtils()

Kevin D

unread,
Mar 25, 2013, 11:27:30 AM3/25/13
to jcl...@googlegroups.com
The reason I wrote my own threading is because my application is in a web server and therefore already threaded for 99% of what I need.    Then to get the files I already have out into the cloud I wrote a batch class to sync the files into the cloud using the jcloud code I had already written (which wasn't using async calls).   I didn't want to change the internals of my code so I put my threading on the outside.    The code I uploaded was attempting to mimmic my code to illustrate the problems that I was having.

I will try the 1.6 release candidate to see if that resolves the problems.   I know there was a error in the rackspace code for 1.5.8 which forced me to back down to 1.5.5, and 1.5.9 has not yet been put into the maven repositories yet, so I question whether the fix has been put into the latest release candidate yet either.

Andrew Phillips

unread,
Mar 25, 2013, 1:26:40 PM3/25/13
to jcl...@googlegroups.com
> I know there was a error in the rackspace code for 1.5.8 which forced me
> to back down to 1.5.5, and 1.5.9 has not yet been put into the maven
> repositories yet

1.5.9 should be in Maven Central, by the way...which JARs are you
looking for specifically?

http://repo1.maven.org/maven2/org/jclouds/jclouds-project/1.5.9/

Thanks!

ap

Everett Toews

unread,
Mar 25, 2013, 4:00:16 PM3/25/13
to <jclouds@googlegroups.com>
I wouldn't recommend 1.5.9. Another problem for Swift/Cloud Files crept in there. Best to try 1.6.0 RC2 when it's ready.

Everett

Kevin D

unread,
Mar 27, 2013, 11:05:06 AM3/27/13
to jcl...@googlegroups.com
Maybe I was looking in the wrong spot.   This is where I was looking.  http://mvnrepository.com/

Andrew Phillips

unread,
Mar 27, 2013, 2:22:42 PM3/27/13
to jcl...@googlegroups.com
> Maybe I was looking in the wrong spot. This is where I was looking.
> http://mvnrepository.com/

Ah, that might explain it. The "official" search engine for Maven
Central is at:

http://search.maven.org/

Regards

ap

Kevin D

unread,
Mar 28, 2013, 4:00:54 PM3/28/13
to jcl...@googlegroups.com, aphi...@qrmedia.com
So I updated to 6 rc2 and still get similar problems.    It seems to be mostly right after the first authentication where the thread hangs for 60 seconds.

Here is the output log file with the problem.

Here is output of the time it took for each sync

Kevin D

unread,
Mar 28, 2013, 4:31:03 PM3/28/13
to jcl...@googlegroups.com, aphi...@qrmedia.com
Unfortunately after all my cleansing of my log file I left the api key in the log file.  I have reset it so it shouldn't work ;-) 

Andrew Phillips

unread,
Mar 29, 2013, 9:28:50 AM3/29/13
to jcl...@googlegroups.com
Hi Kevin

> Unfortunately after all my cleansing of my log file I left the api key in
> the log file. I have reset it so it shouldn't work ;-)

Thanks for those. Curious if Andrew Gaul or Everett have any more insight.

Just a quick check: this is the same delay/error that you were seeing
with 1.5.5? I'm asking because I see 100-continue in the requests and
know there have been some changes in jclouds related to that recently.

Also, which JDK version are you running on?

Thanks!

ap

Kevin D

unread,
Mar 29, 2013, 10:22:22 AM3/29/13
to jcl...@googlegroups.com, aphi...@qrmedia.com
I don't know if they are the exact same error, I never put logging into the program using the older version.   My jdk is 1.6.0_43

Kevin D

unread,
Mar 29, 2013, 10:39:49 AM3/29/13
to jcl...@googlegroups.com, aphi...@qrmedia.com
I generated another set of logs using the older 1.5.5 version.

Andrew Phillips

unread,
Mar 29, 2013, 10:57:54 AM3/29/13
to jcl...@googlegroups.com
> I generated another set of logs using the older 1.5.5 version.

Great, thanks!

ap

Everett Toews

unread,
Mar 29, 2013, 5:38:49 PM3/29/13
to <jclouds@googlegroups.com>
I ran this this sometime last week and was seeing these types of errors. So it was definitely a problem at some point in time.

However, I just recently tried this with 1.6.0-rc.2 and didn't get a single error after running it 5 times in a row. The only slight difference in my code was to comment out the container creation.

Here's exactly what I did.


$ java -jar ivy-2.3.0.jar -dependency org.jclouds.provider cloudfiles-us 1.6.0-rc.2 -retrieve "lib/[artifact]-[revision](-[classifier]).[ext]"

$ javac -cp ".:lib/*" TestFileUpload.java

$ java -cp ".:lib/*" TestFileUpload username apikey

My results are at http://pastie.org/7166347

$ java -version
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

Kevin, can you please try again with Java 1.6? Are you able to try with Java 1.7?

Thanks,
Everett

Reply all
Reply to author
Forward
0 new messages