File Upload and multipart/form-data

jg

unread,

Dec 5, 2011, 12:00:36 PM12/5/11

to asyncht...@googlegroups.com

Hi,

I would like to use AHC for translating a curl command for sending a file to a REST API. I thought it might be easy, but I haven't been successful so far... The working curl command is like so:

$ curl -X POST -u login:passwd http://url -F filedata=@a_file -F foo=bar -F baz=qux -F //etc.

I ended up with this script (in Javascript/Rhino)

var realm = new Realm.RealmBuilder().setPrincipal(login).setPassword(passwd).setScheme(Realm.AuthScheme.BASIC).build();

var cfg = new AsyncHttpClientConfig.Builder().setProxyServer(new ProxyServer("127.0.0.1", 8888)).build();

asyncHttpClient = new AsyncHttpClient(cfg);

url = "http://..."

ahc = asyncHttpClient.preparePost( url)

ahc.setRealm(realm)

ahc.setHeader("Content-Type", "multipart/form-data")

ahc.addBodyPart(new StringPart("foo", "bar"))

ahc.addBodyPart(new StringPart("baz", "qux"))

ahc.addBodyPart( new com.ning.http.client.FilePart(new File("a_file").name, new File("a_file"), "text/plain", "UTF-8"))

ahc.execute().get().responseBody

The request is malformed as params are passed differently than with curl. Should be:

Content-Disposition: form-data; name="foo"

bar

and is, for each field, with AHC:

Content-Disposition: form-data; name="foo"

Content-Type: text/plain; charset=UTF-8

Content-Transfer-Encoding: 8bit

bar

All examples found didn't help, sorry! Hope you have a tip!

Thanks

Best regards

jg

unread,

Dec 5, 2011, 12:53:18 PM12/5/11

to asyncht...@googlegroups.com

As usual, posting brought light! and the former script works in fact, (some params weren't correctly named), even if it might not be perfect! Of course, I tried with bigger and bigger files as the goal is to explore loading large files and I bumped into this exception:

Does it tell anybody something?

java.util.concurrent.ExecutionException: java.io.IOException: Unable to write on channel java.nio.channels.SocketChannel[connected local=/<IP> remote=<remote_IP>]

at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:293)

at com.ning.http.client.providers.netty.NettyAsyncHttpProvider$ProgressListener.operationComplete(NettyAsyncHttpProvider.java:1812)

at com.ning.http.client.providers.netty.NettyAsyncHttpProvider$4.operationComplete(NettyAsyncHttpProvider.java:459)

at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)

at org.jboss.netty.channel.DefaultChannelFuture.addListener(DefaultChannelFuture.java:148)

at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.writeRequest(NettyAsyncHttpProvider.java:452)

at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:78)

at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)

at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:372)

at org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:316)

at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:776)

at org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:258)

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)

at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)

Caused by: java.io.IOException: Unable to write on channel java.nio.channels.SocketChannel[connected local=<IP> remote=<REMOTE_IP>]

at com.ning.http.multipart.MultipartBody.writeToTarget(MultipartBody.java:550)

at com.ning.http.multipart.MultipartBody.transferTo(MultipartBody.java:332)

at com.ning.http.client.providers.netty.BodyFileRegion.transferTo(BodyFileRegion.java:46)

at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$FileSendBuffer.transferTo(SocketSendBufferPool.java:281)

at org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:472)

at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:390)

at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:127)

at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:68)

at org.jboss.netty.handler.codec.http.HttpClientCodec.handleDownstream(HttpClientCodec.java:82)

at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:268)

at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:125)

at org.jboss.netty.channel.Channels.write(Channels.java:611)

at org.jboss.netty.channel.Channels.write(Channels.java:578)

at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)

at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.writeRequest(NettyAsyncHttpProvider.java:445)

at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:78)

at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)

at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:372)

at org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:316)

at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:776)

at org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:258)

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)

... 3 more

Jeanfrancois Arcand

unread,

Dec 6, 2011, 6:27:20 AM12/6/11

to asyncht...@googlegroups.com

Salut,

sound like the remote connection was closed. Can you snoop the network
and post the result here?

Thanks

-- Jeanfrancois

> --
> You received this message because you are subscribed to the Google
> Groups "asynchttpclient" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/asynchttpclient/-/1B6t1gUWYD0J.
> To post to this group, send email to asyncht...@googlegroups.com.
> To unsubscribe from this group, send email to
> asynchttpclie...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/asynchttpclient?hl=en.

jg

unread,

Dec 6, 2011, 10:09:41 AM12/6/11

to asyncht...@googlegroups.com

Hi Jean-François,

thank you for your reply, sorry for not being that accurate with my question (I'm still discovering the project and API).

I did test the upload of a large file (470MB) with curl, and it went OK (upload takes 1 min with curl) to be sure it can work.

Now using AHC--1.6.4, netty-3.2.7Final

1st trial /jdk-7
2011-12-06 14:17:49,250 INFO   MultipartBody - Waiting for writing...
2011-12-06 14:17:50,254 INFO   MultipartBody - Waiting for writing...
2011-12-06 14:17:51,254 INFO   MultipartBody - Waiting for writing...
2011-12-06 14:17:52,254 INFO   MultipartBody - Waiting for writing..
[...]

org.mozilla.javascript.WrappedException: Wrapped java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: No response received after 60000 (<eval'ed string>#1(Function)#341)
        [rhino exception stack ...]
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: No response received after 60000
        at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:219)
        at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:205)
        at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:180)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:161)
        ... 19 more
Caused by: java.util.concurrent.TimeoutException: No response received after 60000
        at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:211)
        ... 26 more
--
Other trial (jdk-7) : (checking with Fiddler)
2011-12-06 14:19:44,911 INFO   MultipartBody - Waiting for writing...
2011-12-06 14:19:45,912 INFO   MultipartBody - Waiting for writing...
2011-12-06 14:19:46,914 INFO   MultipartBody - Waiting for writing...
2011-12-06 14:19:47,915 INFO   MultipartBody - Waiting for writing...
2011-12-06 14:19:48,916 INFO   MultipartBody - Waiting for writing..
[...]
[more or less long depending on trials]

org.mozilla.javascript.WrappedException: Wrapped java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: No response received after 60000 (<eval'ed string>#1(Function)#343)
       [rhino exception stack...]
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: No response received after 60000
        at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:219)
        at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:205)
        at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:180)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:161)
        ... 19 more
Caused by: java.util.concurrent.TimeoutException: No response received after 60000
        at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:211)
        ... 26 more
--- I use Fiddler as HTTP Proxy:
Fiddler has detected a protocol violation in session #1.

Content-Length mismatch: Request Header indicated 489 384 523 bytes, but client sent 15 729 579 bytes (many trials with same value or just slightly different)
Crashes happens just before ~2 min...

Last trials before sending this mail. I used jdk-1.6.0_23 instead of 1.7.0_1. All trials with jdk-7 really failed just before 2 minutes.
Some independant trials with jdk-6 went well, except when doing them in a small series, one test crashes... Then, after a while, it crashed with yesterday exception... I'm puzzled!

Hope this is enough info,
Thanks for any tip as it sounds like bad configuration.

Best regards

Jeanfrancois Arcand

unread,

Dec 6, 2011, 1:22:03 PM12/6/11

to asyncht...@googlegroups.com

Salut,

looks like there is a bug in AHC. Can you try 1.7.0-SNAPSHOT (I think it
got fixed) and let me know? The workaround is to increase the request
timeout.

A+

-- Jeanfrancois

On 11-12-06 10:09 AM, jg wrote:
> Hi Jean-Fran�ois,

> Method)Salut,

> --
> You received this message because you are subscribed to the Google
> Groups "asynchttpclient" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/asynchttpclient/-/EmZxR9TShXwJ.

jg

unread,

Dec 7, 2011, 1:24:44 PM12/7/11

to asyncht...@googlegroups.com

Hi,

couldn't spend much time today, but I'm trying to help. I tried to compile AHC (as 1.7.snapshot jar is not available) but got errors with test (mvn clean install or mvn compile then mvn package with same results) :

(mvn-3.0.1, jdk1.6.0_23 or 1.7 idem).

Hope it helps,

Best regards

C:\Git\async-http-client> mvn package

[INFO] Scanning for projects...

[INFO]

[INFO] ------------------------------------------------------------------------

[INFO] Building Asynchronous Http Client 1.7.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[INFO]

[INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-maven) @ async-http-client ---

[INFO]

[INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-versions) @ async-http-client ---

[INFO]

[INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ async-http-client ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] skip non existing resourceDirectory C:\Git\async-http-client\src\main\resources

[INFO]

[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ async-http-client ---

[INFO] Nothing to compile - all classes are up to date

[INFO]

[INFO] --- animal-sniffer-maven-plugin:1.6:check (check-java-1.5-compat) @ async-http-client ---

[INFO] Checking unresolved references to org.codehaus.mojo.signature:java15:1.0

[INFO]

[INFO] --- maven-resources-plugin:2.4.3:testResources (default-testResources) @ async-http-client ---

[INFO] Using 'UTF-8' encoding to copy filtered resources.

[INFO] Copying 10 resources

[INFO]

[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ async-http-client ---

[INFO] Nothing to compile - all classes are up to date

[INFO]

[INFO] --- maven-surefire-plugin:2.8.1:test (default-test) @ async-http-client ---

[INFO] Surefire report directory: C:\Git\async-http-client\target\surefire-reports

-------------------------------------------------------

T E S T S

-------------------------------------------------------

Running TestSuite

Tests run: 499, Failures: 18, Errors: 0, Skipped: 0, Time elapsed: 386.497 sec <<< FAILURE!

Results :

Failed tests:

testAsyncHttpProviderConfig(com.ning.http.client.async.grizzly.GrizzlyAsyncProviderBasicTest): java.io.IOException: Remotely Closed

testAwsS3(com.ning.http.client.async.grizzly.GrizzlyAsyncProviderBasicTest): java.io.IOException: Remotely Closed

asyncStream301RedirectWithBody(com.ning.http.client.async.grizzly.GrizzlyAsyncStreamHandlerTest): Timeout out

multipleSSLWithoutCacheTest(com.ning.http.client.async.grizzly.GrizzlyBasicHttpsTest): expected:<hello there> but was:<h>

zeroCopyPostTest(com.ning.http.client.async.grizzly.GrizzlyBasicHttpsTest): expected:<This is a simple test file> but was:<T>

reconnectsAfterFailedCertificationPath(com.ning.http.client.async.grizzly.GrizzlyBasicHttpsTest): expected:<hello there> but was:<h>

multipleSSLRequestsTest(com.ning.http.client.async.grizzly.GrizzlyBasicHttpsTest): expected:<hello there> but was:<h>

testFollowRedirect(com.ning.http.client.async.grizzly.GrizzlyFollowingThreadTest): Method public void com.ning.http.client.async.FollowingThreadTest.testFollowRedirect() throws java.io.IOException,java.util.concurrent.ExecutionException,java.util.concurrent.TimeoutException,java.lang.InterruptedException didn't finish within the time-out 30000

notRedirected302Test(com.ning.http.client.async.grizzly.GrizzlyPerRequestRelative302Test): expected:<302> but was:<200>

redirected302Test(com.ning.http.client.async.grizzly.GrizzlyRelative302Test): java.io.IOException: Remotely Closed

evilCoookieTest(com.ning.http.client.async.grizzly.GrizzlyRemoteSiteTest): java.io.IOException: Remotely Closed

stripQueryStringNegativeTest(com.ning.http.client.async.grizzly.GrizzlyRemoteSiteTest): java.io.IOException: Remotely Closed

stripQueryStringTest(com.ning.http.client.async.grizzly.GrizzlyRemoteSiteTest): java.io.IOException: Remotely Closed

asyncFullBodyProperlyRead(com.ning.http.client.async.grizzly.GrizzlyRemoteSiteTest): java.io.IOException: Remotely Closed

testAHC60(com.ning.http.client.async.grizzly.GrizzlyRemoteSiteTest): java.io.IOException: Remotely Closed

zeroCopyPostTest(com.ning.http.client.async.netty.NettyBasicHttpsTest): expected:<This is a simple test file> but was:<T>

positiveHostnameVerifierTest(com.ning.http.client.async.netty.NettyHostnameVerifierTest): expected:<This is a simple test file> but was:<T>

stripQueryStringNegativeTest(com.ning.http.client.async.netty.NettyRemoteSiteTest): com.ning.http.client.MaxRedirectException: Maximum redirect reached: 5

Tests run: 499, Failures: 18, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 6:34.173s

[INFO] Finished at: Wed Dec 07 19:13:43 CET 2011

[INFO] Final Memory: 25M/721M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.8.1:test (default-test) on project async-http-client: There are test failures.

[ERROR]

[ERROR] Please refer to C:\Git\async-http-client\target\surefire-reports for the individual test results.

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

jg

unread,

Jan 31, 2012, 10:03:57 AM1/31/12

to asyncht...@googlegroups.com

Hi,

I've tried latest version 1.7.0 + netty-3.2.7 without success for middle large files. I still get the same exception as with ahc-1.6.5:

Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Unable to write on channel java.nio.channels.SocketChannel[connected local=/<local IP>18897 remote=<remote IP>:8342]