How to create a proxy server for large files uploading using Java Play?

1,007 views
Skip to first unread message

Alexandr Porunov

unread,
Sep 4, 2016, 3:39:47 AM9/4/16
to Play Framework
Hello,

I have to create a proxy for file uploading but the problem is that I don't have to store an entire file on the proxy server. My proxy have to accept very big files (files with size up to 80 GB) and store an entire file in the object storage (openstack swift). But proxy don't have to keep an entire file in the RAM or on the disk. Proxy have to act like a pipe (connect input stream and output stream).
I am new to Play. How to get input stream when a user sends a file? Is it possible to create such a proxy server with Play using Java?

Sincerely,
Alexandr

Christian Schmitt

unread,
Sep 4, 2016, 4:53:05 AM9/4/16
to Play Framework
Actually if you are new to play this task is probably too hard for you.

That will actually accumulate the file as a Akka Stream (Source[ByteString, Object]) this one can then directly be feed to the Play WS Library (which is a async http client) to upload the file to openstack swift.

alan...@gmail.com

unread,
Jan 17, 2017, 4:06:14 AM1/17/17
to Play Framework
Hi, @Christian Schmitt . Can we use this https://www.playframework.com/documentation/2.5.x/ScalaBodyParsers#directing-the-body-elsewhere to proxy request?

Is that one load all request the memory?

Will Sargent

unread,
Jan 17, 2017, 6:31:39 PM1/17/17
to play-fr...@googlegroups.com
There is an example project in Play with a custom body parser that can stream off a file upload that you can use:


This is actually available from the Play Framework download page: https://playframework.com/download



--
Will Sargent
Engineer, Lightbend, Inc.


--
You received this message because you are subscribed to the Google Groups "Play Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framework+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/92bde884-1ea2-4d61-ad3b-e4a70df90502%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

alan...@gmail.com

unread,
Jan 18, 2017, 10:52:35 AM1/18/17
to Play Framework
@Will Sargent thanks for reply. Actually, what I want (and the OP) is generic proxy to another service in my case Play WS. I need to proxy all requests without loading body into memory.
Is it possible in Play to proxy coming request to another service with WS without loading body into memory? (Let me know if I should create another thread for that one for the sake of clarity)


On Wednesday, January 18, 2017 at 2:31:39 AM UTC+3, Will Sargent wrote:
There is an example project in Play with a custom body parser that can stream off a file upload that you can use:


This is actually available from the Play Framework download page: https://playframework.com/download



--
Will Sargent
Engineer, Lightbend, Inc.


On Mon, Jan 16, 2017 at 12:01 PM, <alan...@gmail.com> wrote:
Hi, @Christian Schmitt . Can we use this https://www.playframework.com/documentation/2.5.x/ScalaBodyParsers#directing-the-body-elsewhere to proxy request?

Is that one load all request the memory?


On Sunday, September 4, 2016 at 12:53:05 PM UTC+4, Christian Schmitt wrote:
Actually if you are new to play this task is probably too hard for you.

That will actually accumulate the file as a Akka Stream (Source[ByteString, Object]) this one can then directly be feed to the Play WS Library (which is a async http client) to upload the file to openstack swift.

Am Sonntag, 4. September 2016 09:39:47 UTC+2 schrieb Alexandr Porunov:
Hello,

I have to create a proxy for file uploading but the problem is that I don't have to store an entire file on the proxy server. My proxy have to accept very big files (files with size up to 80 GB) and store an entire file in the object storage (openstack swift). But proxy don't have to keep an entire file in the RAM or on the disk. Proxy have to act like a pipe (connect input stream and output stream).
I am new to Play. How to get input stream when a user sends a file? Is it possible to create such a proxy server with Play using Java?

Sincerely,
Alexandr

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.

Edem Nsefik

unread,
May 25, 2017, 9:24:12 AM5/25/17
to Play Framework
I have tried to upload a 5Gib file using the play-fileupload-scala and I get a Bad Request   [Request Entity Too Large]
In the error stack I noticed ->play.api.mvc.BodyParsers$MaxLengthLimitAttained: null

What change gets me passed this limitation

On Tuesday, January 17, 2017 at 11:31:39 PM UTC, Will Sargent wrote:
There is an example project in Play with a custom body parser that can stream off a file upload that you can use:


This is actually available from the Play Framework download page: https://playframework.com/download



--
Will Sargent
Engineer, Lightbend, Inc.


On Mon, Jan 16, 2017 at 12:01 PM, <alan...@gmail.com> wrote:
Hi, @Christian Schmitt . Can we use this https://www.playframework.com/documentation/2.5.x/ScalaBodyParsers#directing-the-body-elsewhere to proxy request?

Is that one load all request the memory?


On Sunday, September 4, 2016 at 12:53:05 PM UTC+4, Christian Schmitt wrote:
Actually if you are new to play this task is probably too hard for you.

That will actually accumulate the file as a Akka Stream (Source[ByteString, Object]) this one can then directly be feed to the Play WS Library (which is a async http client) to upload the file to openstack swift.

Am Sonntag, 4. September 2016 09:39:47 UTC+2 schrieb Alexandr Porunov:
Hello,

I have to create a proxy for file uploading but the problem is that I don't have to store an entire file on the proxy server. My proxy have to accept very big files (files with size up to 80 GB) and store an entire file in the object storage (openstack swift). But proxy don't have to keep an entire file in the RAM or on the disk. Proxy have to act like a pipe (connect input stream and output stream).
I am new to Play. How to get input stream when a user sends a file? Is it possible to create such a proxy server with Play using Java?

Sincerely,
Alexandr

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages