OutOfMemory error when posting data to MultipartRequest

[jensant...@gmail.com]

Hi,

I have a test case:

try {
            MultipartRequest req = new MultipartRequest();
            req.setUrl("http://www.codenameone.com");
            req.setHttpMethod("POST");

            req.addData("test", Storage.getInstance().createInputStream("TestFile1"), 0, " application/octet-stream");

            NetworkManager.getInstance().addToQueueAndWait(req);

            System.out.println(new String(req.getResponseData(), "UTF-8"));

        } catch (Exception ex){
            ex.printStackTrace();
            Dialog.show("Test", ex.getMessage(), "OK", null );
        }

The "TestFile1" is more than 70 MB. Suppose I do not know the file size for some reasons (e.g. encryption, compression, etc), so that I give 0 as the file size. For the URL, it doesn't matter that it does nothing, because I just want to try sending the file out in this test case.

After some time, it gives java.lang.OutOfMemoryError: Java heap space. I have also tried using ConnectionRequest instead of multipart, so that the file will be uplaoded in "buildRequestBody()", but it behaves just the same. Adding flush() and flushBuffer() all around doesn't help.

It seems like the OutputStream given by ConnectionRequest has buffered all the data before sending out. Just wonder if it is possible to allow streaming upload, so that the file size doesn't matter at all?

Thanks.

[Shai Almog]

Hi,
don't do that.
Copy the file to FileSystemStorage (e.g. and the app home dir). Then use the file URL to add the data. This will be far more efficient in terms of RAM/performance and will work seamlessly with the file size. It will be able to upload files that are much larger than RAM if you do it that way.

[jensant...@gmail.com]

At the backend of "file URL", it is calling FileSystemStorage.getInstance().openInputStream(filePath). So do you mean FileSystemStorage.getInstance().getInputStream(filePath) has some handling different from Storage.getInstance().createInputStream(filePath), so that it can handle big files? Or do you mean we have to provide the file size anyway?

Also, with this method, if I want to compress it before uploading, I will need to save the whole compressed file into the filesystem, and then read it for uploading. Will the extra I/O be a problem?


BTW, I have just tried with your method in emulator but it still gives "java.lang.OutOfMemoryError: Java heap space"

try {

            MultipartRequest req = new MultipartRequest();
            req.setUrl("http://www.codenameone.com");
            req.setHttpMethod("POST");

            req.addData("test", "TestFile1", "application/octet-stream");

            NetworkManager.getInstance().addToQueueAndWait(req);

            System.out.println(new String(req.getResponseData(), "UTF-8"));

        } catch (Exception ex){
            ex.printStackTrace();
            Dialog.show("Test", ex.getMessage(), "OK", null );
        }

Thanks.

[Shai Almog]

Submitting this to our website doesn't make sense and might get your IP blocked by our CDN.
TestFile isn't a valid file path, you must manufacture file paths dynamically since file paths change based on device.
I suggest you look at the file upload sample here:
http://codenameone.com/blog/build-mobile-ios-apps-in-java-using-codename-one-on-youtube
It also includes the server code.

[jensant...@gmail.com]

Alright, I will not continue submitting to your website. As I said before, I just want to show that the output stream of connection request will give outodmemory error if the file size is too big.

I can't see how the sample is related to my problem. What's the difference between uploading a file and an image using multipart connection request? The test file is a valid file which I have created in the filesystem (like what you said "copy" it to the filesystem storage).

I don't need a customized server. Assuming I have a server which accepts users to post files. Now I want to post a very big file to it. The outputstream gives outof memory error when it continues to write after 64MB has been written.

Hopefully I haven't made it too complicated.

Thanks

[Shai Almog]

OK that's the part that was missing, I'm guessing you got the out of memory during the copy phase and not during the upload phase.
We have guys uploading gigabytes of data with the connection request so it should work just fine.

[jensant...@gmail.com]

The copy phase will not have problems because it is already done.
After the file is copied, it is put under the file storage system and I didn't copy again and again for every test. I just upload the file directly like the code above.

[jensant...@gmail.com]

I have setup a server (localhost) for receiving files, and print a system message when the servlet is triggered. The servlet will save the received data to another file.
Then I run the simulator and send the file by the method

addData(String name, String filePath, String mimeType)

After some time, the out of memory error occurs and there is no message printed in the server.
Tested with small files and it works perfectly.
So it will not be the problem of invalid filename, or server settings, etc.

I believe the connection request is actually trying to create a very big request, with the file contents added to the request body directly.

[Shai Almog]

Do you have the network monitor open by any chance?
Its possible that this is an issue related to that, if not we might need to look for a potential regression.

[jensant...@gmail.com]

I had also suspected that it is the network monitor problem before, and so I have tested many times without network monitor. Same result.
The same result can be reproduced with real phones.

Also tested with both NetworkManager.getInstance().addToQueue(req) and NetworkManager.getInstance().addToQueueAndWait(req). Same.

Here is the stacktrace:

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2271)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
    at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
    at com.codename1.io.BufferedOutputStream.write(BufferedOutputStream.java:170)
    at com.codename1.io.MultipartRequest.buildRequestBody(MultipartRequest.java:269)
    at com.codename1.io.ConnectionRequest.performOperation(ConnectionRequest.java:339)
    at com.codename1.io.NetworkManager$NetworkThread.run(NetworkManager.java:261)
    at com.codename1.impl.CodenameOneThread.run(CodenameOneThread.java:176)
[Network Thread] 0:0:0,0 - java.lang.OutOfMemoryError: Java heap space
[Network Thread] 0:0:0,15 - java.lang.OutOfMemoryError: Java heap space

[Shai Almog]

Can you file an issue on this with the test case to the issue tracker?
Which Java version are you using?
We test normally with JDK 7, is it possible you used JDK 8 and this is a regression?

[jensant...@gmail.com]

using JDK1.7.0_65

Issue #1425 submitted.

Thanks.