File upload

[George Moschovitis]

I was looking at the upload example in Jack (and jack/utls).

Form what I see an uploaded file is moved to a temporary directory and some info is returned in params().

This scheme is useful in normal situations. However, I would like to upload files to a Google AppEngine applications.

Appengine does not allow you to create files, so I would like to have access to the  ByteString of the uploaded file.

Is this possible?

regards,

George.

--
blog.gmosx.com

[Tom Robinson]

It's certainly possible, but it will require some modification to utils.parseMultipart()

One possibility would be to add an "options" argument to parseMultipart, with a flag to tell it to return a ByteString in place of the temp file name.

-tom

[George Moschovitis]

That would be nice (and make Jack more AppEngine friendly)

-g.

--
blog.gmosx.com

[George Moschovitis]

A related question. Shouldn't something like this work?

exports.GET = function(env) {

  return [200, {"Content-Type": "image/png"}, [file.read("mypic.png")]];

}

-g.

--
blog.gmosx.com

[Tom Robinson]

Yes, does it not?

The jack/file module will do basically the same thing.

[George Moschovitis]

Yes, does it not?

The image is not shown.

The content length is about two times the size of the file. Perhaps some encoding problem? Am I doing something wrong?

-g.

--
blog.gmosx.com

[George Moschovitis]

On Sun, Aug 23, 2009 at 1:06 PM, George Moschovitis <george.mo...@gmail.com> wrote:

Yes, does it not?

The image is not shown.

I think the image gets corrupted.

--
blog.gmosx.com

[Tom Robinson]

Try reading as binary:

exports.GET = function(env) {
return [200, {"Content-Type": "image/png"}, [file.read("mypic.png",
{ mode : "b" })]];

[George Moschovitis]

That does the trick, thank you!


Another question. Is there a way to convert ByteString/ByteArray from/to java byte[] arrays?

-g.

--
blog.gmosx.com

[Tom Robinson]

Yeah there's basically two ways.

1) In Rhino ByteArrays/Strings currently have _bytes, _offset, and _length properties. The _bytes property is Java byte[], and _offset and _length are JavaScript Numbers. You could just allocate a new byte[] of size _length, and copy _bytes starting at _offset (or use _bytes directly if you don't need a copy)

2) However, it's possible this could change in the future, so another way that is pretty much guaranteed to always work is to use one of the ByteArray/String public APIs, probably either toArray() to get an Array of Numbers, or the length property along with the get() method to get them individually.

Note that Java bytes are signed (ugh) so you have to do some conversion between our bytes (0 to 255) and Java bytes (-128 to 127). Here's the get/set functions from the Rhino binary-platform.js module:

exports.B_GET = function(bytes, index) {

    var b = bytes[index];

    return (b >= 0) ? b : -1 * ((b ^ 0xFF) + 1);

}   

exports.B_SET = function(bytes, index, value) {

    return bytes[index] = (value < 128) ? value : -1 * ((value ^ 0xFF) + 1);

}

The first is likely much faster and easier, but the second is future-proof. I'd probably just use the first unless it will be hard to change down the road.

Perhaps we should have a public API to get the "native" representation so you don't have to rely on a hack but get good performance.

-tom

[George Moschovitis]

> Perhaps we should have a public API to get the "native" representation so you don't have to rely on a hack but get good performance.

that would be great!

-g.

--
blog.gmosx.com

[Hannes Wallnoefer]

Rhino has the concept of Wrappers to expose java objects to JS. By
implementing ByteArrays as wrappers they would automatically be
converted (unwrapped) to java byte arrays when passed to a Java
method. Using a cusom WrapFactory, it would also be possible to
automatically wrap java byte arrays to ByteArrays from Java to
JavaScript.

http://www.mozilla.org/rhino/apidocs/org/mozilla/javascript/Wrapper.html
http://www.mozilla.org/rhino/apidocs/org/mozilla/javascript/WrapFactory.html

Of course, the offset and length attributes would be lost in an
automatic conversion, but I'm not sure how important they are for
actual use.

I promised Kris to help with a native ByteArray implementation for
Rhino some time ago, now may be a good time to do so. I'll see if I
can come up with some code.

Hannes

2009/8/24 George Moschovitis <george.mo...@gmail.com>:

[Hannes Wallnoefer]

FYI, I've committed a native binary/b ByteArray implementation for
Rhino to the Helma NG git repository. It currently implements all
constructors described in the proposal plus a non-standard one that
reads from a java.io.InputStream, indexed element access, the length
property and a few methods.

http://github.com/hns/helma-ng/blob/master/src/org/helma/util/ByteArray.java

The remaining methods can probably be implemented in JavaScript quite
easily by adding to ByteArray.prototype, maybe even using the Array
generics (Array.push, Array.pop, etc).

The nice thing is that you can use this directly when calling java
methods that take a byte[], so no need to make the java byte array
accessible:

var b = new ByteArray('hello world', 'utf8');
new java.lang.String(b)
hello world

To use it simply register the host object class using the
defineClass() function: defineClass(org.helma.util.ByteArray).

If this is useful for Narwhal I'd be happy to help integrating it.

Hannes

2009/8/24 Hannes Wallnoefer <han...@gmail.com>:

[Tom Robinson]

One of the nice things about the current Binary module is that it's
reusable on any platform that can implement the 10 or so simple
methods in "binary-engine" module. After I had implemented it for
Rhino and extracted out the Java parts it was almost trivial to get it
working with a plain old JavaScript Array-backed version, and a V8 one
that used char arrays.

For that reason I'd like to keep it in JavaScript, unless performance
is an issue (which it may be, I have no idea)

I've gone ahead and incorporated the unwrap thing into the Rhino
version of the binary objects, so ByteString and ByteArray will be
unwrapped into byte[] if passed to a function that takes a byte[].

http://github.com/tlrobinson/narwhal/commit/52aad1640deb3177ee2cf29217f5feb5a064a2a2

-Tom

[Dean Landolt]

I've gone ahead and incorporated the unwrap thing into the Rhino
version of the binary objects, so ByteString and ByteArray will be
unwrapped into byte[] if passed to a function that takes a byte[].

I updated to the latest narwhal and went ahead and tried something like:

var ba = new ByteArray([65,66,67]); // "ABC"
var jba = new java.io.ByteArrayInputStream(ba);

But the ByteArrayInputStream constructor chokes saying it can't take an object. Am I doing this right? How exactly is this unwrapping supposed to work?

[Tom Robinson]

Sorry, I had to back out the change because it was causing some tests
to fail and I didn't have a chance to look into it.

If you're interested:

engines/rhino/lib/binary-engine.js

Uncomment the last few lines.

[Dean Landolt]

On Wed, Aug 26, 2009 at 10:50 PM, Tom Robinson <tlrob...@gmail.com> wrote:

Sorry, I had to back out the change because it was causing some tests
to fail and I didn't have a chance to look into it.

If you're interested:

engines/rhino/lib/binary-engine.js

Uncomment the last few lines.

Thanks Tom. I gave that a shot:

js>var ByteArray = require('binary').ByteArray;

js>var ba = new ByteArray([65,66,67]);

js>var jba = new java.io.ByteArrayInputStream(ba);
Java constructor for "java.io.ByteArrayInputStream" with arguments "adapter1" no
t found.


I'm in no big hurry for this -- I just want to confirm if I'm doing this right. Is there something else I need to wrap or pass?

[Tom Robinson]

I went ahead and did this (and in the process finished the "output" half of BinaryIO. Though I'm not sure it's correct: input and output are completely independent)

Currently parseMultipart takes a second "options" parameter, which if it contains a truthy "nodisk" it will return a ByteString in "tempfile" instead of a string containing the path of the temporary file. We'll probably want to change that. Also, we probably want to add some sort of configuration in Request to add that option.

I haven't tested it extensively. We're still using Strings to pass around binary data within parseMultipart, which may be a problem. That function is a beast which needs to be rewritten.

Anyway, give it a try and let me know how it goes.

-tom

[George Moschovitis]

I love you ;-)

I am going to try this right away!

-g.

--
blog.gmosx.com

[Hannes Wallnoefer]

2009/9/2 Tom Robinson <tlrob...@gmail.com>:

> I went ahead and did this (and in the process finished the "output" half of
> BinaryIO. Though I'm not sure it's correct: input and output are completely
> independent)
> Currently parseMultipart takes a second "options" parameter, which if it
> contains a truthy "nodisk" it will return a ByteString in "tempfile" instead
> of a string containing the path of the temporary
> file. We'll probably want to change that. Also, we probably want to add some sort of configuration in Request to add that option.
> I haven't tested it extensively. We're still using Strings to pass around
> binary data within parseMultipart, which may be a problem. That function is
> a beast which needs to be rewritten.

I think the code is actually quite neat. It just highlights some
functionalty that may be missing from the binary and io modules. For
example, if it would be possible to search for multi-byte sequences
with ByteArray.indexOf, and ByteArray.decodeToString took optional
start and end argument to create a string from just a part of itself,
you could just stringify the header part of the buffer.

The next thing that strikes me is that you have to do a slice() to
write a part of a String or ByteArray. That write() method should
really take optional start and end arguments, too, and ByteArray
should have a method to copy or move bytes within itself or to other
ByteArrays, similar to Java's System.arraycopy. Then, once narwhal
implements IO.readInto(), you'll be able to implement this with a
single fixed-size ByteArray buffer :-)

I'm currently implementing file, io, and binary for Helma NG and
taking notes on any such things I find, and I'll post them to a wiki
page soon.

Hannes

[George Moschovitis]

OK, I gave it a try,

I think it does not work :( The data seems corrupted.

Have you actually tried this with a binary file?

-g.

On Wed, Sep 2, 2009 at 2:37 PM, Tom Robinson <tlrob...@gmail.com> wrote:

--
blog.gmosx.com

[George Moschovitis]

This is some code I tried:

var parseMultipart = require("jack/utils").parseMultipart;

exports.POST = function(env) {

    var params = parseMultipart(env, {nodisk: true});

    

    return {

        status: 200, 

        headers: {

            "Content-Type": params.file.type

        },

        body: [params.file.tempfile]

    };  

}

shouldn't this work? The image gets corrupted.

regards,

George.

--
blog.gmosx.com

[Tom Robinson]

I haven't tested it carefully, and as I mentioned it's still using Strings to pass binary data around (though it was before too) so it's possible it's getting corrupted there.

Is it at least working with ASCII or UTF-8 files?

-tom

[George Moschovitis]

Is it at least working with ASCII or UTF-8 files?

Yeap, it works with ASCII/UTF-8 files...

But that's not really useful, typically you upload binary files.

Is this problem too difficult to fix?

thanks,

-g.

--
blog.gmosx.com

[George Moschovitis]

Until the problem gets fixed, here is a workaround using Apache Commons FileUpload:

var JDiskFileItemFactory = Packages.org.apache.commons.fileupload.disk.DiskFileItemFactory,

    JServletFileUpload = Packages.org.apache.commons.fileupload.servlet.ServletFileUpload;

var jfactory = new JDiskFileItemFactory();

jfactory.setSizeThreshold(5000000);

var jupload = new JServletFileUpload(jfactory);

jupload.setSizeMax(2000000);

var ByteString = require("binary").ByteString,

    extension = require("file").extension,

    MIME_TYPES = require("jack/mime").MIME_TYPES;

/**

 * Parses a multipart request in memory using Apache Commons FileUpload.

 * Compatible with Google App Engine.

 * http://commons.apache.org/fileupload/index.html

 */

exports.parseMultipart = function(env) {

    var items = jupload.parseRequest(env["jack.servlet_request"]);

    var params = {};

    for (var item in Iterator(items.iterator())) {

        if (item.isFormField()) {

            params[String(item.getFieldName())] = String(item.getString());

        } else {

            var bs = new ByteString();

            bs._bytes = item.get();

            bs._offset = 0;

            bs._length = Number(bs._bytes.length);

                  

            var name = String(item.getName());                  

                  

            params[String(item.getFieldName())] = {

                name: name,

                type: MIME_TYPES[extension(name)],

                size: Number(item.getSize()),

                data: bs

            }

        }

    }

    return params;

}

This requires a minor patch in the servlet handler (save the java request in jack.servlet_request).

Moreover, I propose that the MethodOverride middleware should ignore requests with multipart/form-data encoding to avoid implicitly calling .POST() that may save big files to the disk (tempfiles).

regards,

George.

--
blog.gmosx.com