pushing the envelope

[Dave Dyer]

This is a bit of a frustration vent about the limitations of the platform, and the

difficulty of straying too far outside the envelope that codename1 has in mind

to support.    The overall task is to add a dictionary with definitions to my

app.  The raw form of the dictionary is 15 megabytes.  Compressed about

5 megabytes.  The natural representation as strings would therefore be in

the range of 30 megabytes in memory (assuming utf-16 is the underlying

representation of strings).   This seemed a bit large, so I schemed to

store the bulk in compressed form in memory, so the data flow would

be <compresed file> ->  <many short strings> -> <many short compressed strings>

Now for the surprise.  The piece of code that compresses short strings

runs 1000 x slower on real android hardware than it does on my desktop.

  public static byte[] zip(final String str) {
    if ((str == null) || (str.length() == 0)) {
      throw new IllegalArgumentException("Cannot zip null or empty string");
    }
 
    try (ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream()) {
      try (GZIPOutputStream gzipOutputStream = new GZIPOutputStream(byteArrayOutputStream)) {
        gzipOutputStream.write(str.getBytes(StandardCharsets.UTF_8));
      }
      byteArrayOutputStream.close();
      return byteArrayOutputStream.toByteArray();
    } catch(IOException e) {
      throw new RuntimeException("Failed to zip content", e);
    }
  }

I haven't determined yet where the 3 order of magnitude loss of efficiency is.

Oh well, so much for elegance - what if I just skip the compression and store

the strings in native form?  Surprise #2 is that without the compression, the

app just crashes without any notice of an "out of memory" condition, though

I'm pretty sure that's what happens.

[Shai Almog]

This is a limitation of mobile devices. They have far less RAM and despite everything ARM is still slower for most cases when compared to x86. On Android we just use the native tools since they're designed for Java so this is the same performance you would get from native Android code.

[Dave Dyer]

I more or less agree, but 1000x worse performance suggests either that something sloppier

than normal is going on, such as perhaps writing an explicit file in GzipOutputStream.

Also, if the app just crashes instead of (presumably) throwing an out of memory exception,

that's on you.  I call to your attention once again the issues I've filed about the garbage

collector crashing in circumstances like this.

[Shai Almog]

You can look at the code of GZIPOutputStream in Codename One. It's all pure Java and as far as.I can tell doesn't do that.

We didn't write it though.

Again, this is Androids subsystem. Not us. So your beef is with Google.

We don't garbage collect on Android. We don't throw exceptions. This is 100% Google.

[Dave Dyer]

We programmers are a lazy bunch; we only work on a problem until a "good enough" solution is achieved.  I suppose

crashing and blaming it on android is good enough in this circumstance.

I've got my own "good enough" solution, which involves bypassing creating all those java strings

in the first place, and compressing the raw ascii using a nifty compression utility called "smaz"

that I found on github.  This is a much better solution than the simple solution that didn't work, and

I never would have got there if the simple solution worked.  So thanks.