Support for J2ME

[Igor Gatis]

Hi protobuf team,

Have you guys considered supporting J2ME?

I've extended protoc to generate J2ME compatible code and built its supporting J2ME runtime library. I was wondering if you guys would consider reviewing my changes.

Thanks,

-Gatis

[Kenton Varda]

There are three J2ME implementations on the third-party list:

  http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns

I'd be happy to list yours as well if you'd like.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[Igor Gatis]

So it is totally out of the scope of this project to add such support?

The changes I've made seem to match with the current architecture. I've added extended protoc to accept a new file option java_platform (where user may specify J2SE(default) or J2ME) which makes it produce J2ME compatible sources.

[Kenton Varda]

Is your implementation a modification of the official Java implementation?  If so, how much is shared?  Could you send me a diff via codereview.appspot.com?

[Igor Gatis]

It is a stripped copy of the official Java implementation. Here are the list of files:

BlockingRpcChannel.java

BlockingService.java

ByteString.java

CodedInputStream.java

CodedOutputStream.java

InvalidProtocolBufferException.java

Message.java

RpcCallback.java

RpcChannel.java

RpcController.java

ServiceException.java

WireFormat.java

Limitations:

1) Biggest limitation is the lack of support for extensions. I plan to fix it soon. 

2) Also, for J2ME devices, (asynchronous) Services is not a huge requirement, so it is not there too (notice there is no Service.java in this list).

As for the compiler, there is A LOT of shared code.  

I'm sending you a code review.

Thanks,

-Gatis

[Igor Gatis]

Here is the code review: http://codereview.appspot.com/445041/show

[Kenton Varda]

Can you explain some of the design decisions:

- Why did you eliminate the builder pattern?

- Why did you choose to implement generic services when this feature is deprecated anyway?

Your diff shows the runtime classes as if they were completely new.  Could you re-arrange the files so that it actually compares your versions of these classes against the official ones?

=====

As far as making this part of the official project, my worry is about maintenance challenges.  It will be much harder for us to make changes to the code generator if we always have to consider J2ME.  For example, when coupled changes to the code generator and runtime library, unless the change is in a part that doesn't apply to J2ME, we would now have to update the J2ME library too.  Not only does this require modifying more files, but it requires setting up J2ME tools for testing -- and our internal build system does not have any support for this AFAIK.

So my thought is that the best way to run this is as a branch.  Changes to the official implementation would not affect J2ME until the J2ME branch maintainers choose to update it, and then they could make sure those changes make sense.

[Igor Gatis]

- Why did you eliminate the builder pattern?

To save jar space. J2ME environment is pretty restricted. Many devices have a few kilo bytes size limit (e.g 128K, 256K). An empty class adds about 200 bytes to jar file. The long time I worked with J2ME convinced me the less classes the better. I tried to keep it as simple as possible without hurting the OO design.

- Why did you choose to implement generic services when this feature is deprecated anyway?

That was overseen. I need to fix that then.

Your diff shows the runtime classes as if they were completely new.  Could you re-arrange the files so that it actually compares your versions of these classes against the official ones?

The some official java sources are not Java 1.3 compatible (for instance, they use generics and for-each loops). I've also flattened some of the classes hierarchy (e.g. Message vs AbstractMessage + GeneratedMessage + MessageLite, etc) to save jar space. I've also removed dependency to Descriptors so I had to change  BlockingRpcChannel and RpcChannel to receive the method as a String rather than a descriptor. So that being said, the new set of files are tailored to J2ME environment and its limitations.

So the bottom line is that I had to squeeze the runtime to get it as small as possible - this is a fully functional protobuf runtime implementation that occupies 26KB against 173KB of standard Java implementation.

Does that make sense?

[Igor Gatis]

What exactly is deprecated in the services API? Is it only the Service and BlockingService interfaces?

[Kenton Varda]

On Thu, Mar 11, 2010 at 1:52 PM, Igor Gatis <igor...@gmail.com> wrote:

- Why did you eliminate the builder pattern?

To save jar space. J2ME environment is pretty restricted. Many devices have a few kilo bytes size limit (e.g 128K, 256K). An empty class adds about 200 bytes to jar file. The long time I worked with J2ME convinced me the less classes the better. I tried to keep it as simple as possible without hurting the OO design.

Hmm, I'm surprised that you could even get anything resembling the Java protobuf implementation to fit in such a small space.  You could possibly get it even smaller by using POJOs (avoid getters and setters).

One thing that worries me about removing the builder pattern is that it changes the whole usage model in a way that could affect other design decisions.  For example, if you have a message-typed field which has not been set yet, the getter for that field should return the message type's default instance.  This was fine when default instances were immutable, but now that they are mutable, you might have bugs where someone writes:

  myMessage.getSubMessage().setFoo(1);

If they haven't previously called setSubMessage(new SubMessage()) then this code will actually modify the shared default instance of SubMessage which could cause all sorts of bugs around the system.  Have you considered how to avoid this problem?

 

- Why did you choose to implement generic services when this feature is deprecated anyway?

That was overseen. I need to fix that then.

Your diff shows the runtime classes as if they were completely new.  Could you re-arrange the files so that it actually compares your versions of these classes against the official ones?

The some official java sources are not Java 1.3 compatible (for instance, they use generics and for-each loops). I've also flattened some of the classes hierarchy (e.g. Message vs AbstractMessage + GeneratedMessage + MessageLite, etc) to save jar space. I've also removed dependency to Descriptors so I had to change  BlockingRpcChannel and RpcChannel to receive the method as a String rather than a descriptor. So that being said, the new set of files are tailored to J2ME environment and its limitations.

So the bottom line is that I had to squeeze the runtime to get it as small as possible - this is a fully functional protobuf runtime implementation that occupies 26KB against 173KB of standard Java implementation.

Did you start from the lite implementation or the full one?

26k is pretty impressive.

On Thu, Mar 11, 2010 at 4:26 PM, Igor Gatis <igor...@gmail.com> wrote:

What exactly is deprecated in the services API? Is it only the Service and BlockingService interfaces?

We are now encouraging RPC implementations to provide their own code generator plugins rather than work with the "generic" services.  This way implementations can tailor the generated code to their purposes.

Note that services were never part of the "lite" implementation.

[Igor Gatis]

  myMessage.getSubMessage().setFoo(1);

If they haven't previously called setSubMessage(new SubMessage()) then this code will actually modify the shared default instance of SubMessage which could cause all sorts of bugs around the system.  Have you considered how to avoid this problem?

Uh, not really. But yeah, that's definitely a problem. Let me think about that. I'll get back to you when I have a solution for this problem. 

 

So the bottom line is that I had to squeeze the runtime to get it as small as possible - this is a fully functional protobuf runtime implementation that occupies 26KB against 173KB of standard Java implementation.

Did you start from the lite implementation or the full one?

26k is pretty impressive.

Yep, I though LITE_RUNTIME would solve all my problems but generics and for-each loops were a problem.

Flattening the Message class has a few side effects I haven't noticed until I implemented equals() and hashCode() methods.

A bad effect is that now that there is no way to handle fields in a generic way, these methods are generated for each message, which makes the size per message bigger. For a small set of messages, that's not a problem - and that is the most common scenario for J2ME apps. 

A good one is that obfuscators usually removed unused methods. If user never uses equals or hashCode, they will be selectively removed. That was not possible when AbstractMessage was there.

BTW, I removed the generic services generation too.

[Igor Gatis]

I think I have a solution for the readonly messages.

Message.java now includes the following header:

public abstract class Message {

private boolean readOnly;

protected Message(boolean noInit) {

this.readOnly = true;

}

public Message() {}

protected void assertNotReadOnly() {

if (readOnly) {

throw new RuntimeException("Read only message!");

}

}

A generated message, HelloRequest, has:

public  final class HelloRequest extends com.google.protobuf.Message {

  public HelloRequest() {

    initFields();

  }

  private HelloRequest(boolean noInit) { super(true); }

All methods that can modify HelloRequest look like this:

  public void setName(java.lang.String value) {

    assertNotReadOnly();

    hasName = true;

    name_ = value;

  }

In other words, that example:

myMessage.getSubMessage().setFoo(1);

Would through an exception. just like Collections.unmodifiableList(java.util.List) does.

Do you think this solves the problem?

[Kenton Varda]

This may solve the problem but adding code to every setter may have a significant cost.  It's harder to inline the setter this way.  But it's hard to say exactly what the cost will be without some sort of benchmarks.

[Igor Gatis]

My experience with J2ME says performance is not the most important feature for the majority of the applications. Trust me when I say JAR size is the one people care the most.

Besides, the extra cost is a method call + condition check, that's fairly cheap. Depending on the compiler and/or obfuscator, that condition could even be inlined saving the extra method call.

I still need to go over all files to check code compliance with the style guide and write tests. But besides that, do you think I still need to change the design? 

[Igor Gatis]

I've uploaded my latest version.

http://codereview.appspot.com/445041/show

[Kenton Varda]

I wonder if we should consider making mutable-message mode a feature of the base implementation, rather than j2me-specific.  Some people have requested this, even though I personally think the builder-based design is better.

On Fri, Mar 12, 2010 at 7:13 PM, Igor Gatis <igor...@gmail.com> wrote:

My experience with J2ME says performance is not the most important feature for the majority of the applications. Trust me when I say JAR size is the one people care the most.

OK, you would know better than me.

 

I still need to go over all files to check code compliance with the style guide and write tests. But besides that, do you think I still need to change the design? 

Based on my high-level look, what you have seems good.  But, you don't need my approval for anything.

Are you still hoping to add this to the official distribution?  If so, can I ask why you prefer this over maintaining your own project?  It seems to me that combining them would only be more work for both of us.  I would need you to be available to update and test your implementation any time I'm trying to do a release or making Java code generator changes.  Also, you would not be able to release updates to your implementation separately from the protobuf release cycle, which is probably far too slow for a new project.  With the decentralized approach, you do not have to depend on me and I do not have to depend on you.

[Igor Gatis]

Hi Kenton,

It seems to me that combining them would only be more work for both of us.  I would need you to be available...

I haven't thought the whole thing through. You're right. It makes more sense to keep my implementation in a separated project. Thanks for all the advices.

Thanks,

-Gatis

[Kenton Varda]

Great!  I look forward to seeing it.