Issue 256 in protobuf: Error Converting Message to array of Bytes (When negative Enum) - Java
prot...@googlecode.com
Owner: ken...@google.com
Labels: Type-Defect Priority-Medium
New issue 256 by bm_...@yahoo.com.au: Error Converting Message to array of
Bytes (When negative Enum) - Java
http://code.google.com/p/protobuf/issues/detail?id=256
What steps will reproduce the problem?
1.Create Proto Definition with Negate Enum in it
2.Create Message in Java
3.
What is the expected output? What do you see instead?
Message Converted to array of Bytes
What version of the product are you using? On what operating system?
2.4.0a Occurs both on Linux & Windows. This worked in
prior versions of Protocol Buffers but not in 2.4.0a
Please provide any additional information below.
Cause
The Compute RawVarint32Size calculates a length of 5 for a negative Enum
while the write will write 10 bytes
/**
* Compute the number of bytes that would be needed to encode an enum
field.
* Caller is responsible for converting the enum value to its numeric
value.
*/
public static int computeEnumSizeNoTag(final int value) {
return computeRawVarint32Size(value);
}
It should be:
public static int computeEnumSizeNoTag(final int value) {
return computeInt32SizeNoTag(value);
}
the write uses Int32NoTag; The Write writes 10 bytes
/**
* Write an enum field to the stream. Caller is responsible
* for converting the enum value to its numeric value.
*/
public void writeEnumNoTag(final int value) throws IOException {
writeInt32NoTag(value);
}
Trace:
com.google.protobuf.CodedOutputStream$OutOfSpaceException:
CodedOutputStream was writing to a flat byte array and ran out of space.
at
com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:798)
at
com.google.protobuf.CodedOutputStream.writeRawByte(CodedOutputStream.java:862)
at
com.google.protobuf.CodedOutputStream.writeRawByte(CodedOutputStream.java:870)
at
com.google.protobuf.CodedOutputStream.writeRawVarint32(CodedOutputStream.java:982)
at
com.google.protobuf.CodedOutputStream.writeInt32NoTag(CodedOutputStream.java:328)
at com.google.protobuf.FieldSet.writeElementNoTag(FieldSet.java:602)
at com.google.protobuf.FieldSet.writeElement(FieldSet.java:579)
at com.google.protobuf.FieldSet.writeField(FieldSet.java:649)
at com.google.protobuf.FieldSet.writeTo(FieldSet.java:523)
at com.google.protobuf.DynamicMessage.writeTo(DynamicMessage.java:210)
at
com.google.protobuf.CodedOutputStream.writeMessageNoTag(CodedOutputStream.java:380)
at com.google.protobuf.FieldSet.writeElementNoTag(FieldSet.java:608)
at com.google.protobuf.FieldSet.writeElement(FieldSet.java:579)
at com.google.protobuf.FieldSet.writeField(FieldSet.java:645)
at com.google.protobuf.FieldSet.writeTo(FieldSet.java:523)
at com.google.protobuf.DynamicMessage.writeTo(DynamicMessage.java:210)
at
com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:64)
at
net.sf.RecordEditor.ProtoBuf.JRecord.Def.ProtoLine.getData(ProtoLine.java:141)
at net.sf.JRecord.IO.LineWriterWrapper.write(LineWriterWrapper.java:51)
at net.sf.RecordEditor.edit.file.FileView.writeToFile(FileView.java:2183)
at net.sf.RecordEditor.edit.file.FileView.saveFile(FileView.java:2162)
at
net.sf.RecordEditor.edit.file.FileView.writeLinesToFile(FileView.java:2096)
at net.sf.RecordEditor.edit.file.FileView.writeFile(FileView.java:2058)
at net.sf.RecordEditor.edit.display.util.SaveAs.saveFile(SaveAs.java:299)
at
net.sf.RecordEditor.edit.display.util.SaveAs.actionPerformed(SaveAs.java:254)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
at
javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
at
javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at
javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
at java.awt.Component.processMouseEvent(Component.java:6263)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3255)
at java.awt.Component.processEvent(Component.java:6028)
at java.awt.Container.processEvent(Container.java:2041)
at java.awt.Component.dispatchEventImpl(Component.java:4630)
at java.awt.Container.dispatchEventImpl(Container.java:2099)
at java.awt.Component.dispatchEvent(Component.java:4460)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4574)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4238)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4168)
at java.awt.Container.dispatchEventImpl(Container.java:2085)
at java.awt.Window.dispatchEventImpl(Window.java:2475)
at java.awt.Component.dispatchEvent(Component.java:4460)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
at
java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
at
java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
at
java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
Trace:
Attachments:
StoreSales6.proto 1.0 KB
protoStoreSales6.bin 15.1 KB
prot...@googlecode.com
Comment #1 on issue 256 by bm_...@yahoo.com.au: Error Converting Message to
http://code.google.com/p/protobuf/issues/detail?id=256
Should start
What steps will reproduce the problem?
1.Create Proto Definition with Negate Enum in it and
2.Create DynamicMessage in Java using Proto
3.Convert to byte array
Attached are the Proto Definition and File (Created with previous version).
I was using Dynamic messages to process the file.
prot...@googlecode.com
Status: Accepted
Owner: jas...@google.com
Comment #2 on issue 256 by jas...@google.com: Error Converting Message to
array of Bytes (When negative Enum) - Java
http://code.google.com/p/protobuf/issues/detail?id=256
I'll take this one; I think I wrote the offending code. Will try to get a
patch up soon.
prot...@googlecode.com
Status: Fixed
Comment #3 on issue 256 by jas...@google.com: Error Converting Message to
array of Bytes (When negative Enum) - Java
http://code.google.com/p/protobuf/issues/detail?id=256
Fixed at r381
(Aside: I believe this bug has been in the last few releases. Did this
really only show up with 2.4.0a?)
prot...@googlecode.com
Comment #4 on issue 256 by bm_...@yahoo.com.au: Error Converting Message to
http://code.google.com/p/protobuf/issues/detail?id=256
You are probably right about the bug being in the last few releases, I
think the last version I downloaded was 2.3 but it could of been 2.2
prot...@googlecode.com
Comment #5 on issue 256 by gus...@veide.se: Error Converting Message to
http://code.google.com/p/protobuf/issues/detail?id=256
What a coincidence. I ran into this error today when upgrading from
protobuf 2.0.3 to 2.4.0a and a patch just came out :) Lucky me...
Will there be a patch release with this fix or should I patch it manually?
prot...@googlecode.com
Cc: liuj...@google.com
Comment #6 on issue 256 by jas...@google.com: Error Converting Message to
array of Bytes (When negative Enum) - Java
http://code.google.com/p/protobuf/issues/detail?id=256
+liujisi, who's been doing the releases. I don't have the bandwidth (or
know how!) to do a point release - Jisi, think it's worth doing one for
this issue?
You'll probably want to patch in manually: even if we do go through a
release it will take some time to get out.
OP: I thought this issue sounded familiar, and did a little more digging.
It turns out that a similar issue was reported in Issue 200 and fixed in
r354, which was included in 2.4.0. That fix changed the way negative enums
are written but not the way their size is computed - the inconsistency
causes the exception, and is indeed new to 2.4.0. Sorry to have doubted
you :-)
prot...@googlecode.com
Comment #7 on issue 256 by liuj...@google.com: Error Converting Message to
http://code.google.com/p/protobuf/issues/detail?id=256
Yes, but we will probably wait for a couple of days in case we discover
other bugs shortly.
prot...@googlecode.com
Comment #8 on issue 256 by jhan...@gmail.com: Error Converting Message to
http://code.google.com/p/protobuf/issues/detail?id=256
Any update on this? I'm running into this exception running version 2.4.1
Is there any way from the public API to set the underlying buffer size of
the CodedOutputStream? We are not creating that directly. We are calling
writeDelimitedTo()
Any help is appreciated.
prot...@googlecode.com
Labels: FixedIn-2.4.1
Comment #9 on issue 256 by jas...@google.com: Error Converting Message to
array of Bytes (When negative Enum) - Java
http://code.google.com/p/protobuf/issues/detail?id=256
2.4.1 should have contained this fix. Can you post a reproduction? If your
serialization does not have a negative enum value, then please open a
separate issue - the exception can occur generally when the ByteSize
computation does not match the size of data written by the serializer.
There is no message level interface, but you can just create the
CodedOutputStream around your OutputStream directly:
http://code.google.com/apis/protocolbuffers/docs/reference/java/com/google/protobuf/CodedOutputStream.html#newInstance(java.io.OutputStream,
int)
However, I think you should only be getting the OutOfSpaceException if
you're using the byte[] version of the methods, in which case the data is
written directly to the array. You need to make sure that the byte array
you provide is sufficiently large.
prot...@googlecode.com
Comment #10 on issue 256 by jhan...@gmail.com: Error Converting Message to
http://code.google.com/p/protobuf/issues/detail?id=256
sorry, yeah, I'm getting this same exception but simply because the
underlying CodedOutputStream default buffer size is not large enough.
We are using writeDelimitedTo() and not write(), and unfortunately, while
write() accepts a CodedOutputStream, writeDelimitedTo() does not, so I'm
unable to figure out how to alter its buffer size.
prot...@googlecode.com
Comment #11 on issue 256 by jas...@google.com: Error Converting Message to
http://code.google.com/p/protobuf/issues/detail?id=256
The internal buffer for CodedOutputStream should just be a performance
optimization; it's an internal buffer so that CodedOutputStream doesn't
make a bunch of small writes to an OutputStream, which might be slow.
I still don't see how it is possible to get an OutOfSpaceException if you
are using writeDelimitedTo; an OutputStream must be provided for this API,
but the OutOfSpaceException is only ever thrown if the CodedOutputStream
has a null OutputStream:
http://code.google.com/p/protobuf/source/browse/trunk/java/src/main/java/com/google/protobuf/CodedOutputStream.java#795
Can you start a new thread in the protobuf mailing list, with a small,
self-contained reproduction of your problem?