Saving a Marshalled object to Mongo: "Mongo::InvalidStringEncoding: String not valid UTF-8"

[Chris Hanks]

Hi -

I'm trying a store a Marshalled Ruby object using MongoMapper. One of
my MongoMapper models has a key of type String, and I simply assign
Marshal.dump(object) to that key and try to save it. When I try to
save, though, I get the error message:

Mongo::InvalidStringEncoding: String not valid UTF-8
from /home/chris/.gem/ruby/1.8.7/gems/mongo-0.18.2/lib/../lib/mongo/
util/bson_c.rb:5:in `serialize'
from /home/chris/.gem/ruby/1.8.7/gems/mongo-0.18.2/lib/../lib/mongo/
util/bson_c.rb:5:in `serialize'
from /home/chris/.gem/ruby/1.8.7/gems/mongo-0.18.2/lib/../lib/mongo/
collection.rb:253:in `update'
from /home/chris/.gem/ruby/1.8.7/gems/mongo-0.18.2/lib/../lib/mongo/
collection.rb:180:in `save'
from /home/chris/daiphe-mcat/vendor/plugins/mongomapper/lib/
mongo_mapper/document.rb:413:in `save_to_collection'

I've been able to do this successfully with smaller Marshalled ruby
objects, so I don't know why it encodes the strings correctly in some
cases but not others. Ruby identifies the Marshalled content as a
string object with length around 17,000, which is pretty large but
should be something Mongo could handle, right?

I've tried changing the key to type Binary, and I can save that way,
but when I try to load the object again I get an object of type
Mongo::Binary, and I'm not sure what to do with it. I've looked at it
in the Ruby Mongo driver, but I can't divine anything useful from it.

Anyone have any suggestions?

[Michael Dirolf]

You should definitely be using Binary instances. They are subclasses
of Mongo::ByteBuffer which has a to_s method defined. Calling that
will get you back your string.

> --
> You received this message because you are subscribed to the Google
> Groups "MongoMapper" group.
> For more options, visit this group at
> http://groups.google.com/group/mongomapper?hl=en?hl=en
>

[Jacques Crocker]

Hi Michael. I'm running into the same issue. Can you give an example of how to save the object using Mongo::ByteBuffer?

say I had `object = MyObject.new`. What would be the ByteBuffer equivalent of Marshal.dump(object)?

[Michael Dirolf]

Sorry I guess my last post wasn't clear. You should save instances of
Mongo::Binary as the OP tried. Then when you get them back you'll have
a Mongo::Binary instance. To get the string version of that just call
to_s on it.

Does that help?

[Jacques Crocker]

Ah, ok. So I can just call marshall dump to save it, then when I get the Mongo::Binary back, run to_s on it to grab the string that I then send into Marshal.load?

[Michael Dirolf]

Right. You should wrap the results of marshall dump as a Binary
instance before saving though (not sure if you just forgot to mention
that step).

All the wrapping does is just tell the BSON encoder to encode as
binary and not string data, so the data doesn't need to be valid
UTF-8.

[Jacques Crocker]

Cool, makes sense. Thanks for the help!

[Chris Hanks]

Hey Michael, thanks for your help - everything is working great now.

Chris

On Feb 10, 1:26 pm, Jacques Crocker <merbj...@gmail.com> wrote:
> Cool, makes sense. Thanks for the help!
>
> On Feb 10, 2010, at 1:24 PM, Michael Dirolf wrote:
>
>
>
> > Right. You should wrap the results of marshall dump as a Binary
> > instance before saving though (not sure if you just forgot to mention
> > that step).
>
> > All the wrapping does is just tell the BSON encoder to encode as
> > binary and not string data, so the data doesn't need to be valid
> > UTF-8.
>

> > On Wed, Feb 10, 2010 at 4:21 PM, Jacques Crocker <merbj...@gmail.com> wrote:
> >> Ah, ok. So I can just call marshall dump to save it, then when I get the Mongo::Binary back, run to_s on it to grab the string that I then send into Marshal.load?
>
> >> On Feb 10, 2010, at 1:15 PM, Michael Dirolf wrote:
>
> >>> Sorry I guess my last post wasn't clear. You should save instances of
> >>> Mongo::Binary as the OP tried. Then when you get them back you'll have
> >>> a Mongo::Binary instance. To get the string version of that just call
> >>> to_s on it.
>
> >>> Does that help?
>

[John Nunemaker]

There is a key type of Binary too. You can do key :thing, Binary and that should convert to binary on save if you pass a string.