Mgo gridfs won't read file (gobson issue)

[Benoît Larroque]

Hello everyone,

I am using mgo to serve files from a gridfs collection. Everything run
smoothly in my development environment. Every files are served up
blazingly fast. On the staging server though I was unable to get a
single file from my collection.

I'm using :
mgo : revno: 69
gobson : revno: 28

After digging into the issue I've found that my dev environment
doesn't use the same BinData format subtype. In dev the collection is
using the new one (0x00) in staging the old one (0x02). As can been
seen on staging, the data is received but not unmarshalled :

2011/07/26 12:29:01 Socket 0xf84001d630 to 10.0.0.2:27017: received
document: bson.M{"n":0, "files_id":"141687-cover-thumbnail-storeitem",
"data":bson.Binary{Kind:0x2, Data:[]byte{0xff, 0xd8, 0xff, 0xe0, 0x0,
0x10, 0x4a, 0x46, 0x49, 0x46, 0x0, 0x1, 0x1, 0x1, 0x0, 0x60, 0x0,
0x60, 0x0, 0x0, 0xff, 0xdb, 0x0, 0x43, [[........]], 0x7b, 0x4e, 0xe,
0x38, 0xa2, 0x8a, 0x51, 0xe3, 0x6a, 0xd9, 0x71, 0x6b, 0x2a, 0x32,
0xe5, 0x2, 0x7d, 0x1c, 0xc0, 0xfe, 0x42, 0x8a, 0x28, 0xa5, 0x14, 0xff,
0xd9}}, "_id":"L\xeb\u034d\xdb|<!\x8d\x00\xd3\x06"}
2011/07/26 12:29:01 Query 0xf8400c4310 document unmarshaled:
&mgo.gfsChunk{Id:"L\xeb\u034d\xdb|<!\x8d\x00\xd3\x06",
FilesId:"141687-cover-thumbnail-storeitem", N:0, Data:[]byte{}}

After modifiying line 299 in gobson/bson/decode.go from :
if b.Kind == 0x00 {
to :
if b.Kind == 0x00 || b.Kind == 0x02 {

I get unmarshalled data :

2011/07/26 12:40:24 Socket 0xf84001d630 to 10.0.0.2:27017: received
document: bson.M{"n":0, "files_id":"141687-cover-thumbnail-storeitem",
"data":[]byte{0xff, 0xd8, 0xff, 0xe0, 0x0, 0x10, 0x4a, 0x46, 0x49,
0x46, 0x0, 0x1, 0x1, 0x1, 0x0, 0x60, 0x0, 0x60, 0x0, 0x0, 0xff, 0xdb,
0x0, 0x43, [[..........]] 0x7b, 0x4e, 0xe, 0x38, 0xa2, 0x8a, 0x51,
0xe3, 0x6a, 0xd9, 0x71, 0x6b, 0x2a, 0x32, 0xe5, 0x2, 0x7d, 0x1c, 0xc0,
0xfe, 0x42, 0x8a, 0x28, 0xa5, 0x14, 0xff, 0xd9},
"_id":"L\xeb\u034d\xdb|<!\x8d\x00\xd3\x06"}
2011/07/26 12:40:24 Query 0xf8400c4310 document unmarshaled:
&mgo.gfsChunk{Id:"L\xeb\u034d\xdb|<!\x8d\x00\xd3\x06",
FilesId:"141687-cover-thumbnail-storeitem", N:0, Data:[]byte{0xff,
0xd8, 0xff, 0xe0, 0x0, 0x10, 0x4a, 0x46, 0x49, 0x46, 0x0, 0x1, 0x1,
0x1, 0x0, 0x60, 0x0, 0x60, 0x0, 0x0, 0xff, 0xdb, 0x0,
0x43,[[.......]]0x7b, 0x4e, 0xe, 0x38, 0xa2, 0x8a, 0x51, 0xe3, 0x6a,
0xd9, 0x71, 0x6b, 0x2a, 0x32, 0xe5, 0x2, 0x7d, 0x1c, 0xc0, 0xfe, 0x42,
0x8a, 0x28, 0xa5, 0x14, 0xff, 0xd9}}

And my program is able to serve data (which are images, and they seem
to display fine).

However this change seems to break the gobson_test test file, so it
might not be the good fix for the issue...
Any idea/remark on that ?

Best regards,

Benoit Larroque

[Gustavo Niemeyer]

Hi Benoît,

> I am using mgo to serve files from a gridfs collection. Everything run
> smoothly in my development environment. Every files are served up
> blazingly fast.

Oh, that's great to know.

> On the staging server though I was unable to get a
> single file from my collection.

(...)

> if b.Kind == 0x00 || b.Kind == 0x02 {

(...)

> However this change seems to break the gobson_test test file, so it
> might not be the good fix for the issue...
> Any idea/remark on that ?

That approach sounds correct, and the test is just not attempting to
decode the obsoleted binary format as it should. Let's integrate the
change.

To be honest, I was never quite sure about how to decode the obsoleted
binary data encoding format properly because the format looks just
wrong (which is likely why it was obsoleted).

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/plus
http://niemeyer.net/twitter
http://niemeyer.net/blog

-- I never filed a patent.