Gridfs & atomicity

[Gambitg]

The document says: Atomic implies that there is an all-or-nothing
semantic to the update; this is not possible with more than one
document.

Gridfs stores the metadata in fs.files and the binary data in
fs.chunks.
How does atomicity work in this case ?

1. So is it possible that in 'kill -9' or similar unclean shutdown,
only the fs.files is updated and not fs.chunks or vice-versa ? Also is
it possible not all the fs.chunks related to a single binary object
are written ?

2. Incase of replica-sets, is it not possible that some binary objects
can get corrupted then. Scenario : one of fs.chunks document or the
fs.files document did not make to majority of server before the
primary crashed. When the original primary comes up again as a
secondary, those documents are rolled-back (to sync with the current
primary). Now you have a case of some binaries being corrupted for
ever, right ?

Thanks.

[Gambitg]

[Kyle Banker]

Notes below:

> Gridfs stores the metadata in fs.files and the binary data in
> fs.chunks.
> How does atomicity work in this case ?

It doesn't. This is a file store, not a file system.

> 1. So is it possible that in 'kill -9' or similar unclean shutdown,
> only the fs.files is updated and not fs.chunks or vice-versa ? Also is
> it possible not all the fs.chunks related to a single binary object
> are written ?

That's certainly possible. Chunks are written first, followed by the
file object.

> 2. Incase of replica-sets, is it not possible that some binary objects
> can get corrupted then. Scenario : one of fs.chunks document or the
> fs.files document did not make to majority of server before the
> primary crashed. When the original primary comes up again as a
> secondary, those documents are rolled-back (to sync with the current
> primary). Now you have a case of some binaries being corrupted for
> ever, right ?

Yes, this is certainly possible.

> Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[Gambitg]

Thanks.
It might be better to update the documentation (maybe
http://www.mongodb.org/display/DOCS/When+to+use+GridFS ) if folks want
to decide between single binary document vs GridFS. Single binary
document for <16MB would be more robust as it will be atomic.

[Kyle Banker]

Done. Thanks for the suggestion.

[Voltron]

Speaking of 1.8, ETA?

On Jan 24, 9:19 pm, Kyle Banker <k...@10gen.com> wrote:
> Done. Thanks for the suggestion.
>
>
>
>
>
>
>
> On Mon, Jan 24, 2011 at 3:11 PM, Gambitg <balkrishna.sha...@gmail.com> wrote:
> > Thanks.
> > It might be better to update the documentation (maybe

> >http://www.mongodb.org/display/DOCS/When+to+use+GridFS) if folks want

[Stephen Yau]

Hi all,

> That's certainly possible. Chunks are written first, followed by the
> file object.

Is it possible than some Chunks will be orphans? i.e. it does not link
to any file object. Will it fix automatically during server restart or
require repair DB to fix the issue?

[Kyle Banker]

Yes, in rare situation, you can get orphaned chunks. A restart or
repair won't fix this because as far as the
database is concerned, orphaned chunks are just another document.
GridFS is a specification for writing files
to MongoDB, but it's all implemented on the driver level, not on the
server level. See the spec for details:
http://www.mongodb.org/display/DOCS/GridFS+Specification