Storing array vs. csv string

[Arvind Satyanarayan]

Hi there,

We're looking to store high dimensional vector data in MongoDB. Each vector consists of approximately 2000 floats and we were originally storing this as a csv string. However, we recently converted these strings to arrays but noticed that doing so caused the physical size of the database files to balloon (going from 30GB to over 150GB for 15M such vectors). 

Is this expected behaviour? Are there any best practices that we could follow?

Thanks!

-Arvind

[Claudio Bisegni]

You could use binary field of bson and group all double togheter. you
need to make some search on that field.

Saluti
Claudio

Il giorno 03/set/2012, alle ore 23:11, Arvind Satyanarayan
<arvin...@gmail.com> ha scritto:

[Scott Hernandez]

As Claudio said, if you don't need to search on them then storing the
data as a single binary field is probably the most compact way.

Storing them as an array has the overhead of storing fields names as
strings of the ordinal position for each element plus the overhead of
the types. A raw binary serialization from your class/language might
be much more succinct, and you could always compress the field as
well.