[erlang-questions] The 2 GB limit
Igor Ribeiro Sucupira
Is there any plan (or work in progress) for removing the 2 GB size
limit of dets files?
And, while this is not accomplished, do you think it could be easier
to try to raise that limit to 4 GB? Assuming the issue is related to
32-bit addressing, I'm guessing (just guessing) that dealing with 4 GB
files should not be difficult. Am I wrong? Why?
Thanks.
Igor.
--
"The secret of joy in work is contained in one word - excellence. To
know how to do something well is to enjoy it." - Pearl S. Buck.
________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org
Masklinn
> Is there any plan (or work in progress) for removing the 2 GB size
> limit of dets files?
>
> And, while this is not accomplished, do you think it could be easier
> to try to raise that limit to 4 GB? Assuming the issue is related to
> 32-bit addressing, I'm guessing (just guessing) that dealing with 4 GB
> files should not be difficult. Am I wrong? Why?
Well if the dets files are merely mmapped when read, given Windows puts the kernel data in the application's address space by default the available memory really is only 2GB (can go to ~3GB using the right boot switch, which may or may not make the system less stable). Even under OSX I'm not sure the memory available to the application process reaches the full 4GB.
Valentin Micic
such a level of "data density" (assuming that 2 GB translates to at least a
few million records). As much as I believe that reading from such a big file
may be relatively fast, I think that updating the data would be
disproportionately slower. In my experience, the only way to maintain a good
performance with sizable data sets (say, 50-150 million records) in dets is
to distribute data over a number of dets files (similar to what mnesia
fragmentation does).
So, if my skepticism is justifiable, this is not a question of how difficult
it is to increase a dets file size limit, but would there be any point in
doing it?
V/
Ulf Wiger
Hash: SHA1
Igor Ribeiro Sucupira wrote:
> Hi.
>
> Is there any plan (or work in progress) for removing the 2 GB size
> limit of dets files?
>
> And, while this is not accomplished, do you think it could be easier
> to try to raise that limit to 4 GB? Assuming the issue is related to
> 32-bit addressing, I'm guessing (just guessing) that dealing with 4 GB
> files should not be difficult. Am I wrong? Why?
Rather than re-writing dets, I would go with something like Tokyo
Tyrant, build on a suitable erlang wrapper (like
http://github.com/mallipeddi/tora), and perhaps put something like
my sext library on top
(http://svn.ulf.wiger.net/sext/trunk/sext/examples/tt_proto.erl).
To match dets, I'd like to see some administrative code around the
creation and opening of Tyrant tables, including the ability to
"repair" the data from the transaction log. Seems doable, though.
http://1978th.net/tokyotyrant/
BR,
Ulf W
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAksL9nUACgkQtqqFieqzed1v7wCfUf9bgDU+JfD8Gey51lvR7fxG
3iIAn0J5+8NpHH7DkLTgRlRuUKWomkXO
=C66x
-----END PGP SIGNATURE-----
Igor Ribeiro Sucupira
> I must express my skepticism regarding dets performance when a file reaches
> such a level of "data density"
Hello.
Since I don't know very much about the inner workings of dets files,
maybe I am confused by the terminology: wouldn't data density be
inversely proportional to file size and directly proportional to data
size?
Thank you.
Igor.
Valentin Micic
is directly proportional to a number of the records.
OTOH, if you take a file size as a constant (as in 2GB), then number of
records is inversely proportional to the data size (as in record size).
I've used the term "data density" (for the lack of the better word) to
express a level of distribution of data among dets slots -- higher the
number of records per slot, higher the data density. In this context, the
value for data density is very much in a function of hashing algorithm used
to map a particular key value to a given slot. Assuming that hashing
algorithm maps things (relatively) evenly, thus the "data density" ends up
being function of number of records in dets file.
What I was trying to tell you is that updates on dets that contains high
number of records (and hence more records per dets slot, therefore higher
"data density") may be quite processing intensive, as probability of not
being able to host the resulting data within the memory allocated to the
particular slot is higher with a higher "data density".
With this in mind, I've been asking if increasing dets file beyond 2GB would
make any sense (because bigger file inevitably leads to a higher "data
density") -- it is far healthier to distribute the data across multiple dets
files and keep the "data density" per file at lower level for the same
amount of data.
My apologies for the confusion.
V/
Jarrod Roberson
HDF5.
I have used it in a couple of projects where I needed to be able to store
arbitrary sized files and file groups with metadata about them.
I have used it from Python, Java and Objective-C. It shouldn't be to hard to
"port" dets to use something like this library to store data in.
Igor Ribeiro Sucupira
I took a quick look at dets_v9.erl and I understand what you mean: as
you insert more records, you will have more and more data in each
slot.
But, from the documentation:
http://erlang.org/doc/man/dets.html
It seems you can tune the number of slots (see options: min_no_slots
and max_no_slots).
Today you have to create enough fragments so that none of them will
approach the 2 GB limit. If that limit was raised to, say, 4 GB, you
could in advance create your dets files with more slots, thus being
able to store more data without losing performance (and using less
fragments).
I have never tried to fine tune the number of slots in dets files
(just seen it in the docs), so maybe what I am saying causes other
performance problems (they do say something about it in the
documentation, but I'll have to see the source code to understand it).
Thank you.
Igor.
Valentin Micic
Yes, you can tune a number of slots -- I did try it once, but for whatever
reason it did not create any difference in my case (when the table grew to
around 400,000 records, performance deteriorated rapidly when inserting or
updating). It could be that the problem was elsewhere and that I did not
persevere in optimizing it -- it was so much easier to settle for mensia
fragmentation, which combined with an appropriate +A value made substantial
performance difference with disc_only_copy tables (dets).
I will be very glad if you arrive to a different conclusion regarding
min_no_slot testing.
Igor Ribeiro Sucupira
<ulf....@erlang-consulting.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Igor Ribeiro Sucupira wrote:
>> Hi.
>>
>> Is there any plan (or work in progress) for removing the 2 GB size
>> limit of dets files?
>>
>> And, while this is not accomplished, do you think it could be easier
>> to try to raise that limit to 4 GB? Assuming the issue is related to
>> 32-bit addressing, I'm guessing (just guessing) that dealing with 4 GB
>> files should not be difficult. Am I wrong? Why?
>
> Rather than re-writing dets, I would go with something like Tokyo
> Tyrant, build on a suitable erlang wrapper (like
Hi.
I'm running Mnesia with fragmented tables, over a pool of servers.
Considering the amount of records to be inserted and the size limits
of dets, I will need a lot of fragments for each table and there seems
to be a big performance penalty on using many Mnesia fragments (even
when they don't have so much data).
The more plausible replacement, then, would be something like tcerl
with mnesiaex, but I've read somewhere that mnesiaex is still
experimental.
I'm dealing with data that affects millions of users, in my current
project, so I have to be very careful. That was the reason for wanting
to raise the limits of dets tables.
Thank you.
Igor.
> http://github.com/mallipeddi/tora), and perhaps put something like
> my sext library on top
> (http://svn.ulf.wiger.net/sext/trunk/sext/examples/tt_proto.erl).
>
> To match dets, I'd like to see some administrative code around the
> creation and opening of Tyrant tables, including the ability to
> "repair" the data from the transaction log. Seems doable, though.
>
> http://1978th.net/tokyotyrant/
>
> BR,
> Ulf W
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAksL9nUACgkQtqqFieqzed1v7wCfUf9bgDU+JfD8Gey51lvR7fxG
> 3iIAn0J5+8NpHH7DkLTgRlRuUKWomkXO
> =C66x
> -----END PGP SIGNATURE-----
--
"The secret of joy in work is contained in one word - excellence. To
know how to do something well is to enjoy it." - Pearl S. Buck.
________________________________________________________________
Ulf Wiger
Mnesiaex is not supported by OTP. As far as I know,
it is stable.
There are some things to consider...
- If you put significantly more than 2GB into
each table, mnesia's method of synchronizing
replicas (copying the entire table over the
network) may start to hurt.
- Tokyo Cabinet has slightly different recovery
characteristics than dets. This may make a
difference in some crash recovery situations.
That is, TC gets its speed partly from making
use of memory-mapped IO, so more data is likely
to get lost in some failure situations. TC is
supposedly also more sensitive to corrupted files
than dets, since it doesn't 'repair' the file
like dets does.
I don't think there are major problems with performance
when using many fragments. You should perhaps do some
measurements first?
It can be a problem that you need to have enough
fragments that you are absolutely sure never to
reach 2 GB in any fragment, since mnesia has no
way of dealing with dets refusing to insert more
records.
BTW, if you use disc_copies (not disc_only_copies),
dets is not used for the persistent storage, and
there is no 2 GB limit per table (only the restriction
that the table must fit in RAM).
BR,
Ulf W
--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com
Igor Ribeiro Sucupira
<ulf....@erlang-consulting.com> wrote:
>
> I don't think there are major problems with performance
> when using many fragments. You should perhaps do some
> measurements first?
Sorry I was not clear enough. The performance penalty I observed was
more related to the number of fragments PER SERVER.
If I insert a total of 10 GB of data on 500 initially empty fragments,
distributed over 2 servers, and then repeat the experiment with 1000
initially empty fragments (without changing the number of servers nor
the total amount of data), the insertion rate will be smaller (I don't
have the data here, but I think it was something around 20% smaller on
my experiments).
Other people in my company have conducted similar experiments (for
completely independent projects) and have arrived at the same general
conclusion: try to keep the number of fragments per server as small as
possible.
> It can be a problem that you need to have enough
> fragments that you are absolutely sure never to
> reach 2 GB in any fragment, since mnesia has no
> way of dealing with dets refusing to insert more
> records.
I'm aware of that. I need to create the pool with enough fragments
that I will not need to add more in the future (adding fragments in
that situation will certainly be a very heavy operation), but, at the
same time, I can't just "start with 100 thousand fragments" (:-)), as
I thought in the beginning.
That is an important trade-off you have to consider when deciding to
use Mnesia's fragmented tables in disc_only_copies.
> BTW, if you use disc_copies (not disc_only_copies),
> dets is not used for the persistent storage, and
> there is no 2 GB limit per table (only the restriction
> that the table must fit in RAM).
I've read about that. But having to keep all my data in RAM would
increase so much the cost of the project that it would be much wiser
to give up Mnesia.
Thank you.
Igor.
--
"The secret of joy in work is contained in one word - excellence. To
know how to do something well is to enjoy it." - Pearl S. Buck.
________________________________________________________________
Ulf Wiger
>
> Sorry I was not clear enough. The performance penalty I observed was
> more related to the number of fragments PER SERVER.
> If I insert a total of 10 GB of data on 500 initially empty fragments,
> distributed over 2 servers, and then repeat the experiment with 1000
> initially empty fragments (without changing the number of servers nor
> the total amount of data), the insertion rate will be smaller (I don't
> have the data here, but I think it was something around 20% smaller on
> my experiments).
Oh, 20% seems pretty good to me. :)
Adding lots of local fragments ought to affect both
the hit ratio of the disk cache and the amount of disk
seeks, as both the locality and overhead per object get
worse.
But whether that's an acceptable penalty for something
that's essentially a workaround is of course application
specific.
BR,
Ulf W
--
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com
________________________________________________________________