[erlang-questions] (not) understanding mnesia transactions
Seth Falcon
I'm trying to use mnesia to store records where I associate a unique
integer for a given string. I'm using mnesia:dirty_update_counter/3
to obtain a sequence of integers. Inside a transaction, I'm checking
to see if the given string is already in the table. If so, return
what is there; if not, get the next integer and store it.
Here's my get_code function:
get_code(Url) ->
F = fun() ->
Q = qlc:q([X || X <- mnesia:table(url), X#url.url =:= Url ]),
case qlc:e(Q) of
[] ->
New = #url{url = Url, code = next_int()},
mnesia:write(New),
New;
[Found] ->
Found
end
end,
{atomic, Val} = mnesia:transaction(F),
Val.
So far so good. Since the check for exists and the write are done
within a single transaction, I'm expecting that I can have concurrent
calls for the same string and that mnesia will serialize the requests
for me.
When I write a test case that has 5 processes making identical
requests, I end up with the same string being mapped to different
integers in the table. Each process in my test does
lists:map(fun get_code/1, lists:seq(1, 10))
And I'm expecting the return values from all processes to match. But
they do not. Can anyone point me in the right direction?
Below is a more complete example that demonstrates what I'm seeing.
You can run it as:
erlc huh.erl
mkdir testdb
erl -mnesia dir '"testdb"'
> huh:test().
%% -- huh.erl
-module(huh).
-export([get_code/1, get_url/1, start/0, stop/0, test/0]).
-record(counter, {id = 0, ver = 1}).
-record(url, {code, url}).
-include_lib("stdlib/include/qlc.hrl").
start() ->
Nodes = [node()],
mnesia:create_schema(Nodes),
mnesia:start(),
mnesia:create_table(url, [{disc_copies, Nodes},
{attributes, record_info(fields, url)},
{index, [url]}]),
mnesia:create_table(counter, [{disc_copies, Nodes},
{attributes, record_info(fields, counter)}]),
mnesia:wait_for_tables([url, counter], 20000).
stop() ->
mnesia:stop().
get_code(Url) ->
F = fun() ->
Q = qlc:q([X || X <- mnesia:table(url), X#url.url =:= Url ]),
Rec = qlc:e(Q),
%%case mnesia:index_read(url, Url, #url.url) of
case Rec of
[] ->
New = #url{url = Url, code = next_int()},
mnesia:write(New),
New;
[Found] ->
Found
end
end,
{atomic, Val} = mnesia:transaction(F),
Val.
get_url(Code) ->
F = fun() ->
Q = qlc:q([X || X <- mnesia:table(url), X#url.code =:= Code ]),
Recs = qlc:e(Q),
case Recs of
[] ->
undefined;
[Rec] ->
Rec
end
end,
{atomic, Val} = mnesia:transaction(F),
Val.
%% @spec next_int() -> integer()
%% @doc This is the integer sequence used to uniquely identify URLs
next_int() ->
mnesia:dirty_update_counter(counter, id, 1).
simple_gather(Pid) ->
receive
{Pid, Val} ->
Val
end.
test() ->
start(),
NumClients = 5,
Seq = lists:map(fun erlang:integer_to_list/1, lists:seq(1, 10)),
Parent = self(),
F = fun() ->
Codes = lists:map(fun(N) ->
get_code("http://" ++ N)
end,
Seq),
Parent ! {self(), Codes}
end,
Pids = lists:map(fun(_X) -> spawn(F) end, lists:seq(1, NumClients)),
%% Results = gather(Pids),
Results = [ simple_gather(Pid) || Pid <- Pids ],
io:format("PIDS: ~p~n", [Pids]),
io:format("Results ~p~n", [Results]),
Codes = [ X#url.code || X <- hd(Results) ],
io:format("Expecting ~p~n", [lists:seq(1, 10)]),
io:format("Got codes ~p~n", [Codes]),
ok.
%% -- end huh.erl
________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org
Chandru
Your code works as you expect it to if you first acquire a write lock on the
table before you read from it.
get_code(Url) ->
F = fun() ->
mnesia:lock({table, url}, write), %% <--------------------------
Q = qlc:q([X || X <- mnesia:table(url), X#url.url =:= Url ]),
Rec = qlc:e(Q),
%% Rec = mnesia:index_read(url, Url, #url.url),
case Rec of
[] ->
New = #url{url = Url, code = next_int()},
mnesia:write(New),
New;
[Found] ->
Found
end
end,
{atomic, Val} = mnesia:transaction(F),
Val.
What I think is happening is that a read lock is acquired when you first
read from the table. According to the mnesia manual:
"Read locks may be shared, which means that if one transaction manages to
acquire a read lock on an item, other transactions may also acquire a read
lock on the same item. However, if someone has a read lock no one can
acquire a write lock at the same item. If some one has a write lock no one
can acquire a read lock nor a write lock at the same item."
So all your clients acquire read locks, decide that a record for a URL does
not exist, and create a record. I'm not sure what locks are acquired for
records which do not exist. I'm sure the answer lies in reading the mnesia
source code...
cheers
Chandru
2009/6/17 Seth Falcon <se...@userprimary.net>
Seth Falcon
Thanks for the explanation.
* On 2009-06-17 at 23:57 +0100 Chandru wrote:
> Your code works as you expect it to if you first acquire a write
> lock on the table before you read from it.
Yes, that seems to fix it.
> So all your clients acquire read locks, decide that a record for a URL does
> not exist, and create a record. I'm not sure what locks are acquired for
> records which do not exist. I'm sure the answer lies in reading the mnesia
> source code...
Makes some sense. I guess the big piece that I was missing is that
transactions can run concurrently and the programmer needs to manage
the locking. A more subtle point, is that the locking mechanism is
such that once a lock is aquired it is kept for the remainder of the
transaction (I think).
So in the following example from the mnesia man page, the call to
mnesia:wread has the same effect as calling mnesia:lock and then
mnesia:read.
raise(Name, Amount) ->
mnesia:transaction(fun() ->
case mnesia:wread({person, Name}) of
[P] ->
Salary = Amount + P#person.salary,
P2 = P#person{salary = Salary},
mnesia:write(P2);
_ ->
mnesia:abort("No such person")
end
end).
Dan Gudmundsson
If you remove the index the code works as expected.
Except for io:format("Expecting ~p~n", [lists:seq(1, 10)]), which is wrong,
since you use dirty_update_counter which will be invoked
several times when there is a lock conflict on the write lock.
But back to the index lock problem.
Whether you use qlc or index_read in that query the same code will be invoked.
mnesia:index_read grabs a read lock on the rows it hits.
In your case when 5 parallel processes read the index for "url-1"
none is found, no locks are grabbed.
You then generate a unique key for each process, and write the record down,
that will succeed since they have unique keys.
This is bug in mnesia, 12 years old :-(
/Dan
Seth Falcon
> The problem here lies in how mnesia handles indexes and locks on indexes.
>
> If you remove the index the code works as expected.
I'll take working code over fast but wrong code :-)
But I'm thinking that adding the locking is the way to go since
without an index the look up is a full table scan.
> Except for io:format("Expecting ~p~n", [lists:seq(1, 10)]), which is wrong,
> since you use dirty_update_counter which will be invoked
> several times when there is a lock conflict on the write lock.
Oh, right. Since I'm already going to lock the url table for the
index read and record write, perhaps there isn't much additional value
in using dirty_update_counter here? If instead, I also locked the
counter table and did the counter increment "by hand", then repeated
calls due to failed transactions (lock conflict) would not lose
counter values. Does that sound right?
> But back to the index lock problem.
>
> Whether you use qlc or index_read in that query the same code will be invoked.
>
> mnesia:index_read grabs a read lock on the rows it hits.
> In your case when 5 parallel processes read the index for "url-1"
> none is found, no locks are grabbed.
>
> You then generate a unique key for each process, and write the record down,
> that will succeed since they have unique keys.
>
> This is bug in mnesia, 12 years old :-(
Interesting. I was surprised not to find a LockType option for
index_read after seeing the option for other read operations. Anyhow,
thanks for the additional explanation.
+ seth
Scott Lystig Fritchie
sf> * On 2009-06-18 at 12:05 +0200 Dan Gudmundsson wrote:
>> The problem here lies in how mnesia handles indexes and locks on
>> indexes.
>>
>> If you remove the index the code works as expected.
sf> I'll take working code over fast but wrong code :-)
sf> But I'm thinking that adding the locking is the way to go since
sf> without an index the look up is a full table scan.
I have a comment that may or may not apply to Seth, but it's worth
noting by all Mnesia users...
... Mnesia secondary indexes are *not ordered*. Queries that require
anything but an exact key lookup in a secondary index will instead scan
the entire secondary index. Which, if memory serves correctly, will
take a table lock. See the top of mnesia:info() (or related) to watch
locking behavior, if reading code isn't fun enough :-).
-Scott
Seth Falcon
I'm hoping to get a bit more clarification on what to expect when
transactions fail due to lock contention.
* On 2009-06-18 at 12:05 +0200 Dan Gudmundsson wrote:
> Except for io:format("Expecting ~p~n", [lists:seq(1, 10)]), which is wrong,
> since you use dirty_update_counter which will be invoked
> several times when there is a lock conflict on the write lock.
I think I understand the issue with transaction code possibly getting
executed multiple times, but for a transaction like:
mnesia:transaction(fun() ->
mnesia:lock({table, foo}, write),
Id = mnesia:dirty_update_counter(counter, id, 1),
New = #foo{ code = Id, created = now() },
mnesia:write(New)
end)
The docs seem to suggest that mnesia:lock will exit if it can't obtain
a lock. In that case, one would assume that we don't end up
incrementing the counter. Once the lock _is_ aquired, I would imagine
it would be a very exceptional case in which the transaction bombs
out. So now I'm thinking that the expectation that the result gives
consequitve elements from the counter is reasonable (and at least a
small amount of testing seems to bear this out). Or am I missing
another detail?
Cheers,
+ seth