scalaris beginner quick start

[adr...@gmail.com]

Hello,

I'd like to share these scalaris beginner quick start snippets.

Once you have (easyly) build scalaris (configure+make), "For simple tests, you do not need to install Scalaris. You can run it directly from the source directory".

Let's experiment a scalaris DBMS cluster with only one computer thanks to Erlang or virtualisation.

Just slightly copy-modify scalaris.cfg:
{listen_ip, {127,0,0,1}}.
{mgmt_server, {{127,0,0,1},14195,mgmt_server}}.
{known_hosts, [    {{127,0,0,1},14195, service_per_vm},
                {{127,0,0,1},14196, service_per_vm},
                {{127,0,0,1},14197, service_per_vm},
                {{127,0,0,1},14198, service_per_vm}
                % no n5 99, not listed here
            ]}.

In shell S1, start the first node ("premier"):
./bin/scalarisctl -m -n pre...@127.0.0.1 -p 14195 -y 8000 -s -f start
(note the -m and -f options, management and first_node)

In a monitoring shell MS, see the "premier" node:
./bin/scalarisctl list
epmd: up and running on port 4369 with data:
name premier at port 47235

In a web browser pointed at 127.0.0.1:8000 you can see the first node ("premier") and its management role.

Let's add 4 nodes in the cluster (use new shells S2 to S5, each simulating a physical computer):
./bin/scalarisctl -n sec...@127.0.0.1 -p 14196 -y 8001 -s start
./bin/scalarisctl -n n...@127.0.0.1 -p 14197 -y 8002 -s start
./bin/scalarisctl -n n...@127.0.0.1 -p 14198 -y 8003 -s start
./bin/scalarisctl -n n...@127.0.0.1 -p 14199 -y 8004 -s start

See them joined (use shell MS):
./bin/scalarisctl list
epmd: up and running on port 4369 with data:
name n5 at port 47801
name n4 at port 54614
name n3 at port 41710
name second at port 44329
name premier at port 44862

See each of their web console on 127.0.0.1:8001 to 8004. (all the same and not the same as 8000 started with management role)
All nodes claim a 5 nodes cluster ring.

Now let us use the web console client interface:
Go to 8000 and lookup something -> {fail,not_found} (ok).
Add k1/v1 and k2/v2 -> {ok} and {ok}.
lookup k1 & k2 -> {ok,"v1"} & {ok,"v2"}.
look them up from other nodes -> ok, same.
update k1 to v1updated from second(8001) -> {ok}.
update k2 to v2updated from n5(8004) -> {ok}.
look them up from other nodes -> ok, updated.

Now let's stop/kill n4 (use shell MS):
./bin/scalarisctl -n n...@127.0.0.1 stop
Other nodes notice the crash.
See new 4-nodes list in shell MS with ./bin/scalarisctl list
Look up k1 & k2 from all other nodes -> ok, available.

Now let's restart n4 (in shell S4):
./bin/scalarisctl -n n...@127.0.0.1 -p 14198 -y 8003 -s start
See restored node list (in shell MS) with ./bin/scalarisctl list
Look up k1 & k2 from n4 or other nodes -> ok, available.

We have a 5 node cluster again and no data loss.

Let's now use the erlang shell of a server node to experiment with the erlang API.
Hit <return> to see the erlang shell prompt in shell S1 (premier@127...):
(pre...@127.0.0.1)1> api_tx:read("k0").
{fail,not_found}
(pre...@127.0.0.1)2> api_tx:read("k1").
{ok,"v1updated"}
(pre...@127.0.0.1)3> api_tx:read("k2").
{ok,"v2updated"}
(pre...@127.0.0.1)4> api_tx:read(<<"k1">>).
{ok,"v1updated"}
(pre...@127.0.0.1)5> api_tx:read(<<"k2">>).
{ok,"v2updated"}
(pre...@127.0.0.1)6> api_tx:write(<<"k3">>,<<"v3">>).
{ok}
(pre...@127.0.0.1)7> api_tx:read(<<"k3">>).          
{ok,<<"v3">>}
(pre...@127.0.0.1)8> api_tx:read("k3").   
{ok,<<"v3">>}
(pre...@127.0.0.1)9> api_tx:write(<<"k4">>,{1,2,3,four}).
{ok}
(pre...@127.0.0.1)10> api_tx:read("k4").                 
{ok,{1,2,3,four}}

Let's now connect a true client to our 5 nodes scalaris DBMS cluster.
We use a new shell (again) to run an erlang VM to do remote API calls to the server nodes.
This is a quick-and-dirty discovery (using rpc:call/4). A production system would have some more sophisticated client side module. For example it would automatically dispatch requests to server nodes.

erl -name cli...@127.0.0.1 -hidden -setcookie 'chocolate chip cookie'
(cli...@127.0.0.1)1> net_adm:ping('n...@127.0.0.1').
pong
(cli...@127.0.0.1)2> rpc:call('n...@127.0.0.1', api_tx, read, [<<"k0">>]).
{fail,not_found}
(cli...@127.0.0.1)3> rpc:call('n...@127.0.0.1', api_tx, read, [<<"k4">>]).
{ok,{1,2,3,four}}
(cli...@127.0.0.1)4> rpc:call('n...@127.0.0.1', api_tx, read, [<<"k4">>]).
{ok,{1,2,3,four}}
(cli...@127.0.0.1)5> rpc:call('n...@127.0.0.1', api_tx, write, [<<"num5">>,55]).
{ok}
(cli...@127.0.0.1)6> rpc:call('n...@127.0.0.1', api_tx, read, [<<"num5">>]).   
{ok,55}
(cli...@127.0.0.1)7> rpc:call('n...@127.0.0.1', api_tx, add_on_nr, [<<"num5">>,2]).
{badrpc,nodedown}
(cli...@127.0.0.1)8> rpc:call('sec...@127.0.0.1', api_tx, add_on_nr, [<<"num5">>,2]).
{ok}
(cli...@127.0.0.1)9> rpc:call('n...@127.0.0.1', api_tx, read, [<<"num5">>]).
{ok,57}
(cli...@127.0.0.1)10> rpc:call('n...@127.0.0.1', api_tx, test_and_set, [<<"num5">>,57,59]).
{ok}
(cli...@127.0.0.1)11> rpc:call('n...@127.0.0.1', api_tx, read, [<<"num5">>]).
{ok,59} 
(cli...@127.0.0.1)12> rpc:call('n...@127.0.0.1', api_tx, test_and_set, [<<"num5">>,57,55]).
{fail,{key_changed,59}}
(cli...@127.0.0.1)13> rpc:call('n...@127.0.0.1', api_tx, read, [<<"num5">>]).
{ok,59}
(cli...@127.0.0.1)14> rpc:call('n...@127.0.0.1', api_tx, test_and_set, [<<"k2">>,"v2updated",<<"v2updatedTWICE">>]).
{ok}
(cli...@127.0.0.1)15> rpc:call('n...@127.0.0.1', api_tx, read, [<<"k2">>]).
{ok,<<"v2updatedTWICE">>}
(cli...@127.0.0.1)16> rpc:call('n...@127.0.0.1', api_tx, add_on_nr, [<<"num5">>,-4]).                              
{ok}
(cli...@127.0.0.1)17> rpc:call('n...@127.0.0.1', api_tx, read, [<<"num5">>]).
{ok,55}
(cli...@127.0.0.1)18> q().
ok

Just for the fun, another client computer now connects to the cluster and read updates made by the first:
erl -name clien...@127.0.0.1 -hidden -setcookie 'chocolate chip cookie'
(clien...@127.0.0.1)1> net_adm:ping('n...@127.0.0.1').
pong
(clien...@127.0.0.1)2> rpc:call('n...@127.0.0.1', api_tx, read, [<<"k0">>]).
{fail,not_found}
(clien...@127.0.0.1)3> rpc:call('n...@127.0.0.1', api_tx, read, [<<"k1">>]).
{ok,"v1updated"}
(clien...@127.0.0.1)4> rpc:call('n...@127.0.0.1', api_tx, read, [<<"k2">>]).
{ok,<<"v2updatedTWICE">>}
(clien...@127.0.0.1)5> rpc:call('sec...@127.0.0.1', api_tx, read, [<<"num5">>]).
{ok,55}

In shell MS we now list and stop the nodes:
./bin/scalarisctl list
epmd: up and running on port 4369 with data:
name n4 at port 52504
name n5 at port 47801
name n3 at port 41710
name second at port 44329
name premier at port 44862

./bin/scalarisctl -n sec...@127.0.0.1 stop
'sec...@127.0.0.1'
./bin/scalarisctl -n n...@127.0.0.1 stop
'n...@127.0.0.1'
./bin/scalarisctl -n n...@127.0.0.1 stop
'n...@127.0.0.1'
./bin/scalarisctl -n n...@127.0.0.1 stop
'n...@127.0.0.1'

./bin/scalarisctl list
epmd: up and running on port 4369 with data:
name premier at port 44862

./bin/scalarisctl -n pre...@127.0.0.1 stop
'pre...@127.0.0.1'
./bin/scalarisctl list
epmd: up and running on port 4369 with data:
(nothing)

To go further see also:
-api_tx:add_del_on_list
-api_tx bulk ops
-Transaction TLog (new tlog then commit)
-PubSub

Hope this little guide has demonstrated how to setup a 5-50 nodes scalaris cluster, how to recover a failing node and how to make remote client requests.

May be this could be used for a page on scalaris' wiki ?
Have a nice read.

Pierre M.

[adr...@gmail.com]

Hello again,

here is a client side discovery followup about Transactions using api_tx:req_list(Tlog, List).

The setup is similar to the 5 nodes previous cluster. To simplyfy API calls are typed inside erlang shells of nodes n4 & n5.

Let us imagine 2 concurrent transactions : long A on n5 has disturbing short B on n4 meanwhile.
Transaction A starts before Transaction B and Transaction B ends before Transaction A. B is "timely" nested in A and disturbs A.

(n...@127.0.0.1)10> api_tx:read(<<"k1">>).
{ok,<<"v1">>}
(n...@127.0.0.1)17> api_tx:read(<<"k1">>).
{ok,<<"v1">>}
(n...@127.0.0.1)18> T5longA0 = api_tx:new_tlog().
[]
(n...@127.0.0.1)19> {T5longA1, R5longA1} = api_tx:req_list(T5longA0, [{read, <<"k1">>}]).
{[{76,<<"k1">>,1,75,'$empty'}],[{ok,<<"v1">>}]}
(n...@127.0.0.1)11> T4shortB0 = api_tx:new_tlog().
[]
(n...@127.0.0.1)12> {T4shortB1, R4shortB1} = api_tx:req_list(T4shortB0, [{read, <<"k1">>}]).
{[{76,<<"k1">>,1,75,'$empty'}],[{ok,<<"v1">>}]}
(n...@127.0.0.1)13> {T4shortB2, R4shortB2} = api_tx:req_list(T4shortB1, [{write, <<"k1">>, <<"v1Bshort">>}]).
{[{77,<<"k1">>,1,75, <<131,109,0,0,0,8,118,49,66,115,104,111,114,116>>}],
 [{ok}]}
(n...@127.0.0.1)14> {T4shortB3, R4shortB3} = api_tx:req_list(T4shortB2, [{read, <<"k1">>}]).                
{[{77,<<"k1">>,1,75, <<131,109,0,0,0,8,118,49,66,115,104,111,114,116>>}],
 [{ok,<<"v1Bshort">>}]}
(n...@127.0.0.1)15> {T4shortB4, R4shortB4} = api_tx:req_list(T4shortB3, [{commit}]).       
{[],[{ok}]} % SEE B ends OK
(n...@127.0.0.1)16> [R4shortB1,R4shortB2,R4shortB3,R4shortB4].
[[{ok,<<"v1">>}],[{ok}],[{ok,<<"v1Bshort">>}],[{ok}]]
(n...@127.0.0.1)20> {T5longA2, R5longA2} = api_tx:req_list(T5longA1, [{read, <<"k1">>}]).
{[{76,<<"k1">>,2,{fail,abort},'$empty'}],
 [{ok,<<"v1Bshort">>}]} % SEE #### FAIL & ABORT ####
(n...@127.0.0.1)21> {T5longA3, R5longA3} = api_tx:req_list(T5longA2, [{write, <<"k1">>,<<"v1Along">>}]).
{[{76,<<"k1">>,2,{fail,abort},'$empty'}],[{ok}]}
(n...@127.0.0.1)22> {T5longA4, R5longA4} = api_tx:req_list(T5longA3, [{read, <<"k1">>}]).              
{[{76,<<"k1">>,2,{fail,abort},'$empty'}],
 [{ok,<<"v1Bshort">>}]}
(n...@127.0.0.1)23> {T5longA5, R5longA5} = api_tx:req_list(T5longA4, [{commit}]).
{[],[{fail,abort,[<<"k1">>]}]} % SEE #### FAIL & ABORT ####
(n...@127.0.0.1)17> api_tx:read(<<"k1">>).
{ok,<<"v1Bshort">>}
(n...@127.0.0.1)24> api_tx:read(<<"k1">>).
{ok,<<"v1Bshort">>}

The first coherent commit (short B on n4) has won. (as expected)

First try, first success. It seems scalaris transactions "just work" :-)

Of course in api_tx:req_list(Tlog, List) the List should group together several reads&writes and a trailing {commit}. The individual separation of all reads, writes and commits was done here on purpose to study the transactional behaviour.

Pierre M.

[Magnus Müller]

This could be a good starting point to amend chapter 4 ("Using the
system") in the manual. Chapter 4 describes the usable data types and
how to use the command line clients, but doesn't mention how to work
with Scalaris using Erlang. If the other developers also think that
chapter 4 would be a good place for such a tutorial, I'd use your
guide for such a subsection. Otherwise, a wiki entry combining this and
your prior usage guide would be nice as well.

Magnus

[Florian Schintke]

Yes, I would like to see it in the user-dev-guide. Maybe in a new
section 4.3 "Using Scalaris from the Erlang-Shell" or "Using Scalaris
from Erlang"?

Florian

> --
> You received this message because you are subscribed to the Google Groups "scalaris" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to scalaris+u...@googlegroups.com.
> To post to this group, send email to scal...@googlegroups.com.
> Visit this group at http://groups.google.com/group/scalaris?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.

[adr...@gmail.com]

Hello back,

thank you all for your enthousiastic feedback on my feedback.

Feel free to correct these two little walkthrough, to amend them and to publish them.
I think manual and wiki are complementary :

• The manual serves as a canonical reference to show the recommended way of best software experience and sophisticated cases.
• The wiki aims at first look. Its pages are best found by web search engines. It helps developing the community.
  

Pierre M.

[Magnus Müller]

Hey Pierre.


I think you are right that the wiki is better reachable via search
engines, so it would probably it's good idea to put those guides there.
But I also think that the manual should contain first steps to play
around with the system, for which your guides would be a good starting
point. I don't see the manual as only being a reference guide,
additional information won't hurt. Maybe we could put it in both
places?

I attached a proposal for the user guide, which I compiled from your
text and amended with extra information. If others think this should be
put in the Wiki, we can do that as well.

Magnus

On Tue, 5 Feb 2013 05:41:20 -0800 (PST)
adr...@gmail.com wrote:

>>Hello back,
>>
>>thank you all for your enthousiastic feedback on my feedback.
>>
>>Feel free to correct these two little walkthrough, to amend them and
>>to publish them.
>>I think manual and wiki are complementary :
>>

>> - The manual serves as a canonical reference to show the

>> recommended way of best software experience and sophisticated cases.

>> - The wiki aims at first look. Its pages are best found by web

>> search engines. It helps developing the community.
>>
>>
>>Pierre M.
>>
>>
>>Le mardi 5 février 2013 09:40:53 UTC+1, Florian a écrit :
>>>
>>> Yes, I would like to see it in the user-dev-guide. Maybe in a new
>>> section 4.3 "Using Scalaris from the Erlang-Shell" or "Using
>>> Scalaris from Erlang"?
>>>
>>> Florian
>>>
>>> > This could be a good starting point to amend chapter 4 ("Using
>>> > the system") in the manual. Chapter 4 describes the usable data
>>> > types and how to use the command line clients, but doesn't
>>> > mention how to work with Scalaris using Erlang. If the other
>>> > developers also think that chapter 4 would be a good place for
>>> > such a tutorial, I'd use your guide for such a subsection.
>>> > Otherwise, a wiki entry combining this and your prior usage guide
>>> > would be nice as well.
>>> >
>>> > Magnus
>>> >
>>> > --
>>> > You received this message because you are subscribed to the
>>> > Google
>>> Groups "scalaris" group.
>>> > To unsubscribe from this group and stop receiving emails from it,
>>> > send

>>> an email to scalaris+u...@googlegroups.com <javascript:>.

>>> > To post to this group, send email to

>>> > scal...@googlegroups.com<javascript:>.

>>>
>>> > Visit this group at
>>> > http://groups.google.com/group/scalaris?hl=en. For more options,
>>> > visit https://groups.google.com/groups/opt_out.
>>>
>>

>>--
>>You received this message because you are subscribed to the Google
>>Groups "scalaris" group. To unsubscribe from this group and stop
>>receiving emails from it, send an email to scalaris

>>+unsub...@googlegroups.com. To post to this group, send email to

[adr...@gmail.com]

Hello Magnus,

wwwo, I'm impressed. You have made 7 pages of high quality documentation out of my shell cut-n-pasting. I very much appreciate it. It makes me happy and I hope other beginners will improve their scalaris skills with this guide.

I agree with you that all content (howtos & reference) can be in both places (web pages and PDF manual).

Have fun

Pierre M.

[Florian Schintke]

Hi Magnus,

thanks a lot, it looks good. Please submit it, I would like to do some
minor polishing here and there.

Florian

> Hey Pierre.
>
>
> I think you are right that the wiki is better reachable via search
> engines, so it would probably it's good idea to put those guides there.
> But I also think that the manual should contain first steps to play
> around with the system, for which your guides would be a good starting
> point. I don't see the manual as only being a reference guide,
> additional information won't hurt. Maybe we could put it in both
> places?
>
> I attached a proposal for the user guide, which I compiled from your
> text and amended with extra information. If others think this should be
> put in the Wiki, we can do that as well.
>
> Magnus
>
> On Tue, 5 Feb 2013 05:41:20 -0800 (PST)
> adr...@gmail.com wrote:
>
> >>Hello back,
> >>
> >>thank you all for your enthousiastic feedback on my feedback.
> >>
> >>Feel free to correct these two little walkthrough, to amend them and
> >>to publish them.
> >>I think manual and wiki are complementary :
> >>
> >> - The manual serves as a canonical reference to show the
> >> recommended way of best software experience and sophisticated cases.
> >> - The wiki aims at first look. Its pages are best found by web
> >> search engines. It helps developing the community.
> >>
> >>
> >>Pierre M.
> >>
> >>

> >>Le mardi 5 f�vrier 2013 09:40:53 UTC+1, Florian a �crit :

> To unsubscribe from this group and stop receiving emails from it, send an email to scalaris+u...@googlegroups.com.

> To post to this group, send email to scal...@googlegroups.com.
> Visit this group at http://groups.google.com/group/scalaris?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Florian