Scalaris performance problem
dory.thibault
I have set up a cluster oi 3 computer running ubuntu server. The
cluster if configured like that :
- 1 boot node that is also the only logging node
- 2 simple node
- Each of them uses the default configuration, except for the cluster
conf
The boot node and all the nodes starts without any problem and I can
see all of them in the web monitoring interface.
I have downloaded more or less 20000 articles from wikipedia in XML
format summing up to 620Mb. I'm using the java api to insert them with
this function :
public int updateDB(String ID, String newValue) {
int res;
OtpErlangString newValueO = new OtpErlangString(newValue);
OtpErlangString IDO = new OtpErlangString(ID);
try{
sc.writeObject(IDO, newValueO);
res = 1;
}catch (Exception e) {
e.printStackTrace();
res = -1;
}
return res;
}
The inserts works but are very slow, it would take more than 30
minutes to insert all the articles! But the main problem is that I ran
out of memory way before all the articles were inserted. The nodes
were only running Scalaris at that time and each of them as 8Gb of
RAM. At 8000 files inserted, the total RAM used by Scalaris on the 3
nodes was more than 13Gb!
I assume this behavior is not normal and that something must be wrong
in my configuration or in the way I use the java api.
Do you have any idea of what could be the problem here?
dory.thibault
I have installed Scalaris using the repo given in the faq. Add the
following line in your /etc/apt/source.list :
deb http://download.opensuse.org/repositories/home:/tschuett/xUbuntu_10.04/
./
then simply run sudo apt-get update and sudo apt-get install scalaris
On the first node edit the file /etc/scalaris/scalaris.cfg and at the
bottom of the file change the listen_ip parameter to match the IP of
the node. Then put the same IP in the fields boot_host and log_host
without touching anything else.
On the other nodes edit the same file this way : edit the listen_ip
field to match the node's IP and the boot_host and log_host to match
the IP of the first node.
When it is done, on the first node execute the command : scalarisctl
boot start on the first node. That will start the boot and log servers
and a simple node on the first server. Then on both other nodes
execute scalarisctl node start
You should now be able to access firstNodeIP:9001 in your web browser
to verify that all the nodes are in the ring.
This works for setting up a ring but is it possible that this way of
configuring the cluster gives me bad performance? Should I start more
than one log_host?
On 30 nov, 21:09, Anvesh <anvesh.pra...@gmail.com> wrote:
> Hi,
> You said that you were able to successfully start boot node and other
> nodes.
> I`m having a problem setting up a boot node and other nodes. Each node
> is starting it`s own bootserver.
> Can u post the boot.sh settings and scalaris.properties settings of
> bootserver and simple nodes.
> This will be very helpful for me,
>
> Thanks,
> Anvesh.
dory.thibault
problem, so could someone at least tell me if it is a normal behaviour
for Scalaris to use this quantity of RAM for strings that sums up to
less than 620Mb on the disk?
On 1 déc, 14:36, "dory.thibault" <dory.thiba...@gmail.com> wrote:
> Hi Anvesh,
>
> I have installed Scalaris using the repo given in the faq. Add the
> following line in your /etc/apt/source.list :
>
Florian Schintke
> Ok I can see that I do not generate a lot of attention with this
> problem, so could someone at least tell me if it is a normal behaviour
> for Scalaris to use this quantity of RAM for strings that sums up to
> less than 620Mb on the disk?
> At 8000 files inserted, the total RAM used by Scalaris on the 3
> nodes was more than 13Gb!
We store 4 replicas: makes 2.5 GB
If you store strings, they are linked lists in Erlang with a single
byte and a pointer to the next element.
If you use 64bit machine, this makes: 2.5 * (1 + 8) = 22.5 GB
If you use 32bit machine, this makes: 2.5 * (1 + 4) = 12.5 GB
Regards,
Florian
dory.thibault
The machines are indeed 64bit and that explains why so much RAM is
used.
Would it be possible to store the same information using a different
type to use less memory?
For example is there a way to create an OtpErlangObject that contains
all bytes in one sequence instead of a linked list?
I have observed that the repartition of data changes everytime I
restart the cluster, it seems to be random. Is it possible to tell
scalaris to split the data in equal size between the nodes? With the
current way the nodes join the ring, some of them takes a lot of data
under their responsibility and therefore run out of memory very fast
while other nodes have plenty of free memory but only store a small
part of the data.
Thibault
Florian Schintke
anwers compiled from earlier posts:
Our java-api only reads and writes strings. If you want to write
binary data, you will need to extend the Main.java (or write your
own). You need to write a wrapper class that extends CustomOtpObject
(see CustomOtpFastStringObject in the examples directory) and maps a
Java-type to an Erlang-type. Then you can use
Scalaris/Transaction.writeCustom with such an object which will write
it to a running Scalaris ring.
Reading works analogously.
Regarding the partitioning, you can place nodes on the ring
explicitly. To create a symmetric Scalaris ring with two resp. four
nodes:
on PC1:
./bin/scalarisctl -k 0 boot start
on PC2:
./bin/scalarisctl -k 170141183460469231731687303715884105728 node start
if you want to add two more nodes, they would have to have the following keys:
on PC3:
./bin/scalarisctl -k 85070591730234615865843651857942052864 node start
on PC4:
./bin/scalarisctl -k 255211775190703847597530955573826158592 node start
Florian
[dory.thibault]
> Thank you for your answer Florian, that is very nice of you :)
>
> The machines are indeed 64bit and that explains why so much RAM is
> used.
> Would it be possible to store the same information using a different
> type to use less memory?
> For example is there a way to create an OtpErlangObject that contains
> all bytes in one sequence instead of a linked list?
>
> I have observed that the repartition of data changes everytime I
> restart the cluster, it seems to be random. Is it possible to tell
> scalaris to split the data in equal size between the nodes? With the
> current way the nodes join the ring, some of them takes a lot of data
> under their responsibility and therefore run out of memory very fast
> while other nodes have plenty of free memory but only store a small
> part of the data.
>
> Thibault
>
> On 3 d�c, 16:38, Florian Schintke <schin...@gmail.com> wrote:
> > Yes, its normal...
> >
> > > Ok I can see that I do not generate a lot of attention with this
> > > problem, so could someone at least tell me if it is a normal behaviour
> > > for Scalaris to use this quantity of RAM for strings that sums up to
> > > less than 620Mb on the disk?
> > > At 8000 files inserted, the total RAM used by Scalaris on the 3
> > > nodes was more than 13Gb!
> >
> > We store 4 replicas: makes 2.5 GB
> >
> > If you store strings, they are linked lists in Erlang with a single
> > byte and a pointer to the next element.
> >
> > If you use 64bit machine, this makes: 2.5 * (1 + 8) = 22.5 GB
> > If you use 32bit machine, this makes: 2.5 * (1 + 4) = 12.5 GB
> >
> > Regards,
> >
> > Florian
>
> --
> You received this message because you are subscribed to the Google Groups "scalaris" group.
> To post to this group, send email to scal...@googlegroups.com.
> To unsubscribe from this group, send email to scalaris+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/scalaris?hl=en.
Florian
Nico Kruber
> Our java-api only reads and writes strings. If you want to write
> binary data, you will need to extend the Main.java (or write your
> own). You need to write a wrapper class that extends CustomOtpObject
> (see CustomOtpFastStringObject in the examples directory) and maps a
> Java-type to an Erlang-type. Then you can use
> Scalaris/Transaction.writeCustom with such an object which will write
> it to a running Scalaris ring.
>
> Reading works analogously.
the CustomOtpFastStringObject class in our examples directory could do the job for you - just be advised to use the same architecture on all machines. The class is simply using the bytes from the Java string and the writes them into an OtpErlangBinary.
Nico Kruber
memory...
> > > The machines are indeed 64bit and that explains why so much RAM is
> > > used.
> > > Would it be possible to store the same information using a different
> > > type to use less memory?
> > > For example is there a way to create an OtpErlangObject that contains
> > > all bytes in one sequence instead of a linked list?
We have recently decided to (internally) compress all values into binaries
before transferring and storing them in Scalaris. Recent Scalaris trunk
checkouts include that change and for those you can now still use ordinary
strings and won't get too much memory overhead. In fact due to the
compression, memory use will even be less than the (uncompressed) strings.
> > > I have observed that the repartition of data changes everytime I
> > > restart the cluster, it seems to be random. Is it possible to tell
> > > scalaris to split the data in equal size between the nodes? With the
> > > current way the nodes join the ring, some of them takes a lot of data
> > > under their responsibility and therefore run out of memory very fast
> > > while other nodes have plenty of free memory but only store a small
> > > part of the data.
Passive load balancing should do that job for you and is enabled by default
now. It first tries to split address ranges uniformly during join. If the
existing nodes have had contact to the other nodes for a while and aggregated
the average load in the system (gossip), they will split the actual load
instead of the address space.
Please check if your problem still occurs.
Nico