Gluster on Scalr

[mikeytag]

Hi everyone,

I just added a page on how to run GlusterFS on Scalr/Amazon. There was
some interest earlier today and I thought the community could benefit
from learning how to use this awesome clustered file system.

Click on http://groups.google.com/group/scalr-discuss/web/gluster-on-scalr
- or copy & paste it into your browser's address bar if that doesn't
work.

Mike

[Hareem Haque]

Thanks for the article Mike.

Its awesome. I am running two servers all sync'd with the proper
content without me using the Synchronize to all feature.

Its just super awesome. I can't wait for your article on GlusterFS and
Stunnel.

Other then that. Great work.

Works like a charm. One has to adjust the cache size according to the
usage but it works great.

I am right now running one master server to which i have attached my
EBS volume. I did not use the XFS.. standard EXT3 all the way. It
works fine so far.

Both machines can sync content from the master server. I also use it
to run on the fly web server changes etc.

One thing i would like to know .. As i increase the number of clients
for Glusterfs. Would my EBS volume I/O start to go bezerk.

Humble Regards
Hareem. Haque

[handybitesize]

Many Thanks Mike

Could you please add an auto start for server for the init.deficient
please :)

[Hareem Haque]

Also mike could you please tell us whats better AFR or Unify. And how
to properly set them up.

Best Regards
Hareem. Haque

On Oct 29, 1:56 pm, mikeytag <mikey...@gmail.com> wrote:
> Hi everyone,
>
> I just added a page on how to run GlusterFS on Scalr/Amazon. There was
> some interest earlier today and I thought the community could benefit
> from learning how to use this awesome clustered file system.
>

> Click onhttp://groups.google.com/group/scalr-discuss/web/gluster-on-scalr

[mikeytag]

handybitesize,
The reason I haven't added an autostart for the server is because I am
not sure how to do the auto EBS volume mounting and such. I haven't
had a chance yet to play around with the new Scalr features on auto
EBS mount. However, the script would look like the client start but
the glusterfs line would look like this:

glusterfsd -f /etc/glusterfs/glusterfs-server.vol

Hareem,
AFR and Unify are two separate and distinct modules that can be used
separately or together. Gluster allows you to use AFR to replicate
files across the cluster and Unify to expand space on the cluster.

Here is a hypothetical example. Imagine two machines each with a
Gluster mount point /dev/sdb that is 50GB.

If you wanted to accomplish redundancy you could use AFR to replicate
all files between the two servers and Server A and B would have an
exact copy of the files between them. This is similar to a RAID 1
setup.

Let's say you really want the 100GB for space. Well then you could use
Unify to turn the two server's mount points into one big drive. Unify
would ..umm...unify the two 50GB mountpoints to give you one drive of
100GB. This is similar to a RAID 0 setup.

The immense power of Gluster is that you could intermix these two and
come up with a RAID-0+1 setup or virtually any other type of setup
that you could want or need.

Mike

[mikeytag]

Hareem,

I wanted to add one more thing. I keep all my clients and servers in
the same datacenter at Amazon. This is for speed and performance. I
also think that Amazon charges a fee for transfers between their
datacenters.

Mike

On Nov 6, 5:44 am, Hareem Haque <hareem.ha...@gmail.com> wrote:

[Hareem Haque]

Thanks for the info Mike. I really appreciate it.

I read some more details about setting up AFR. I think it might not be
that hard to work with the only thing i am worried about is that when
the master dies what happens to the AFR client.

The following is my Scalr.net setup.

1. Base AMI serving as master for Glusterfs

2. Two App servesr that serve as Glusterfs clients.

Now the EBS volume is a 10GB Ext3 file system that auto mounts on boot
on the Base AMI.

The issue is lets assume that an error occured in the farm and the
Glusterfs master dies out. The clients would retain the data on their
end but would that not mess up the connectivity of the clients. Is
there a was i can setup some sort of auto heal. So when the master
comes back online everything resumes as normal.

Also, thanks for the info on unify. I have a really good idea about
it. I have a client who wants to setup an in house 4 TB storage unit.
I will use comodity servers for that experiment and link them via
unify.

You are right. Amazon will charge for traffic between multiple data
centers. So i usually keep my instances on the zone c.

Best Regards
Hareem. Haque

[Sam]

Hi,

I am using gluster and automatic feature of Scalr to mount the device
when the farm gets restarted.
What I am not still getting is persistence.

In the scalr when you select edit farm I could see this option:

>>When instance based on this role boots up:
>>Do not use EBS
>>Attach empty volume with size: GB
>>Attach volume from snapshot:

>>Automatically mount device to mount point.

Scalr clearly says:
When new instance initialized, Scalr will
1. Attach a first detached volume, left by terminated or crashed
instance or create a new EBS volume, attach it, and create an ext3
filesystem on it.
2. If "Automatically mount device" option selected, volume will be
mounted.

So even when I select Attach emty volume of size X GB it should check
for the storage of the existing volume. It is every time creating a
new volume which is a bit surprising.

I also tried creating EBS volume independently and then attaching it
with scalr but every time I restart the instance it gets detached
which is a bit of setback.

How do I make scalr create EBS storage at the first start up and then
use the storage from terminated instance instead of creating a new one
thereafter?

Please help!

Thanks

[Alex Kovalyov]

Not quite sure what your problem is.

On 07.11.08 14:24, "Sam" <msem...@googlemail.com> wrote:

> So even when I select Attach emty volume of size X GB it should check
> for the storage of the existing volume. It is every time creating a
> new volume which is a bit surprising.

What is "check for the storage of the existing volume"?

1. Scalr should create new volume for each instance, and mount it, if
needed.
2. If instance crashes or terminates, Scalr launches a new one (scaling),
then attaches a volume that were freed during crash/terminate.

Does it behave differently for you?

[Hareem Haque]

Hi Sam.

That sounds odd. I loaded up Glusterfs on my farm and i was running
them fine. Initially i attached a 10GB Ext3 EBS volume to my master
server. After the client and server config i started the synchronize
to all feature and once that completed my EBS volume sticks to my
master instance everytime i use it.

Regards
Hareem. Haque

[Mahadev Semwal]

Here it the problem:

I have mysqllvm, app and www role.


My application is fine with mysqllvm as it has enough storage.
My application writes images and lots of animation files to app server directory.
This is the reason I wanted to attach EBS volume to app server.
I installed gluster on app server and tried the following two tasks:

1. I specified in the role to create 15 GB everytime the instance is started and attached it to /gluster directory.
Other option was not to use EBS (Which I do not want) or to use existing volume (which I do not have.)So I selected create a new volume of 15GB everytime the instance is craeted. It says in the note that if the volume already exist it will be used. SO next time it will use existing volume.
I started the farm. I could see 15GB ESB attached to my instance mounted to /gluster directory.
I created two directories /upload and /images on /gluster directory and put two images there.
Upto here fine.

I terminated the farm and restarted it.
I could see /gluster attached to my instance with EBS storage.

BUT.....
My data and directories (upload and images) are missing.

i.e. scalr created a new Volume instead of using previous volume.

2. I created EBS of 15GB seperately and attached it to the instance.
I restarted the farm and the EBS volume is detached. I do not know how to make that attachment permanently on the role so that I do not have to mount it everytime.
i.e if I do not use auto mount feature of scalr there is no way to automatically mount EBS on instance, you have to attach and detach automatically.


Could you tell me what I am doing wrong here? preferably for option 1 using Scalr?

Thanks

[Hareem Haque]

Hi Mahadev.

Did you use the synchronize to all feature prior to terminating your
farm.

[Sam]

I had only one instance and when I was presented with Synchronise
instance, I selected YES.

This happened 3 times. Every time I start the instance, the
directories created inside /gluster directory dissapears.

[Alex Kovalyov]

Did you choose to keep EBS volumes upon farm termination?

[Sam]

Yes, I chose to keep EBS storage when terminating.

[Sam]

Alex,

Can you tell me please what option I should be selecting to auto mount
EBS volume and at the same time making sure that Scalr does not create
a new volume of EBS if it already exist?

Thanks

[Hareem Haque]

Mike are you seeing the same problem as Sam.

Regards
Hareem

[Alex Kovalyov]

Ok this was reproduced and going to be fixed shortly.
The key was to choose Synchronize upon farm terminate for the role that has
auto-EBS enabled.

On 07.11.08 15:21, "Mahadev Semwal" <msem...@googlemail.com> wrote:

[Sam]

I can confirm that when you terminate a particular instance and that
instance comes back on after rebooting, you DO retain the data on the
ESB which was attached to the role.

On the other hand when you terminate the farm and select keep my EBS
data storage and synchronise all the instances of the farm, once
terminated you relaunch the farm and go to the directory where you
have mounted EBS, your data is all GONE.

Could you please provide us the fix as soon as possible as this is
critical for some of my applications?

Many thanks

On Nov 7, 4:06 pm, Alex Kovalyov <alex.koval...@gmail.com> wrote:
> Ok this was reproduced and going to be fixed shortly.
> The key was to choose Synchronize upon farm terminate for the role that has
> auto-EBS enabled.
>

[Alex Kovalyov]

It was fixed.

[Carey Hiles]

Hi Alex - is that a global fix or on a per instance basis as I have seen
the same behavior previously

[Alex Kovalyov]

Carey,

That was a global fix.
Let us know if you will see it again (provide steps to reproduce if
possible).

Alex

[mikeytag]

Hi guys,
I had a similar question about EBS persistence related to my Gluster
storage nodes and posted it here:

http://groups.google.com/group/scalr-discuss/t/8e8f779c00c2ae92

Mike

On Nov 10, 5:24 am, Alex Kovalyov <alex.koval...@gmail.com> wrote:
> Carey,
>
> That was a global fix.
> Let us know if you will see it again (provide steps to reproduce if
> possible).
>
> Alex
>

> On 10.11.08 11:46, "Carey Hiles" <carey.hi...@gmail.com> wrote:
>
>
>
> > Hi Alex - is that a global fix or on a per instance basis as I have seen
> > the same behavior previously
>
> > Alex Kovalyov wrote:
> >> It was fixed.
>

> ...
>
> read more »

[BTE]

What happens if your gluster storage node instance dies? What happens
if that happens at the same time as the ebs? Are there regular
snapshots to s3 a possibility like with the mysql ebs drive?

[BTE]

Was the ssh tunnel doc ever published?

[Hareem Haque]

BTE you can use the encryption module for Glusterfs

Link:

http://www.gluster.org/docs/index.php/Translators_v1.4#Encryption_Translators

[Hareem Haque]

BTE i think regular snapshots of EBS are beyond the scope of
Glusterfs. You can setup your own scripts to regularly backup EBS
volume to S3. Basically, Glusterfs is a better replacement for NFS at
the moment. I use it to distribute config files amongst my farm
instances.

[BTE]

I understand it is beyond the scope of gluster... my hope was that it
is not beyong the scope of sclar. If it is then I think it would be a
good additional role or script addition to app.

Can all the apps act as both sotrage nodes and client nodes?

> > snapshots to s3 a possibility like with the mysql ebs drive?- Hide quoted text -
>
> - Show quoted text -

[marten]

How do I make sure the gluster *server* is started automatically ? I'm
guessing I need to add something to /etc/init.d and run an update-
rc.d, but not sure what and how...

[Hareem Haque]

I don't think that Stunnel info was ever posted

On Jan 10, 9:53 pm, BTE <ke...@rateladder.com> wrote:

[mikeytag]

For everyone wondering about the stunnel tutorial. I have actually
currently abandoned my own stunnel setup as I was having issues with
stunnel crashing and causing me all sorts of headaches. Also, Gluster
is currently working on their own certificate based authentication
that might be out with the next release. Here is a Gluster tutorial on
how to do it for those interested though:

http://www.gluster.org/docs/index.php/GlusterFS_Encrypted_network