Azure Hosting

[Alexander Zeitler]

Hi,

I know there's RavenHQ and I would love to use it but I need to get the DB hosted in Europe...

So my question is what's the best/common practice to host RavenDb on Azure including sharding?

Alex

[Matt Warren]

The best way is to run RavenDB on 2 web-roles and enable replication between them. This way you'll be able to cope when the roles are recycled and you can run the 2 roles needed to get the SLA.

As RavenDB needs to write to the disk, you can load/stored the filesystem on a paged block. There's a code sample at https://github.com/markrendle/AzureRavenDB

[jalchr]

Matt,

When you say 2-web roles, you don't mean 2-instances of the same web role ... right ?

(If 2-instances of the same webrole, then they are going to access the same storage account ... wouldn't this causes transactional problems ?)

So I think you mean 2 different web roles that have different end-points and store their data in different storage locations + enable replication between them as you would go with any 2 "regular" servers on a network.

Did I get you right ?

If so, how does the application communicate with these 2 - web roles when they have different urls, and does failover work.

[Matt Warren]

On Friday, 27 April 2012 08:37:31 UTC+1, jalchr wrote:

Matt,

When you say 2-web roles, you don't mean 2-instances of the same web role ... right ?

(If 2-instances of the same webrole, then they are going to access the same storage account ... wouldn't this causes transactional problems ?)

No 2 instances of the same web (or worker role), see https://github.com/markrendle/AzureRavenDB/blob/master/AzureRavenDB/ServiceConfiguration.cscfg#L4. This example has one actual worker role C# project (one WorkerRole.cs file) but 2 instances of it are spun up, both running on difference machines/hosts.

 

So I think you mean 2 different web roles that have different end-points and store their data in different storage locations + enable replication between them as you would go with any 2 "regular" servers on a network.

Yes

 

Did I get you right ?

If so, how does the application communicate with these 2 - web roles when they have different urls, and does failover work.

 

The worker roles communicate between themselves via internal endpoints, see the code here https://github.com/markrendle/AzureRavenDB/blob/master/RavenWorker/WorkerRole.cs#L80 and the settings here https://github.com/markrendle/AzureRavenDB/blob/master/AzureRavenDB/ServiceDefinition.csdef#L15

Then I guess you could also have a couple of web roles that were publicity facing and they could each talk to either 1 of the worker roles (each hosting RavenDB).

[Alexander Zeitler]

Thanks, a friend of mine has been migrating it to Build 888 today (AFAIK blog post will published).

 

Alex

[jalchr]

Matt, 

I think something is not going to work in case of 2 intances per 1 worker role. 

http://social.msdn.microsoft.com/Forums/nb-NO/windowsazuredata/thread/8b2c31e5-cf20-458d-aac6-4efc5a2dadac 

The first instance starts, it mounts the cloud drive in a read/write way.

The second instance starts, it can NOT mount that same drive in a writable way, but in a ready-only way.

I'm I missing something here ?

[Alexander Zeitler]

They‘re are different VHDs

 

Alex

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of jalchr
Sent: Samstag, 28. April 2012 11:13
To: rav...@googlegroups.com
Subject: [RavenDB] Re: Azure Hosting

 

Matt, 

[André Andersen]

Hi Alex,

I'm very interested in the solution that your friend is doing with the 888 build with replication on Azure. Do you where he will share his experiences?

[jalchr]

Alex, 

Did you test this ... or have any reference ?

Cause I'm trying to test the solution posted by Matt on github and when I delpoy them to azure .... all I get is "Role is initializing" status when never reaches the "Ready" status

[Matt Warren]

This normally happends when something is throwing an exception in the OnStart(..) method in WebRole/WorkerRole.cs. What happens if you run it under the Azure emulator on your PC, does it work okay?

When I've had this type of problem before it's been because I has a config setting missing that, try putting a try/catch round the startup code and logging the error so you can see what's going on.

[Alexander Zeitler]

Hi,

 

at the moment we’re facing some issues.

I’ll post a link to the sources when available here.

 

Alex

[André Andersen]

I talked to a technical azure evangelist from Microsoft the other day regarding Azure and RavenDB. Allthough he didn't have experience with Raven in particular, his idea was that you would host the RavenDB instances in the same deployment as your web project (or whatever you're doing). This brings me to a question:

- What options do we have when it comes to load balancing RavenDB with replication (I presume master-master replication due to lack of experience).
- What's the best practice when configuring the DocumentStore in this sense, so it uses different instances (load balanced)?

[Chris Marisic]

I'm pretty sure in this scenario you need to tell RavenDB servers the azure internal host address of the other servers in the replication configuration.

Then from the RavenDB client side, that you use a normal document store and then you have the URL of the document store to be public.hostname which will then allow azure to load balance all traffic. With RavenDB servers talking to internal host names those requests won't be load balanced and will prevent the servers from getting confused.

Using master/master you may theoretically have to deal with conflict documents if multiple users attempt to modify the same things and it gets load balanced to other servers that they are out of sync.

[Matt Warren]

If you're still having problems you should take a look at this article. It uses the same approach as Mark's code, but is a much more thorough walk-through and might help you out with the issues you are having.

On Monday, 30 April 2012 20:54:38 UTC+1, Alexander Zeitler wrote:

Hi,

at the moment we’re facing some issues.

I’ll post a link to the sources when available here.

Alex

[André Andersen]

Hi Matt,

I found the article you linked to a while ago, but the thing is that he is only using 1 instance in his solution. My idea is to add more than 1 instance to "load balance" them with master-master replication. I am aware there might be conflicts using this type of solution in some cases, but it seems to be the easiest way to "scale" in my solution (creating some kind of social website). Perhaps I am approaching this in a wrong manner?

[Matt Warren]

Yeah I missed that bit, sorry for the confusion.

No you're right, you do want to have more than one instance, for scaling and more importantly for the Azure SLA, see  http://www.windowsazure.com/en-us/support/sla/ 

[André Andersen]

You're right about the SLA there. Allthough you can probably turn off the automatic updating, but if something happens to the rack your node is in - you're in trouble. Are you guys working on a load balanced version of what Mike Hanley is doing in his article?

[Oren Eini (Ayende Rahien)]

Just to let you know, Azure hosting is something that RavenHQ is likely to offer soon

[Matt Warren]

That's cool, it's much better to have an "official" hosted version that just works!

[André Andersen]

Oren, that sounds very interesting.

Can you please tell me more about the plans? We're planning to go into alpha/acceptance-testing very soon, and would love to test any service you might be developing.

[jalchr]

So, what is the solution for have a single role with multiple instances of raven where it gets replicated and scales well in Azure ?

Probably there is  None

On Wednesday, May 2, 2012 12:02:01 PM UTC+3, André Andersen wrote:

You're right about the SLA there. Allthough you can probably turn off the automatic updating, but if something happens to the rack your node is in - you're in trouble. Are you guys working on a load balanced version of what Mike Hanley is doing in his article?

[jalchr]

Excellent ... any estimation for how much "soon" is and any suggestions for possible workarounds in the mean-time ?

[Oren Eini (Ayende Rahien)]

Andre,

We are going to start working on Azure hosting in a couple of weeks or so. The intent it to basically give you the exact same RavenHQ experience, inside Azure.

We haven't drawn exact plans yet for the implementation, but I don't think that this is something that should worry you for now

[Oren Eini (Ayende Rahien)]

For workarounds, see this thread :-)

For soon, I don't want to promise a release date yet. We are going to start working on it in a couple of weeks or so.

Then, it mostly depend on how complex it is. The EC2 one was relatively simple, and I don't expect major issues, but still...

[André Andersen]

Oren, this is great news!

[André Andersen]

Matt, I agree - an official hosted version is what most people would like to see. Especially startups!

[Oren Eini (Ayende Rahien)]

Have independent nodes that replicate to one another.

[André Andersen]

Oren,

How would you solve the load balancing between these nodes if you were to build a website on Azure today?

What I mean is.. Let's say you have X web workers, and Y RavenDB instances that are m-m replicated. As I mentioned earlier in this post, I talked to a guy at Microsoft that is working with Azure, and his general recommendation (without having specific knowledge about RavenDB itself) would be to have the RavenDB in the same hosted service. This means you'll be using internal endpoints and won't be able to utilize the round-robin load balancing that you can use if you have RavenDB in it's own hosted service with the 8080 port opened (which opens up to the whole world).

[Oren Eini (Ayende Rahien)]

Andre,

See:

http://ayende.com/blog/4719/data-roles-dont-scale-up-down-dynamically 

RavenDB is _stateful_, you don't _want_ to do things that way, it make no sense to try to do load balancing in this manner, because the servers contains different information

[Alexander Zeitler]

+1

 

Alex

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Matt Warren
Sent: Mittwoch, 2. Mai 2012 11:11
To: rav...@googlegroups.com
Subject: Re: [RavenDB] Re: Azure Hosting

 

That's cool, it's much better to have an "official" hosted version that just works!

[Chris Marisic]

You want the RavenDB server replication document to know about the internal, non load balanced endpoints. This will prevent a server trying to replicate to itself.

You want the RavenDB client to communicate to the public load balanced end point (you can use authentication/authorization still to limit access). This will then allow the load balancer to distribute the load reasonably between the multiple server roles.

[Chris Marisic]

Although this might require using a web role instead of a worker role, I can't remember if worker roles support public load balanced end points.

[jalchr]

Oren,

Check this out,

http://code.msdn.microsoft.com/windowsazure/Azure-Library-for-83562538  

A lucene AzureDirectory impelmentation that is smart enough to use local cache and support multiple instances in storage blobs ... it thing it should be fairly easy to add a configuration for azure (yes/no) and azure connection string and everything should work.

for example the code here https://github.com/ravendb/ravendb/blob/master/Raven.Database/Indexing/IndexStorage.cs uses FSDirectory ... and could be replace with AzureDirectory

...

 // default AzureDirectory stores cache in local temp folder 
            AzureDirectory azureDirectory = new AzureDirectory(CloudStorageAccount.FromConfigurationSetting("blobStorage"), "TestCatalog6"); 
            bool findexExists = IndexReader.IndexExists(azureDirectory); 
 
            IndexWriter indexWriter = null; 
            while (indexWriter == null) 
            { 
                try 
                { 
                    indexWriter = new IndexWriter(azureDirectory, new StandardAnalyzer(), !IndexReader.IndexExists(azureDirectory)); 
                } 
                catch (LockObtainFailedException) 
                { 
                    Console.WriteLine("Lock is taken, Hit 'Y' to clear the lock, or anything else to try again"); 
                    if (Console.ReadLine().ToLower().Trim() == "y" ) 
                        azureDirectory.ClearLock("write.lock"); 
                } 
            }; 
            Console.WriteLine("IndexWriter lock obtained, this process has exclusive write access to index"); 
            indexWriter.SetRAMBufferSizeMB(10.0); 
            indexWriter.SetUseCompoundFile(false); 
            indexWriter.SetMaxMergeDocs(10000); 
            indexWriter.SetMergeFactor(100); 
             
            for (int iDoc = 0; iDoc < 10000; iDoc++) 
            { 
                if (iDoc % 10 == 0) 
                    Console.WriteLine(iDoc); 
                Document doc = new Document(); 
                doc.Add(new Field("id", DateTime.Now.ToFileTimeUtc().ToString(), Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO)); 
                doc.Add(new Field("Title", GeneratePhrase(10), Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO)); 
                doc.Add(new Field("Body", GeneratePhrase(40), Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO)); 
                indexWriter.AddDocument(doc); 
            } 
            Console.WriteLine("Total docs is {0}", indexWriter.DocCount()); 
            indexWriter.Close(); 
 
            IndexSearcher searcher; 
            using (new AutoStopWatch("Creating searcher")) 
            { 
                searcher = new IndexSearcher(azureDirectory);  
            } 
            SearchForPhrase(searcher, "dog"); 
            SearchForPhrase(searcher, _random.Next(32768).ToString()); 
            SearchForPhrase(searcher, _random.Next(32768).ToString()); 
        } 

[Oren Eini (Ayende Rahien)]

What is this in reference to?

The lucene part of running RavenDB in azure is actually quite simple.

[jalchr]

So ... where is the complex part in this process ?

[Oren Eini (Ayende Rahien)]

Data storage in general, handling multiple nodes, replication to them, etc.

[André Andersen]

Hey Oren,

How are the plans for Azure hosting coming along?

Thanks,
André

[Oren Eini (Ayende Rahien)]

Now that ravenhq is out of beta, we are going to concentrate on this as one of two or three high level features for the next release

[André Andersen]

Do you mean concentrating on Azure or the existing one for Amazon?

We for sure will sign up to RavenHQ the same minute it's available on Azure!

Let me know if you have a timeline for this.

Many Thanks,

André Andersen

[Oren Eini (Ayende Rahien)]

Both, actually.

We are going to offer more regions, and we are going to offer Azure support.

[André Andersen]

Good to hear.

Do you have an estimate/guesstimate when it's possible to try the Azure version? Very excited about this.

[Oren Eini (Ayende Rahien)]

Few months, I guess.

[André Andersen]

Hey Ayende,

Any news on the RavenHQ for Azure?

On Wednesday, August 29, 2012 7:05:10 AM UTC+2, Vitaliy wrote:

Hi

If a working example for multiple instances for windows azure?

воскресенье, 27 мая 2012 г., 16:26:36 UTC+6 пользователь Oren Eini написал:

[Oren Eini (Ayende Rahien)]

The problem with Azure is that the IO speeds to disk are atrocious.

You are better off using the standard RavenHQ, you'll get better speeds.

[David Pfeffer]

Can you give more details about this? We're currently in the development stage of porting our application to RavenDB on Azure, using two VMs that are master-master replicating, with cloud drives -- each VM gets a cloud drive for data and another for indexes.

Is this a bad idea in your experience, and do you have any numbers from when you did testing?

We'd love to use RavenHQ, but we're storing a massive amount of data and pricing would be cost prohibitive.

[André Andersen]

That means that I will have external network traffic and latency instead. Which is kind of bad.

[Chris Marisic]

My organization had to leave Azure VMs because of the disk speed making indexing take large portions of hours instead of minutes. We also had issues where our azure vms would just die and receive basically zero support from Microsoft. The weekend we left Azure i spent downloading my data from azure blob storage because my VM was entirely inaccessible.

[David Pfeffer]

To clarify, do you mean VMs generically (any VM on Azure) or VM as in the new persistent virtual machines?

We're currently planning on using dedicated worker roles rather than the persistent virtual machines, which are currently a preview and have no SLA. Maybe this is why you experienced issues?

[Chris Marisic]

The real VMs, not Azure PAAS. I've had an app using the web roles in azure that to my knowledge has never faulted a single request.

[David Pfeffer]

Is there an easy way (besides writing a bunch of code myself) to run some benchmarks of Raven? I'd like to see some measurements how it will perform.

What I'm hearing here is may be true, but it flies in the face of Microsoft's claim that you can run SQL Server on the new PaaS and get superior performance than hosted SQL Azure. I can't imagine that would be the case if disk I/O is so poor.

[Oren Eini (Ayende Rahien)]

RavenDB is sensitive to IO speed.

That is accurate both in terms of absolute throughput (MB/ sec) and even more in terms of latency ( how long it takes to complete a single IO operation ).

Both are very slow in Azure. This is especially the case when you are doing writes, and even more so when you call fsync, which is required if you want to support transactions.

[Oren Eini (Ayende Rahien)]

They are neighbouring data center, latency isn't going to be high.

[Oren Eini (Ayende Rahien)]

Run a RavenDB instance and put some load on it, writes and indexing in particular. 

You can try using the FreeDB sample set, located in the Raven.Performance project, IIRC.

[David Pfeffer]

I'm launching a medium-size PaaS VM right now, and I'll setup RavenDB with various disk configurations and report back shortly.

Wish me luck. :-)

[Chris Marisic]

SQL Azure is entirely separate from everything else in azure, that's like comparing apples to elephants.

[David Pfeffer]

SQL Azure performs well though (at least in my testing), and Microsoft claims SQL running on PaaS performs better. I'm comparing SQL Server to SQL Server, apples to apples.

[Rudy Hinojosa]

I guess Caching services enabled/disabled also have a big play in any i/o situation.  Appharbor.com is an azure alternative, thats where I found cloudbird, basically ravendb as an add on.  That seems to have pretty good speeds for beta testing.

[Mauro Servienti]

As soon as you change the storage type from munin to esent it fails with:

“Could not open transactional storage: b:\Data\Data”

 

And it fails even on:

NonAdminHttp.EnsureCanListenToWhenInNonAdminContext(port);

 

But I suppose the second one is due to the second role instance being misconfigured and trying to use the same port of the first instance.

 

.m

 

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Sebastian Burgstaller
Sent: martedì 16 ottobre 2012 10.19
To: rav...@googlegroups.com
Subject: [RavenDB] Re: Azure Hosting

 

Hi André!

 

Do you mean hosting RavenDB on the same web role where you host your web site? This would be especially interesting to me as we are a new BisSpark member and get exactly 2 small instances per month for free. So in order to meet the SLA requirement we would have to run RavenDB on both instances. I guess this would require RavenDB to run in embedded mode, or are there other options?

 

BTW: I have found the following implementation of a scaling RavenDB worker role. Whenever you add a new instance it automatically adds itself to the replication document and sets up a new VHD.

https://github.com/SaschaDittmann/RavenDbOnAzure

 

Looks nice to me!

 

Best Regards

Sebastian

On Tuesday, May 1, 2012 3:47:26 AM UTC+2, André Andersen wrote:

I talked to a technical azure evangelist from Microsoft the other day regarding Azure and RavenDB. Allthough he didn't have experience with Raven in particular, his idea was that you would host the RavenDB instances in the same deployment as your web project (or whatever you're doing). This brings me to a question:

- What options do we have when it comes to load balancing RavenDB with replication (I presume master-master replication due to lack of experience).
- What's the best practice when configuring the DocumentStore in this sense, so it uses different instances (load balanced)?

[Oren Eini (Ayende Rahien)]

What is the inner exception for the esent error?

[Mauro Servienti]

“{Disk IO Error}”

[Oren Eini (Ayende Rahien)]

That is it? No further details?

Can you check the event log?

[Mauro Servienti]

In the event log:

 

Raven (7812) b:\Data\Datafc8e4b47-ffa7-4c9b-bc3c-bdf11a9738e1: An attempt to create the folder "b:\" failed with system error 183 (0x000000b7): "Cannot create a file when that file already exists. ".  The create folder operation will fail with error -1022 (0xfffffc02).

 

Where “b:\” is the path of the mounted drive.

[Oren Eini (Ayende Rahien)]

 "b:\"  ??

That isn't a folder, that is a drive letter.

[Oren Eini (Ayende Rahien)]

What is the actual data dir that you defined?

[Mauro Servienti]

Interesting isn’t it? :-)

 

I do not define anything:

-          running locally using the emulator;

-          mounted the cloud drive, and the Azure API returns me as “path” of the drive b:\;

-          created a config:

 

var config = new RavenConfiguration

{

   DataDirectory = _dataDrive.LocalPath.EndsWith("\\")

                                        ? _dataDrive.LocalPath + "Data\\"

                                        : _dataDrive.LocalPath + "\\Data\\",

   AnonymousUserAccessMode = anonymousUserAccessMode,

   HttpCompression = httpCompression,

   DefaultStorageTypeName = defaultStorageTypeName,

   Port = port,

   PluginsDirectory = "Plugins"

};

 

Trying to start the db I get the above exception:

_database = new DocumentDatabase(config);

 

In my environment _dataDrive.LocalPath is “b:\”

 

I’ve not written any code, I’m just playing with the sample from Github:
https://github.com/SaschaDittmann/RavenDbOnAzure

[Oren Eini (Ayende Rahien)]

What happen if you don't use string concat, but Path.Combine?

[Mauro Servienti]

Exactly the same exception.

[Oren Eini (Ayende Rahien)]

Are you available on skype?

[Mauro Servienti]

For a chat, yes, for a conf-call not now, tomorrow the whole day.

[Mauro Servienti]

I’ve solved the issue:

 

The full directory path to the data directory must exists, ESENT will not create it, creating the full path b:\data (by default when the cloud drive is mounted the Data directory does not exists) ESENT does not complain any more.

 

Now I have another issue at: _server.StartListening();

 

It fails adding the listening prefix, but I’ll have a look tomorrow.

 

If you still want to do the conf call tomorrow I’ll be available.

 

Cheers,

.m

[Mauro Servienti]

No way, it does not work.

 

In the end now I have it running with replica setup, the followed steps (and fixes in the downloaded sample) are:

 

-          When running with esent:

o   The “Data” directory must be “manually” created otherwise esent complains with a Disk IO Error;

o   The “Tenants” directory must be “manually” created, same issue as above;

-          The downloaded sample:

o   has a misconfigured endpoint in the role configuration: the endpoint must be http and not tcp;

o   utilizes the RoleEnvironment.CurrentRoleInstance.Id to setup the vhd name but, at least in the development emulator, the id changes at each deployment and is prefixed with the deployment id, thus each time the vhd is new and you lose access to the previous debug session data; I managed to create a vhd name using RoleEnvironment.CurrentRoleInstance.Id value but removing the RoleEnvironment.DeploymentId portion;

-          In my environment running with more than 1 instance breaks everything because of the way the emulator assigns dynamic ports to the second instance, resulting in the choice of an already used port, I suppose it does something like “current port” + 1, thus 8080 + 1 = 8081…exception;

-          I have also removed all the references to the embedded version;

 

The next step is to upgrade everything to build 960, since the sample is running on 888.

 

.m

 

 

From: rav...@googlegroups.com [mailto:rav...@googlegroups.com] On Behalf Of Maverix
Sent: mercoledì 17 ottobre 2012 00.50
To: rav...@googlegroups.com
Subject: Re: [RavenDB] Re: Azure Hosting

 

<Runtime executionContext="elevated" />

 

Add the above line in your ServiceDefinition.csdef in the azure project

Add it just after the definition of the Worker Role you are running ravenDb in

[Mircea Chirea]

Oren,

Any updates on RavenHQ hosting on Azure? I am debating between AppHarbor and Azure for a new project and currently RavenHQ runs in EC2. Will you introduce Azure hosting for 2.0 or will we have to host RavenDB ourselves?

[Oren Eini (Ayende Rahien)]

There will be some hosted solution for RavenDB on Azure, yes.

[Mircea Chirea]

Any ETA on that?

[Oren Eini (Ayende Rahien)]

After 2.0, but not sure by how much, not very long.

[Alexander Kushnir]

Any update on this? maybe a beta?

[Oren Eini (Ayende Rahien)]

We are working with RavenHQ to offer this as part of the Azure Store.

--
You received this message because you are subscribed to the Google Groups "ravendb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.