How to use Smuggler on embedded DB

[Sean Kearon]

I'm trying to use Smuggler to export a single document from an embedded database that does not use the embedded.  I'm getting URI format errors from HttpRavenRequest.  Can this be done?  If so, where am I going wrong in the test below:

Thanks :)

      [Test]
      public void can_export_document_by_id()
      {
         using (var store = new EmbeddableDocumentStore { DataDirectory = "Data", RunInMemory = true, UseEmbeddedHttpServer = false })

         {
            store.Initialize();
 
            var company = new Company {Name = "Sean"};
            using (var session = store.OpenSession())
            {
               session.Store(company);
               session.SaveChanges();
            }
 
            var api = new SmugglerApi(new EmbeddedRavenConnectionStringOptions { DataDirectory = "Data", AllowEmbeddedOptions = true, });
            var options = new SmugglerOptions { File = @"c:\users\sean\desktop\sean.raven" };
            options.Filters.Add("Id", company.Id);
            api.ExportData(options);
         }
      }

[Oren Eini (Ayende Rahien)]

You cannot export from an embedded DB without enabling the embedded http server.

Smuggler works over HTTP.

[Sean Kearon]

Oren

I'm going to be running on the user's desktop and I don't want to have the embedded http server running so that I can lock down the system so that users can't tamper with the data in any way.

Is there any other way I can use to export documents and attachments?

Also, I have just realised that even if I encrypt the documents in the database, the user could download and run Studio against the data.  Is there any way currently of preventing this, such as whole database encryption?  If not, will that be available with the encryption features in 1.2?

Thanks

Sean

[Oren Eini (Ayende Rahien)]

Sean,

If the data is on the user's computer, he got it.

Oh, you can make it awkward to get it, but they can do that. 

There is literally no way to hide that if your are running on his machine.

I think that there is some code around (Tobi or Justin wrote it, I believe) that will work like the smuggler for the embedded version.

[Sean Kearon]

Thanks, I'll look for Tobi or Justin's code.

As for the user accessing the data - so there is no way that I can prevent the user from editing or deleting documents whatever lengths I go to?

On Monday, 30 April 2012 10:34:19 UTC+1, Oren Eini wrote:

Sean,

If the data is on the user's computer, he got it.

Oh, you can make it awkward to get it, but they can do that. 

There is literally no way to hide that if your are running on his machine.

I think that there is some code around (Tobi or Justin wrote it, I believe) that will work like the smuggler for the embedded version.

[Oren Eini (Ayende Rahien)]

At the end, it is a file on his machine.

Sure, you can encrypt that, but the encryption key also has to be on the machine at some point, so that is pretty meaningless.

Who is the user? What is the data? How important is it to prevent tampering? Are you likely to be dealing with devs / hackers or with standard users?

[Sean Kearon]

The end users are electrical engineers as private engineers or companies.  The product is licensed to the PC itself and we do eperience a fair amount of end user tampering.  Part of the licensing and machine locking is tracked in the database itself, so it['s important that they cannot remove this.

The other motivation is to prevent competitors from accessing our data.  To be able to upgrade from our data is an attractive selling point and I'm a small guy in a market where there are some very established players who have far more resources that I do.  That precise scenario happened to me about 10 years ago now, and it hurt!

The encryption key is shipped, yes, but it's obfuscated.  Sure, you can never totally prevent access, but to raise the bar to the level that the attacker has to crack commercial obfuscation is good enough for our needs.  At least I then have legal recourse if that were a competitor.

So, is there any way currently or in the fairly near future to prevent user tampering?  It sounds from your responses that there isn't, which would be a shame. 

On Monday, 30 April 2012 10:45:39 UTC+1, Oren Eini wrote:

At the end, it is a file on his machine.

Sure, you can encrypt that, but the encryption key also has to be on the machine at some point, so that is pretty meaningless.

Who is the user? What is the data? How important is it to prevent tampering? Are you likely to be dealing with devs / hackers or with standard users?

[Oren Eini (Ayende Rahien)]

Sean,

Encrypt the data, that should take you pretty far in only allowing access from your own software.

[Sean Kearon]

Sure, that does prevent a competitor from reading our data.  But it doesn't prevent user tampering nor does it allow me to use the database to track licensing (as it can be tampered with).  There's no way currently around that then?

On Monday, 30 April 2012 14:19:02 UTC+1, Oren Eini wrote:

Sean,

Encrypt the data, that should take you pretty far in only allowing access from your own software.

[Oren Eini (Ayende Rahien)]

Sean,

Let me rephrase that, what do you want? How do you envision it?

[Sean Kearon]

Given that the documents can be encrypted and that (an equivalent to) Smuggler can be run without enabling http, then to require a password to enable http access to the database from would be all that's needed.  

I presume that this, or something similar, is something that you're going to add for the enterprise encryption in 1.2?  

On Monday, 30 April 2012 15:03:19 UTC+1, Oren Eini wrote:

Sean,

Let me rephrase that, what do you want? How do you envision it?

[Oren Eini (Ayende Rahien)]

You _can_ require a password for HTTP access. That is what the OAuth bundle does.

[Sean Kearon]

Fantastic, thanks!  I'll take a look at that now.

On Monday, 30 April 2012 15:48:42 UTC+1, Oren Eini wrote:

You _can_ require a password for HTTP access. That is what the OAuth bundle does.

[Sean Kearon]

Oren, thanks again for your help.  It looks like exactly what I needed.

I'm using OAuth as I don't want the domain to control access.  This sets up a default user and spits out the password to a file (EnsureAtLeastOneUserExists startup task).  What's the best way to prevent this happening?

[Oren Eini (Ayende Rahien)]

Create a user first

Note that embedded access had no security

[Sean Kearon]

Thanks, happy for embedded to have no security from the application, it's just other channels that matter.

As for creating the user first, I have tried to do that, but it's being created from an IStartupTask.  Is there a way to get in before that runs?

[Oren Eini (Ayende Rahien)]

Honestly, let it create the user, then, using the embedded, just modify the password.

Or don't add the Oauth bundle, create your own that implements IAuthenciateClient

[Tobi]

On 01.05.2012 06:00, Oren Eini (Ayende Rahien) wrote:

> Or don't add the Oauth bundle, create your own that implements
> IAuthenciateClient

@Sean: That's what I do in my app as well. I have a fixed password (no
username) to protect access to RavenDB via http:

https://gist.github.com/2565534

The application is a touch application running in "kiosk mode" and has a
service screen where it opens an embedded browser with Raven.Studio. To
avoid the need to enter a password there (The service screen itself
already is password-protected), I temporary set:

DocumentStore.Configuration.AllowLocalAccessWithoutAuthorization = true

Tobias

[Sean Kearon]

Thanks Tobi, that's much cleaner than messing around with the default user from the OAuth bundle.  How do you register the FixedPasswordAuthenticateClient?  I can't find how to do that.

That's an interesting scenario too.  I will be putting Raven onto the desktop, so I want to use AllowLocalAccessWithoutAuthorization = false.  Works a treat for me.

Do you then run Smuggler across http, or have you embedded that too?  If you've embedded it, could you show how you do that?

Many thanks

Sean

[Tobi]

On 01.05.2012 11:44, Sean Kearon wrote:

> Thanks Tobi, that's much cleaner than messing around with the default user
> from the OAuth bundle. How do you register the
> FixedPasswordAuthenticateClient? I can't find how to do that.

I've added it to the Gist code snippet:

https://gist.github.com/2565534

> Do you then run Smuggler across http, or have you embedded that too? If
> you've embedded it, could you show how you do that?

I've embedded this too. It's as simple as this:

https://gist.github.com/617852830394aaaa7160

I use this by default for backups because the app is running on different
versions of WinXP / Win7 which makes the Esent backups pretty useless if
you want to restore a backup on a different machine.

The above code imports/exports the whole database. But I will soon change
this to support incremental exports. The Smuggler from RavenDB/unstable
already supports this, but I'll do it in a slightly different way.

Tobias

[Sean Kearon]

Tobias

Thanks very much, that's been really useful and is much appreciated.  I'm pretty sure that's all I need to look at now, so I'm off to buy myself an embedded licence and look forward to shipping on Raven :)

Cheers

Sean

[Sean Kearon]

Tobi: using your RavenDB Dumper as a basis, I've added attachment capabilities (I have a lot of images in my app).  I've tested this only lightly as yet, and it seems to allow my app to backup and restore very smoothly.  I've posted this here and would appreciate any comments you may have:  https://gist.github.com/2638423 

Oren + team: for my scenario - applying OAuth locally and closing down HTTP access - using Smuggler OOTB isn't really an option as far as I can see.  To have a fully embedded Smuggler API similar to what Tobi has skinned up would be really great for us embedded-on-the-desktop guys.  

On Tuesday, 1 May 2012 11:57:38 UTC+1, Tobi wrote:

[Itamar Syn-Hershko]

It's pretty trivial actually, the important parts are already exposed by the API.

I'm leaning towards asking you to post this as a KB article instead of taking this as a pull request to RavenDB.Embedded

[Tobi]

On 08.05.2012 21:03, Sean Kearon wrote:

> Tobi: using your RavenDB Dumper as a basis, I've added attachment
> capabilities

Great!

> (I have a lot of images in my app).

How many?

I'm going to add support for product images soon, but haven't decided yet
if it's better to store them as attachments or in the file system.

> Oren + team: for my scenario - applying OAuth locally and closing down
> HTTP access - using Smuggler OOTB isn't really an option as far as I can
> see. To have a fully embedded Smuggler API similar to what Tobi has
> skinned up would be really great for us embedded-on-the-desktop guys.

+1 Having import/export in the client-API would certainly be nice.

Tobias

[Sean Kearon]

Apologies for the delayed response - I stupidly decided to fall down a hole on Monday evening and break my foot and I've just come out of hospital. :(

Tobi - about the images: my main root object has up to 5 images.  The root has a child which also has an image and there are typically 50 of these children per root.  So, about 60 images per root currently.  We are going to add a feature to store more images in the near future, which will add up to about about another 200 images per root.  I want to store the images in the database.  That just makes it easier to backup and restore and is a lot cleaner to code, imo.  I also process the images to reduce their size when importing into the application.  The main motivation for this is that we have a data sharing facility and I want to ensure that the size of the exchanged files remains as small as possible.  I've also added a reference back to your original Gist from mine too :)

Itamar - I really think that this should be a core feature and not rely on a KB article.  You should be able to run Smuggler - or similar - in embedded mode without having to enable HTTP.  Using HTTP when Raven is running on the user's desktop throws up issues about user permission levels.  Yes, I know there is support for this in Raven, but in my scenario I specifically do not want to enable access through Studio as I want to prevent the user from accessing the data easily.  How do I add a feature request for this?

[Itamar Syn-Hershko]

A pull request

If you can get it to work with the current smuggler application that would be ideal, otherwise adding this to Raven.Client.Embedded seems like the best bet

[Sean Kearon]

Thanks Itamar.

Tobi - the export facility was your work, which I just benefited from (very happily too ;) ).  I am wondering if you want to do the pull request.  If not, I'm happy to give it a try too.

On Saturday, 12 May 2012 19:05:06 UTC+1, Itamar Syn-Hershko wrote:

A pull request

If you can get it to work with the current smuggler application that would be ideal, otherwise adding this to Raven.Client.Embedded seems like the best bet

[Tobi]

Am 14.05.2012 11:36, schrieb Sean Kearon:

> Tobi - the export facility was your work, which I just benefited from (very
> happily too ;) ). I am wondering if you want to do the pull request. If
> not, I'm happy to give it a try too.

Feel free to do so.

Tobias

[Chris Marisic]

On Saturday, May 12, 2012 6:12:14 AM UTC-4, Sean Kearon wrote:

Itamar - I really think that this should be a core feature and not rely on a KB article.  You should be able to run Smuggler - or similar - in embedded mode without having to enable HTTP.  Using HTTP when Raven is running on the user's desktop throws up issues about user permission levels.  Yes, I know there is support for this in Raven, but in my scenario I specifically do not want to enable access through Studio as I want to prevent the user from accessing the data easily.  How do I add a feature request for this?

I don't use embedded but i certainly agree with Sean that this feature should be support OOTB.

[Itamar Syn-Hershko]

Yes, waiting on a pull request

[Sean Kearon]

I'm working on it :)

On Monday, 14 May 2012 16:00:57 UTC+1, Itamar Syn-Hershko wrote:

Yes, waiting on a pull request

[Sean Kearon]

Itamar

Having taken a look at this, can I ask for some design guidance?  The situation so far is:

To look at putting non-HTTP import/export functionality into Raven.Smuggler, I need to use the following parts of DocumentDatabase:

GetIndexes()

GetDocuments()

PutIndex()

PutStatic();

Batch()

GetAttachments()

GetStatic()

TransactionalStorage.Batch()

However, Raven.Smuggler currently takes a dependency only on Raven.Abstractions.  Adding a depencency to Raven.Database does not seem suitable to me.  So, the options are:

1. Put the non-HTTP import/export functionality into Raven.Client.Embedded.
   
2. Add an abstraction to Raven.Abstractions to allow access to the above API.
   

The benefits of the first are that it's simple and the non-HTTP import/export does mainly relate to that namespace.  The disadvantage is that there would then be two sections of code that handled import/export.

Adding an abstraction does really seem to be the best way to go to keep the API clean.  If you agree, can you suggest how this could be approached?

On Saturday, 12 May 2012 19:05:06 UTC+1, Itamar Syn-Hershko wrote:

[Oren Eini (Ayende Rahien)]

Since when running in embedded mode, you are already doing this inside your app, is there a reason not to add this to the Raven.Database project?

[Sean Kearon]

Okay, thanks.

On Thursday, 17 May 2012 00:58:49 UTC+1, Oren Eini wrote:

Since when running in embedded mode, you are already doing this inside your app, is there a reason not to add this to the Raven.Database project?

[Sean Kearon]

Pull request here.

[tur...@gmail.com]

Im using RavenDb as service. I did not SmugglerApi in Raven.Abstractions.Smuggler namespace. what am i missing?

On Sunday, April 29, 2012 9:04:53 PM UTC-7, Oren Eini wrote:

You cannot export from an embedded DB without enabling the embedded http server.

Smuggler works over HTTP.

On Mon, Apr 30, 2012 at 2:52 AM, Sean Kearon <kearo...@googlemail.com> wrote:

I'm trying to use Smuggler to export a single document from an embedded database that does not use the embedded.  I'm getting URI format errors from HttpRavenRequest.  Can this be done?  If so, where am I going wrong in the test below:

Thanks :)

      [Test]
      public void can_export_document_by_id()
      {
         using (var store = new EmbeddableDocumentStore { DataDirectory = "Data", RunInMemory = true, UseEmbeddedHttpServer = false })

         {
            store.Initialize();
 
            var company = new Company {Name = "Sean"};
            using (var session = store.OpenSession())
            {
               session.Store(company);
               session.SaveChanges();
            }
 
            var api = new SmugglerApi(new EmbeddedRavenConnectionStringOptions { DataDirectory = "Data", AllowEmbeddedOptions = true, });
            var options = new SmugglerOptions { File = @"c:\users\sean\desktop\sean.raven" };
            options.Filters.Add("Id", company.Id);
            api.ExportData(options);
         }
      }

[Oren Eini (Ayende Rahien)]

A reference to Raven.Smuggler

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-674-7811

Fax:      + 972-153-4622-7811

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[turkush .]

I'v reffered Raven.Abstraction.dll and Raven.Client.Lightweight.dll. 

I dont have Raven.Smuggler rather Ive  Raven.Abstractions.Smuggler;

--
You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/Yfk81TcK9UU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Sense ur odd moments as reaction of ur odd deeds, gradually u'll find Allah...

[Oren Eini (Ayende Rahien)]

You need to reference the Raven.Smuggler.exe 

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-674-7811

Fax:      + 972-153-4622-7811