JClouds BlobStore features

[monster]

I stumbled upon AppScale today, and saw that there was a certain
amount of similarity with JClouds, which I had bookmarked weeks ago,
so I decided to have a closer look and compare them.

While AppScale is not appropriate for me because I'm not doing a web-
app, it seems that their Blobstore has more features than the one from
JClouds, and that I would need those missing features.

As far as I can tell, you can do object versionning in AppScale, which
is a form of "append", and you can also read only a byte "range" from
a value, instead of the whole value.

Since I want to keep an "history" of all changes to my data, and be
able to do selective rollback, not just in case of crashes, but also
in case a user "misbehave", being able to append changes as a new
version, and only reading the last version at startup (range read),
would fit my use-case perfectly.

Is this something that is currently possible with the JClouds
Blobstore? If not, is it a planned feature, or something that is
unlikely/impossible to implement? It seems that AppScale Blobstore is
based on "NoSQL" DBs, while JClouds Blobstore is based on "Cloud
Storage", so I suppose the possible features might not fully overlap.

Also, I want to do a (fat) client-server app, and I would like to be
able to reuse the same code on both sides, so it would be useful if I
had a local, embedded, version of the Blobstore that I could run in
the client app, for "local caching"/"offline work". Is something like
that possible?

[Andrew Phillips]

> I stumbled upon AppScale today, and saw that there was a certain
> amount of similarity with JClouds, which I had bookmarked weeks ago,
> so I decided to have a closer look and compare them.
>
> While AppScale is not appropriate for me because I'm not doing a web-
> app, it seems that their Blobstore has more features than the one from
> JClouds, and that I would need those missing features.
>
> As far as I can tell, you can do object versionning in AppScale, which
> is a form of "append", and you can also read only a byte "range" from
> a value, instead of the whole value.
>
> Since I want to keep an "history" of all changes to my data, and be
> able to do selective rollback, not just in case of crashes, but also
> in case a user "misbehave", being able to append changes as a new
> version, and only reading the last version at startup (range read),
> would fit my use-case perfectly.

Of you thought of using an SCM like svn or git for storage? There
would seem to be pretty good coverage in terms of the desired feature
set, and I see some people have evidently already tried to do this [1].

You're indeed correct in stating that most of the implementers of the
jclouds BlobStore API are cloud storage services (many with limited or
no support for things like appending).

That doesn't mean, of course, that it wouldn't be possible to add an
API or provider for something that *does* cover your needs.

For local blob storage you could have a look at the Filesystem API [2].

Regards

ap

[1] http://www.google.com/search?q=git+on+s3
[2] https://github.com/jclouds/jclouds/tree/master/apis/filesystem

[Andrew Phillips]

> Is this something that is currently possible with the JClouds
> Blobstore? If not, is it a planned feature, or something that is
> unlikely/impossible to implement? It seems that AppScale Blobstore is
> based on "NoSQL" DBs, while JClouds Blobstore is based on "Cloud
> Storage", so I suppose the possible features might not fully overlap.

PS: I notice AppScale refers to "Datastores" on their site rather than
"Blobstores". The supported stores are indeed more complex things than
most "simple" cloud storage services, supporting more of features you
need for your application.

ap

[1] http://appscale.cs.ucsb.edu/datastores.html

[Adrian Cole]

Hi, Monster.  Thanks for stopping by :)

Notes inline

On Mar 3, 2012 1:45 PM, "monster" <skunki...@googlemail.com> wrote:
>
> I stumbled upon AppScale today, and saw that there was a certain
> amount of similarity with JClouds, which I had bookmarked weeks ago,
> so I decided to have a closer look and compare them.
>
> While AppScale is not appropriate for me because I'm not doing a web-
> app, it seems that their Blobstore has more features than the one from
> JClouds, and that I would need those missing features.
>
> As far as I can tell, you can do object versionning in AppScale, which
> is a form of "append", and you can also read only a byte "range" from
> a value, instead of the whole value.

Object versioning is supported on a good deal of BlobStores now (at least aws-s3 & swift iirc), so we could look into writing portable support for it in jclouds.

Append is not portable.

Reading a byte range is already supported in GetOptions

>
> Since I want to keep an "history" of all changes to my data, and be
> able to do selective rollback, not just in case of crashes, but also
> in case a user "misbehave", being able to append changes as a new
> version, and only reading the last version at startup (range read),
> would fit my use-case perfectly.
>
> Is this something that is currently possible with the JClouds
> Blobstore? If not, is it a planned feature, or something that is
> unlikely/impossible to implement? It seems that AppScale Blobstore is
> based on "NoSQL" DBs, while JClouds Blobstore is based on "Cloud
> Storage", so I suppose the possible features might not fully overlap.

Probably pushing the limits of a simple BlobStore, also you'll have to consider eventual consistency, but seems worth investigating.

>
> Also, I want to do a (fat) client-server app, and I would like to be
> able to reuse the same code on both sides, so it would be useful if I
> had a local, embedded, version of the Blobstore that I could run in
> the client app, for "local caching"/"offline work". Is something like
> that possible?

Yep.  Though we'd need to implement versioning for filesystem and mem (git?)

>
> --
> You received this message because you are subscribed to the Google Groups "jclouds" group.
> To post to this group, send email to jcl...@googlegroups.com.
> To unsubscribe from this group, send email to jclouds+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/jclouds?hl=en.
>

[monster]

I have to use SVN at work. It's slow as hell! I've her GIT is pretty fast, but using a SCM as database sounds more like a hack than a solution. I think I would rather just use plain files.

On Sunday, March 4, 2012 11:26:46 AM UTC+1, Andrew Phillips wrote:

would seem to be pretty good coverage in terms of the desired feature  
set, and I see some people have evidently already tried to do this [1].

Of you thought of using an SCM like svn or git for storage? There  

You're indeed correct in stating that most of the implementers of the  
jclouds BlobStore API are cloud storage services (many with limited or  
no support for things like appending).

That doesn't mean, of course, that it wouldn't be possible to add an  
API or provider for something that *does* cover your needs.

Well, given enough effort, I could probably add those features to the existing code, but I wanted to know if they would be "compatible" blobstore API, or if I should just ignore it go for some native interface.
 

For local blob storage you could have a look at the Filesystem API [2].

I've seen it mentioned on the JClouds site, but reading some posts here, I had the impression that it was being "dropped"/"orphaned". Maybe I got it wrong.

Regards

ap

[1] http://www.google.com/search?q=git+on+s3
[2] https://github.com/jclouds/jclouds/tree/master/apis/filesystem

PS: Not sure how my reply will look like; the new Google Groups interface is behaving weirdly...

[monster]

On Sunday, March 4, 2012 3:13:40 PM UTC+1, Adrian Cole wrote:

Object versioning is supported on a good deal of BlobStores now (at least aws-s3 & swift iirc), so we could look into writing portable support for it in jclouds.

Append is not portable.

If I can create "new versions", and have access to specific previous versions, and can delete old versions, then that will work as an "append" for me. I'm most comfortable with Ubuntu, and I wanted to use OpenStack to build my "private cloud", so Swift sounds like a good match, but I don't know precisely what features it supports.

Reading a byte range is already supported in GetOptions

Great!

Probably pushing the limits of a simple BlobStore, also you'll have to consider eventual consistency, but seems worth investigating.

 In my planed architecture, each peace of data would be owned by a specific server (based on some hashing, possibly duplicated on a warm-standby), so that there are no concurrent write access to the *same* piece of data, and therefore no consistency problem.

[Adrian Cole]

looks like versions are planned for OpenStack Essex, but not quite in.
We should track this and then there will be at least 2 major clouds
with versioned blobs:
https://blueprints.launchpad.net/swift/+spec/version-manifest

-A

> --
> You received this message because you are subscribed to the Google Groups
> "jclouds" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/jclouds/-/MhDvKlg50v4J.

[mdiggory]

Hello,

Atmos supports object versioning, we are interested in seeing jclouds have versioning support on it.

Mark

[Adrian Cole]

Hi, Mark.

We've had a similar discussion about S3. would the below approach work for you?  Do you have time to help?

https://groups.google.com/forum/m/#!msg/jclouds/2A5Y4AZT8mo/V1zEx85_650J

http://code.google.com/p/jclouds/issues/detail?id=143&q=versioned

--

You received this message because you are subscribed to the Google Groups "jclouds" group.

To view this discussion on the web visit https://groups.google.com/d/msg/jclouds/-/z0FtftukXz0J.