memcached and access control
KaiGai Kohei
I could find the following entry in the FAQ.
http://code.google.com/p/memcached/wiki/FAQ
| * How does memcached's authentication mechanisms work?
|
| It doesn't! Memcached is the soft, doughy underbelly of your application.
| Part of what makes the clients and server lightweight is the complete lack
| of authentication. New connections are fast, and server configuration is
| nonexistent.
But the development roadmap says as follows:
http://code.google.com/p/memcached/wiki/DevelopmentRoadmap
| * Development Branch: 1.5
| - protocol extension support
| think mod_auth for type of mechanism to add to core
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Amazing!
Is these any design proposals?
Or, could you introduce me who is working on this efforts?
I've worked on development of secure web application platform using SELinux
for a few years. Nowadays, memcached becomes a significant facility for
various kind of web applications, so we cannot ignore access controls on
the key-value store shared by multiple web applications.
So, I'm interested in the description on the roadmap, and looking for more
detailed information about this project.
Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei <kai...@ak.jp.nec.com>
Dustin
On Jan 5, 10:06 pm, KaiGai Kohei <kai...@ak.jp.nec.com> wrote:
> Is these any design proposals?
> Or, could you introduce me who is working on this efforts?
>
> I've worked on development of secure web application platform using SELinux
> for a few years. Nowadays, memcached becomes a significant facility for
> various kind of web applications, so we cannot ignore access controls on
> the key-value store shared by multiple web applications.
>
> So, I'm interested in the description on the roadmap, and looking for more
> detailed information about this project.
I suppose we should update those docs a bit:
http://code.google.com/p/memcached/wiki/SASLHowto
Let me know how that goes.
KaiGai Kohei
Thanks for the information.
Hmm, indeed, memcached already provides authentication feature, but it is
different from what I would like to do.
It seems to me it allows authenticated clients to access all the objects
stored in this memcached server. However, we cannot control accesses on
certain objects like filesystem permissions, although SASL support enables
to identify the client.
(BTW, access control does not always require authentication. For example,
we can assume a security model based on the source ip addresses.)
Is there any activity to support access controls, not only authentication?
Or, is it open for new idea or proposition? :)
Henrik Schröder
/Henrik Schröder
pub crawler
kingdoms would have implied effect of slowing memcached down and
adding unnecessary complexity. Unsure how much either would truly
impact it however. Someone might want to substantiate this point.
If you have a memcached need for multiple clients or different sites
you are best running separate instances and using other 3rd party
methods for attempting to secure memcached.
Needless to say, permissions and authentication is a feature set that
is going to re-requested for addition now and in the future. It opens
the door for someone to create a memcached variation with such a
feature set - anyone?
Aaron Stone
> Obviously dissecting a memcached instance up into separate user
> kingdoms would have implied effect of slowing memcached down and
> adding unnecessary complexity. Unsure how much either would truly
> impact it however. Someone might want to substantiate this point.
>
> If you have a memcached need for multiple clients or different sites
> you are best running separate instances and using other 3rd party
> methods for attempting to secure memcached.
>
> Needless to say, permissions and authentication is a feature set that
> is going to re-requested for addition now and in the future. It opens
> the door for someone to create a memcached variation with such a
> feature set - anyone?
Please don't. Please nobody even think of doing that. Really. Don't.
Matt Ingenthron
>> Needless to say, permissions and authentication is a feature set that
>> is going to re-requested for addition now and in the future. It opens
>> the door for someone to create a memcached variation with such a
>> feature set - anyone?
>>
>
> Please don't. Please nobody even think of doing that. Really. Don't.
>
>
Authentication, at least, has been in the community memcached release
for the last two micro releases:
http://code.google.com/p/memcached/wiki/ReleaseNotes143
If you need security labels and such, running separate processes would
seem to be the way to go. Of course, security labels can also be
applied to network traffic, meaning that the authentication features are
redundant. :)
- Matt
KaiGai Kohei
> Access controls on a per-key basis is insane for lots of reasons. If you
> need separate applications to only be able to access their own keys, set
> up a separate memcached instance for each app. Problem solved without
> incurring the access control overhead, without introducing access
> control syntax, and without enabling apps to break each other by
> accidentally reserving each other's keys.
I assume access control feature shall be optional, even if we implement
this feature in the memcached. Basically, security needs its tradeoff
more or less, not free.
Your proposition is an idea, but it needs a few assumptions.
- We don't need to consider any cases except for the strict-separation.
If administrator want to set up read-only privileges for other users,
this idea does not allow such a flexibility.
- We need to assume web applications are bug/vulnerability free.
The web-apps can decide what memcached instance should be used.
However, the total code size of web-apps is much larger than
memcached, so it might not be impossible, but it is hard.
Thanks,
> /Henrik Schröder
>
> 2010/1/6 KaiGai Kohei <kai...@ak.jp.nec.com <mailto:kai...@ak.jp.nec.com>>
>
> (2010/01/06 15:14), Dustin wrote:
> >
> > On Jan 5, 10:06 pm, KaiGai Kohei<kai...@ak.jp.nec.com
> KaiGai Kohei <kai...@ak.jp.nec.com <mailto:kai...@ak.jp.nec.com>>
KaiGai Kohei
> Obviously dissecting a memcached instance up into separate user
> kingdoms would have implied effect of slowing memcached down and
> adding unnecessary complexity. Unsure how much either would truly
> impact it however. Someone might want to substantiate this point.
>
> If you have a memcached need for multiple clients or different sites
> you are best running separate instances and using other 3rd party
> methods for attempting to secure memcached.
In the case when we can assume all the data are strictly separated,
it is a reasonable approach to run separate memcached instances.
However, it does not cover all the cases.
> Needless to say, permissions and authentication is a feature set that
> is going to re-requested for addition now and in the future. It opens
> the door for someone to create a memcached variation with such a
> feature set - anyone?
Sorry, it is unclear for me whether you suggest me to develop my own
branch, or to work in the upstream.
Basically, I'm not interested in making my branch apart from the
community, because it does not utilize the benefit from OSS.
Thanks,
KaiGai Kohei
> On Wed, Jan 6, 2010 at 2:48 AM, pub crawler<pubcraw...@gmail.com> wrote:
>> Obviously dissecting a memcached instance up into separate user
>> kingdoms would have implied effect of slowing memcached down and
>> adding unnecessary complexity. Unsure how much either would truly
>> impact it however. Someone might want to substantiate this point.
>>
>> If you have a memcached need for multiple clients or different sites
>> you are best running separate instances and using other 3rd party
>> methods for attempting to secure memcached.
>>
>> Needless to say, permissions and authentication is a feature set that
>> is going to re-requested for addition now and in the future. It opens
>> the door for someone to create a memcached variation with such a
>> feature set - anyone?
>
> Please don't. Please nobody even think of doing that. Really. Don't.
Sorry, It is unclear for me what you opposed to.
(Sorry for the stupid reply, since English is not my native language...)
If you opposed to making my own branch for access control purpose,
I can understand its reason.
If you opposed to add (optional) access control feature in the memcached,
I'd like to see the reason why.
Thanks,
KaiGai Kohei
> Aaron Stone wrote:
>> On Wed, Jan 6, 2010 at 2:48 AM, pub crawler <pubcraw...@gmail.com>
>> wrote:
> (snip...)
>>> Needless to say, permissions and authentication is a feature set that
>>> is going to re-requested for addition now and in the future. It opens
>>> the door for someone to create a memcached variation with such a
>>> feature set - anyone?
>>
>> Please don't. Please nobody even think of doing that. Really. Don't.
>>
>
> Authentication, at least, has been in the community memcached release
> for the last two micro releases:
> http://code.google.com/p/memcached/wiki/ReleaseNotes143
>
> If you need security labels and such, running separate processes would
> seem to be the way to go.
Yes, it is an approach to enforce strict separation between users.
However, there are a few assumptions:
- We don't need to consider any cases except for the strict-separation.
- We need to assume web applications are bug/vulnerability free.
In some cases, it is not suitable for the purpose. :(
The development roadmap mentions about a modular approach in storage
engine support. It seems to me similar approach may be possible in
various kind of access control models.
> Of course, security labels can also be applied
> to network traffic, meaning that the authentication features are
> redundant. :)
Yes, SELinux provides an API to obtain privilege of the peer process
for the given socket file descriptor (IIRC, Solaris also has similar
API). We can use it instead of the authenticated username.
Aaron Stone
> Aaron Stone wrote:
>>
>> On Wed, Jan 6, 2010 at 2:48 AM, pub crawler <pubcraw...@gmail.com>
>> wrote:
>>
>>>
>>>
>
> (snip...)
>>>
>>> Needless to say, permissions and authentication is a feature set that
>>> is going to re-requested for addition now and in the future. It opens
>>> the door for someone to create a memcached variation with such a
>>> feature set - anyone?
>>>
>>
>> Please don't. Please nobody even think of doing that. Really. Don't.
>>
>>
>
> Authentication, at least, has been in the community memcached release for
> the last two micro releases:
> http://code.google.com/p/memcached/wiki/ReleaseNotes143
It's a reasonable effort at keeping someone from writing to your
cache. Surely they can still sniff the network to get your data.
memcached with encryption is like a Ferrari with monster truck tires.
Aaron Stone
> (2010/01/07 9:14), Matt Ingenthron wrote:
>> Aaron Stone wrote:
>>> On Wed, Jan 6, 2010 at 2:48 AM, pub crawler <pubcraw...@gmail.com>
>>> wrote:
>> (snip...)
>>>> Needless to say, permissions and authentication is a feature set that
>>>> is going to re-requested for addition now and in the future. It opens
>>>> the door for someone to create a memcached variation with such a
>>>> feature set - anyone?
>>>
>>> Please don't. Please nobody even think of doing that. Really. Don't.
>>>
>>
>> Authentication, at least, has been in the community memcached release
>> for the last two micro releases:
>> http://code.google.com/p/memcached/wiki/ReleaseNotes143
>>
>> If you need security labels and such, running separate processes would
>> seem to be the way to go.
>
> Yes, it is an approach to enforce strict separation between users.
If "users" means users of your site, then are you going to apply
per-user access controls to the rows in your database, too?
If "users" means admins running their own web apps, then run separate
instances. There are many other ways for multiple users of the same
memcached instance to cause each other trouble. Separate instances
with SASL auth to keep each to their own instance is the way to go at
this time.
KaiGai Kohei
> 2010/1/6 KaiGai Kohei<kai...@ak.jp.nec.com>:
>> (2010/01/07 9:14), Matt Ingenthron wrote:
>>> Aaron Stone wrote:
>>>> On Wed, Jan 6, 2010 at 2:48 AM, pub crawler<pubcraw...@gmail.com>
>>>> wrote:
>>> (snip...)
>>>>> Needless to say, permissions and authentication is a feature set that
>>>>> is going to re-requested for addition now and in the future. It opens
>>>>> the door for someone to create a memcached variation with such a
>>>>> feature set - anyone?
>>>>
>>>> Please don't. Please nobody even think of doing that. Really. Don't.
>>>>
>>>
>>> Authentication, at least, has been in the community memcached release
>>> for the last two micro releases:
>>> http://code.google.com/p/memcached/wiki/ReleaseNotes143
>>>
>>> If you need security labels and such, running separate processes would
>>> seem to be the way to go.
>>
>> Yes, it is an approach to enforce strict separation between users.
>
> If "users" means users of your site, then are you going to apply
> per-user access controls to the rows in your database, too?
Yes, see the page.14 of the slides:
http://sepgsql.googlecode.com/files/JLS2009-KaiGai-LAPP_SELinux.pdf
Web users are authorized by apache/httpd and we can restrict privileges
of web applications prior to its launch. We can restrict accesses to the
filesystem, database and probably cached data based on the restricted
privileges.
> If "users" means admins running their own web apps, then run separate
> instances. There are many other ways for multiple users of the same
> memcached instance to cause each other trouble. Separate instances
> with SASL auth to keep each to their own instance is the way to go at
> this time.
It is not my intention. Sorry for this confusion.
Thanks,
Aaron Stone
> (2010/01/07 11:07), Aaron Stone wrote:
>> 2010/1/6 KaiGai Kohei<kai...@ak.jp.nec.com>:
>>> (2010/01/07 9:14), Matt Ingenthron wrote:
>>>> Aaron Stone wrote:
>>>>> On Wed, Jan 6, 2010 at 2:48 AM, pub crawler<pubcraw...@gmail.com>
>>>>> wrote:
>>>> (snip...)
>>>>>> Needless to say, permissions and authentication is a feature set that
>>>>>> is going to re-requested for addition now and in the future. It opens
>>>>>> the door for someone to create a memcached variation with such a
>>>>>> feature set - anyone?
>>>>>
>>>>> Please don't. Please nobody even think of doing that. Really. Don't.
>>>>>
>>>>
>>>> Authentication, at least, has been in the community memcached release
>>>> for the last two micro releases:
>>>> http://code.google.com/p/memcached/wiki/ReleaseNotes143
>>>>
>>>> If you need security labels and such, running separate processes would
>>>> seem to be the way to go.
>>>
>>> Yes, it is an approach to enforce strict separation between users.
>>
>> If "users" means users of your site, then are you going to apply
>> per-user access controls to the rows in your database, too?
>
> Yes, see the page.14 of the slides:
> http://sepgsql.googlecode.com/files/JLS2009-KaiGai-LAPP_SELinux.pdf
Neat, you're really serious! Ok, so two ways I can see going about this:
Use SASL authentication to validate each "web user" and then tag each
key in memcached with a hash that can be verified to be owned by the
user. Get and Set would check this tag before allowing a key to be
written -- or build compound keys where the hash is the first part, so
that separate user data exists each in its own namespace. In this
case, you don't need protocol changes, only server changes to do more
with the SASL information.
Or, similar but without the SASL part, tagging each key with some
secret value. If you have protected the database rows, then "SELECT *
FROM secret_memcached_keys" will only return the keys the user is
allowed to see. They would use those keys in a way similar to the CAS
values; operations are only allowed to proceed if the secret tags
match. In this case, you need a protocol change to carry the extra
values.
Anyhow, I'm not even close to a core contributer around here, so those
are just some thoughts.
Matt Ingenthron
>> Authentication, at least, has been in the community memcached release for
>> the last two micro releases:
>> http://code.google.com/p/memcached/wiki/ReleaseNotes143
>>
>
> It's a reasonable effort at keeping someone from writing to your
> cache. Surely they can still sniff the network to get your data.
> memcached with encryption is like a Ferrari with monster truck tires.
>
>
Oops, I see I brought up a few things that were already in the thread,
sorry!
I think I agree here, but it may depend. In my opinion, with most
things security related, the level of controls inserted should be
proportional to the anticipated threat. The SASL implementation was put
into memcached (and libmemcached and spymemcached) with a use case in mind.
The very real use case that some people were trying to deal with in
cloud deployments (and possibly some cloudy enterprise deployments) is a
situation where it'd be hard to sniff the network, but it would be
useful to ensure you know who someone is when they connect. This was
covered back when it was released:
http://code.google.com/p/memcached/wiki/SASLHowto
http://blog.northscale.com/northscale-blog/2009/11/sasl-memcached-now-available.html
http://blogs.sun.com/trond/entry/sasl_support_in_libmemcached
There are some who run on clouds (public or private) who may need this
kind of thing. There are others who run on pretty well controlled
networks that don't need any of it. Right now, memcached provides both.
I could see situations where one may need additional controls, but the
kind of security controls that started the thread off (namely SELinux)
which to me means security labels, usually means you don't trust
non-audited programs and instead have the platform look after them. I
guess KaiGai has taken that further with other application components
though.
Getting down to the item level would be tough to accept due to the
overhead involved, one would think though. There may be some ways of
getting closer to access control though without going down to the item
level.
- Matt
dormando
> involved, one would think though. There may be some ways of getting closer to
> access control though without going down to the item level.
This seems like it'll be solved by an engine dealio. Mapping a user to an
internal instance of the storage system. Sort of like running multiple
instances, *cough*. Getting super granular access levels for a web cache
(more than a few dozen users) would be insane, but if someone really wants
to they could implement a storage engine for it.
It'd be incredibly inefficient on memory, compared to keeping the number
of users down or even running multiple instances.
-Dormando
Matt Ingenthron
>> Getting down to the item level would be tough to accept due to the overhead
>> involved, one would think though. There may be some ways of getting closer to
>> access control though without going down to the item level.
>>
>
> This seems like it'll be solved by an engine dealio. Mapping a user to an
> internal instance of the storage system. Sort of like running multiple
> instances, *cough*. Getting super granular access levels for a web cache
> (more than a few dozen users) would be insane, but if someone really wants
> to they could implement a storage engine for it.
>
My thoughts exactly. That's what engine is for!
> It'd be incredibly inefficient on memory, compared to keeping the number
> of users down or even running multiple instances.
Only if you were trying to go all the way down to the item level. It's
possible to have groups of slabs that are dedicated to one label/auth or
something like that, right?
- Matt
KaiGai Kohei
>>> If "users" means users of your site, then are you going to apply
>>> per-user access controls to the rows in your database, too?
>>
>> Yes, see the page.14 of the slides:
>> http://sepgsql.googlecode.com/files/JLS2009-KaiGai-LAPP_SELinux.pdf
>
> Neat, you're really serious! Ok, so two ways I can see going about this:
>
> Use SASL authentication to validate each "web user" and then tag each
> key in memcached with a hash that can be verified to be owned by the
> user. Get and Set would check this tag before allowing a key to be
> written -- or build compound keys where the hash is the first part, so
> that separate user data exists each in its own namespace. In this
> case, you don't need protocol changes, only server changes to do more
> with the SASL information.
It is my preference rather than the second idea, because it requires
application to handle the security tag explicitly for each accesses,
but most of access control feature generally performs below the surface.
For example, we don't provide filesystem permissions any additional
credentials for each read(2) or write(2) system call.
If we don't assume a certain security model, in other word, if memcached
provides a pluggable access control mechanism, it always does not need
SASL authentication. For example, SELinux's security label of the peer
process is a candidate.
If we can identify the client (web application instance) in some way,
memcached will be able to store the something identifier of the cached
object. It may be called owner of the cached object (depending on the
security model).
Then, access control mechanism can provide its access control decision
based on the client's identifier and object's identifier.
It basically does not need protocol changes. However, we may want
a new command to set/get security properties (or not only security
properties) of the cached objects in the future.
Thanks,
dormando
> > of users down or even running multiple instances.
>
> Only if you were trying to go all the way down to the item level. It's
> possible to have groups of slabs that are dedicated to one label/auth or
> something like that, right?
A few dozen users, yeah. Giving each registered user on a website his own
private cache is just silly. Maybe in ten years when terabytes of ram are
cheap.
KaiGai Kohei
> dormando wrote:
>>> Getting down to the item level would be tough to accept due to the
>>> overhead
>>> involved, one would think though. There may be some ways of getting
>>> closer to
>>> access control though without going down to the item level.
>>
>> This seems like it'll be solved by an engine dealio. Mapping a user to an
>> internal instance of the storage system. Sort of like running multiple
>> instances, *cough*. Getting super granular access levels for a web cache
>> (more than a few dozen users) would be insane, but if someone really
>> wants
>> to they could implement a storage engine for it.
>
> My thoughts exactly. That's what engine is for!
Does the engine mean something pluggable or modular?
If so, it is similar to what I think.
One question. Is the storage engine an appropriate layer for access
controls on item level?
It might be possible to provide strict separation between users.
Is it possible to provide read-only or append-only rules for a certain
users in the item level granularity?
>> It'd be incredibly inefficient on memory, compared to keeping the number
>> of users down or even running multiple instances.
>
> Only if you were trying to go all the way down to the item level. It's
> possible to have groups of slabs that are dedicated to one label/auth or
> something like that, right?
It may be possible. However, if the usage of slabs are not balanced,
it may remove cached object, although we have unallocated cache in
the slab owned by other label/auth. right?
Toru Maesaka
Looks like I've jumped on this band wagon a little late but allow me
to throw in my thoughts. Firstly, I'm totally against item-level
authentication.
I see KaiGai Kohei's motivation to want this but in the context of
memcached, this is the applications job. I think it could upset mid to
large scale memcached users if we added this. They're using it because
they love the simplicity and the high throughput of memcached. They
can take care of security themselves. I guess things are different in
a hosting environment but there are different solutions for this which
Dustin has been looking into.
So the question is, is it worth the effort for the memcached project?
> One question. Is the storage engine an appropriate layer for access
> controls on item level?
Yes. This wouldn't be too difficult to pull off with Binary Protocol
+ Engine API. You could create a scramle buffer and send that with the
most appropriate binary protocol field (worst case, use the reserve
field). This scramble buffer can be appended into the key of the item
inside your engine. So you could pull this off with keys with the
following structure:
<key_value><scramble_buffer> (or vice versa).
Pretty much you're free to do anything in the engine. Memcached is
only a darn efficient network interface and protocol parser in the
context of Engine API.
Cheers,
Toru
2010/1/7 KaiGai Kohei <kai...@ak.jp.nec.com>:
--
Toru Maesaka <d...@torum.net>
Toru Maesaka
I mentioned is not that good. It limits access to a certain item to
one user. A better approach would be to create a security map that
associates scrambled data with items.
Cheers,
Toru
--
Toru Maesaka <d...@torum.net>
KaiGai Kohei
> Yo all,
>
> Looks like I've jumped on this band wagon a little late but allow me
> to throw in my thoughts. Firstly, I'm totally against item-level
> authentication.
Please use the correct term to avoid confusion :(
Authentication is a different concept from access control.
> I see KaiGai Kohei's motivation to want this but in the context of
> memcached, this is the applications job. I think it could upset mid to
> large scale memcached users if we added this. They're using it because
> they love the simplicity and the high throughput of memcached. They
> can take care of security themselves. I guess things are different in
> a hosting environment but there are different solutions for this which
> Dustin has been looking into.
>
> So the question is, is it worth the effort for the memcached project?
Obviously, it is valueable.
My proposition is just a framework to host an additional access control
feature without any assumption for security models. In other word, the
default installation does not apply any access controls, unless
administrator explicitly set up its own security module.
Please note that any security solution has its own assumption, so the
preferable answer depending on the situation.
In our assumption, it is difficult to keep application bug/vulnerability
free, so we need to minimize the number of points to be checked.
It is also why operating system checks filesystem permission on the system
call invocations, not application side.
If we can trust applications, it is not a wrong approach to control
all the things in the application, not memcached.
>> One question. Is the storage engine an appropriate layer for access
>> controls on item level?
>
> Yes. This wouldn't be too difficult to pull off with Binary Protocol
> + Engine API. You could create a scramle buffer and send that with the
> most appropriate binary protocol field (worst case, use the reserve
> field). This scramble buffer can be appended into the key of the item
> inside your engine. So you could pull this off with keys with the
> following structure:
>
> <key_value><scramble_buffer> (or vice versa).
>
> Pretty much you're free to do anything in the engine. Memcached is
> only a darn efficient network interface and protocol parser in the
> context of Engine API.
> After sending the previous email, I realized that the engine solution
> I mentioned is not that good. It limits access to a certain item to
> one user. A better approach would be to create a security map that
> associates scrambled data with items.
Are you suggesting that applications has to handle the scramble buffer
correctly for each accesses? It seems to me we can obtain credential of
the client using SASL authentication, without any additional hints.
If the security map means something like access control list, what we
are talking about is not fundamentally different.
The issue is the way to store the properties within the item.
BTW, Is the storage engine stackable? If not so, it seems to me we will
face a tradeoff between persistent storage and access controls.
Am I missing something?
Thanks,
dormando
> correctly for each accesses? It seems to me we can obtain credential of
> the client using SASL authentication, without any additional hints.
>
> If the security map means something like access control list, what we
> are talking about is not fundamentally different.
> The issue is the way to store the properties within the item.
>
> BTW, Is the storage engine stackable? If not so, it seems to me we will
> face a tradeoff between persistent storage and access controls.
>
> Am I missing something?
I think you should just grab the latest engine branch and go for it. It's
tracked under trondn's github.com account. Hit up the list for
feedback/etc, but just be forewarned that very little will go into the
core to slow down or increase memory requirements.
However the engine API can be adjusted to better allow the tradeoffs if
necessary. If you really want to blow tons of CPU/RAM on having granular
access controls, you should be able to do it without having to patch
memcached... Unless you need to make significant changes to the protocol,
which again will be very hard since that absolutely has to stay simple.
So yeah. Most of the core devs are speed freaks, and the intent of
memcached is to supply minimal, if any, access control (or authentication)
for speed. That shouldn't stop you from using it as a proper framework if
you absolutely must. It's become enough of a standard that we can accept
this.
-Dormando
Toru Maesaka
> Authentication is a different concept from access control.
Point taken :)
> Obviously, it is valueable.
>
> My proposition is just a framework to host an additional access control
> feature without any assumption for security models. In other word, the
> default installation does not apply any access controls, unless
> administrator explicitly set up its own security module.
>
> Please note that any security solution has its own assumption, so the
> preferable answer depending on the situation.
> In our assumption, it is difficult to keep application bug/vulnerability
> free, so we need to minimize the number of points to be checked.
> It is also why operating system checks filesystem permission on the system
> call invocations, not application side.
Like I said, I understand your argument. The question is what
proportion of the users would benefit from this? To be honest I don't
have the answer for this. However, taking into account that memcached
has been used worldwide
without serious complaints by players of all sizes in it's history, I
don't think it's necessary. Damn, I was even skeptical of the SASL
support at the beginning (I agree with this now though). I just don't
want to see memcached get bloated by adding database-like
functionalities.
However, I have nothing against people providing this functionality
inside their engines. In fact, I think it would be fantastic! This was
my original view of the Engine Project (additional things should be
provided by the engine) which was fortunately embraced by the
community. The binary protocol turned out to be useful for this too.
> BTW, Is the storage engine stackable? If not so, it seems to me we will
> face a tradeoff between persistent storage and access controls.
No. It's not stackable. If you want to do this you'd have to create an
abstraction layer within the storage engine. There were discussions on
this a while back though (I forget if it was online or offline).
Cheers,
Toru
--
Toru Maesaka <d...@torum.net>
Toru Maesaka
engine.
KaiGai Kohei
>> Are you suggesting that applications has to handle the scramble buffer
>> correctly for each accesses? It seems to me we can obtain credential of
>> the client using SASL authentication, without any additional hints.
>>
>> If the security map means something like access control list, what we
>> are talking about is not fundamentally different.
>> The issue is the way to store the properties within the item.
>>
>> BTW, Is the storage engine stackable? If not so, it seems to me we will
>> face a tradeoff between persistent storage and access controls.
>>
>> Am I missing something?
>
> I think you should just grab the latest engine branch and go for it. It's
> tracked under trondn's github.com account. Hit up the list for
> feedback/etc, but just be forewarned that very little will go into the
> core to slow down or increase memory requirements.
http://github.com/memcached/memcached/tree/engine-pu
Is it correct branch for the discussion base?
> However the engine API can be adjusted to better allow the tradeoffs if
> necessary. If you really want to blow tons of CPU/RAM on having granular
> access controls, you should be able to do it without having to patch
> memcached... Unless you need to make significant changes to the protocol,
> which again will be very hard since that absolutely has to stay simple.
If the engine approach is right, I can implement my access control module
for proof of the concept. Anyway, I'll try to check the engine branch.
> So yeah. Most of the core devs are speed freaks, and the intent of
> memcached is to supply minimal, if any, access control (or authentication)
> for speed. That shouldn't stop you from using it as a proper framework if
> you absolutely must. It's become enough of a standard that we can accept
> this.
>
> -Dormando
>
--
dormando
>
> Is it correct branch for the discussion base?
http://github.com/trondn/memcached/tree/engine is the tip. engine-pu is
... not quite master yet.
-Dormando
KaiGai Kohei
>> Please use the correct term to avoid confusion :(
>> Authentication is a different concept from access control.
>
> Point taken :)
>
>> Obviously, it is valueable.
>>
>> My proposition is just a framework to host an additional access control
>> feature without any assumption for security models. In other word, the
>> default installation does not apply any access controls, unless
>> administrator explicitly set up its own security module.
>>
>> Please note that any security solution has its own assumption, so the
>> preferable answer depending on the situation.
>> In our assumption, it is difficult to keep application bug/vulnerability
>> free, so we need to minimize the number of points to be checked.
>> It is also why operating system checks filesystem permission on the system
>> call invocations, not application side.
>
> Like I said, I understand your argument. The question is what
> proportion of the users would benefit from this? To be honest I don't
> have the answer for this. However, taking into account that memcached
> has been used worldwide
> without serious complaints by players of all sizes in it's history, I
> don't think it's necessary. Damn, I was even skeptical of the SASL
> support at the beginning (I agree with this now though). I just don't
> want to see memcached get bloated by adding database-like
> functionalities.
I don't deny it is niche. However, it is absolutely necessary feature
for people who want to set up secure web application platform.
So, I suggested this idea with modular approach, not built-in.
> However, I have nothing against people providing this functionality
> inside their engines. In fact, I think it would be fantastic! This was
> my original view of the Engine Project (additional things should be
> provided by the engine) which was fortunately embraced by the
> community. The binary protocol turned out to be useful for this too.
It seems to me there is no difference from what I said, except for name
of the framework.
Right now, I'm not clear whether the engine is a reasonable framework
to host access control feature, or not. So, I represented it just a
security framework.
Anyway, system administrators can decide whether the memcached should
load the access control module on the "engine" framework, or not.
Of course, he can pay mention about its tradeoff.
>> BTW, Is the storage engine stackable? If not so, it seems to me we will
>> face a tradeoff between persistent storage and access controls.
>
> No. It's not stackable. If you want to do this you'd have to create an
> abstraction layer within the storage engine. There were discussions on
> this a while back though (I forget if it was online or offline).
It seems to me it is a bit ad-hoc workaround.
For example, when the primary module allocate an item object, what should
the secondary module perform?
However, it is not a bad idea to start it from the upcoming engine framework.
Rami Badran
Regards
Rami Badran
Dustin
On Jan 7, 2:02 am, Rami Badran <ramibadran...@gmail.com> wrote:
> i would like to unsubscribe from this mailing list
OK. You don't have to announce it, though, just silently duck out.
Nobody will be offended.
Toru Maesaka
> I don't deny it is niche. However, it is absolutely necessary feature
> for people who want to set up secure web application platform.
> It seems to me there is no difference from what I said, except for name
> of the framework.
Indeed. Apologies for not realizing this earlier... Good luck on
hacking your Access Control engine module :)
- Toru
P.S. It might save your time if you look at this first:
http://github.com/trondn/memcached/blob/engine/doc/engine-interface.xml
--
Toru Maesaka <d...@torum.net>
Dustin
On Jan 7, 12:55 am, Toru Maesaka <d...@torum.net> wrote:
> > BTW, Is the storage engine stackable? If not so, it seems to me we will
> > face a tradeoff between persistent storage and access controls.
>
> No. It's not stackable. If you want to do this you'd have to create an
> abstraction layer within the storage engine. There were discussions on
> this a while back though (I forget if it was online or offline).
They're not entirely not stackable. The vhosting bucket engine I
wrote works by storing connection-private data pointing to another
engine instance. That is, you can load pretty much any other engine
behind this engine to provide multi-tenancy over it. The connection
overhead is a pointer. The bucket overhead is one hashtable entry and
whatever an instance of the underlying engine has (e.g. for the
default engine, that'd be the hash table plus a slabber).
Each underlying engine can be configured differently as well since
the initialization occurs for every child instance. You can have two
instances of the default engine running with different slabber configs
and max memory sizes and have them totally not conflict. One can
preallocate its maximum memory at once and disable evictions, the next
can dynamically grow and evict when a slab is full.
It would be possible to use a similar model for a MAC engine.
That said, the API for stacking them... isn't all that clean. If
this were more in demand, we could expose the loading and
initialization system. That'd kill a good deal of my code. :) The
majority of my code is reproducing parts of memcached for things like
loading engines and -- most importantly -- providing a unit test
framework that can hit all of the aspects of the engine *outside* of
memcached.
Brandon...@elementk.com
TCP/IP-based application protocol be encapsulated in an SSH tunnel?
Brandon Ramirez | Office: 585.214.5013 | Fax: 585.295.4848
Software Engineer II | Element K | www.elementk.com
From: KaiGai Kohei <kai...@ak.jp.nec.com>
To: memc...@googlegroups.com
Cc: Toru Maesaka <d...@torum.net>, Matt Ingenthron <inge...@cep.net>
Date: 01/07/2010 03:26 AM
Subject: Re: memcached and access control
Sent by: memc...@googlegroups.com
Aaron Stone
With a filesystem, the kernel knows which process is making the call
and what user and group ids the process is running as, so it can
perform access checks with that sideband information. Is there a
mechanism for this with network sockets in SELinux?
Guille -bisho-
> proportion of the users would benefit from this? To be honest I don't
> have the answer for this. However, taking into account that memcached
> has been used worldwide
> without serious complaints by players of all sizes in it's history, I
> don't think it's necessary. Damn, I was even skeptical of the SASL
> support at the beginning (I agree with this now though). I just don't
> want to see memcached get bloated by adding database-like
> functionalities.
Agree!
Memcache is meant to be a fast cache. If you need to deploy a shared
memcache used by several users, you could add a proxy that prefixes
all mc keys by the user id, so there are no conflicts, and you have
the advantage of all the memory being shared in a pool instead of
having different instances infrautilized.
--
Guille -ℬḭṩḩø- <bish...@gmail.com>
:wq
KaiGai Kohei
> Can an SSH tunnel not be used to add an authentication layer? Can't any
> TCP/IP-based application protocol be encapsulated in an SSH tunnel?
This idea focuses on different topic.
SSH/SSL tunnel prevent information leaks with tapping between the server
and the clients. However, it is an separated issue from access controls.
KaiGai Kohei
>> If we can identify the client (web application instance) in some way,
>> memcached will be able to store the something identifier of the cached
>> object. It may be called owner of the cached object (depending on the
>> security model).
>> Then, access control mechanism can provide its access control decision
>> based on the client's identifier and object's identifier.
>
> With a filesystem, the kernel knows which process is making the call
> and what user and group ids the process is running as, so it can
> perform access checks with that sideband information. Is there a
> mechanism for this with network sockets in SELinux?
SELinux has an API to get privileges of the peer process for the given
socket descriptor.
#include <selinux/selinux.h>
int getpeercon(int sockfd, security_context_t *context);
The sockfd argument is the returned value from accept(2).
This API set a cstring representation of the peer (client) process's
privilege (called security context), then return.
Then, the server process can ask in-kernel SELinux whether the required
access can be allowed, or not, based on the pair of client's security
context and item's security context.
SELinux returns its decision: allowed or denied.
If denied, the server process returns ERROR without running the request.
KaiGai Kohei
This idea does not allow to share an item from several users or groups.
As I mentioned before, it might be a solution, if we would always
enforce strict-separation rules.
But this assumption is different from what we are talking about.
KaiGai Kohei
> Ahh, now I'm with you.
>
>> I don't deny it is niche. However, it is absolutely necessary feature
>> for people who want to set up secure web application platform.
>
>> It seems to me there is no difference from what I said, except for name
>> of the framework.
>
> Indeed. Apologies for not realizing this earlier... Good luck on
> hacking your Access Control engine module :)
>
> - Toru
>
> P.S. It might save your time if you look at this first:
> http://github.com/trondn/memcached/blob/engine/doc/engine-interface.xml
I began to trace the engine implementation.
In overview level, it seems to me the engine interface well organize the
physical structures in memcached, like filesystem interface in the operating
system.
However, I could find several places unsuitable for infrastructure of
the access controls, because it needs to handle logical schema.
For example, process_bin_delete() calls the get() method at first to
confirm existence of the item for the given key. Then, if exist,
it calls remove() method to delete the specified item.
<quote>
static void process_bin_delete(conn *c) {
:
if (settings.engine.v1->get(settings.engine.v0, c, &it, key, nkey) == ENGINE_SUCCESS) {
uint64_t cas = ntohll(req->message.header.request.cas);
if (cas == 0 || cas == settings.engine.v1->item_get_cas(it)) {
MEMCACHED_COMMAND_DELETE(c->sfd, settings.engine.v1->item_get_key(it), it->nkey);
settings.engine.v1->remove(settings.engine.v0, c, it);
write_bin_response(c, NULL, 0, 0, 0);
} else {
write_bin_error(c, PROTOCOL_BINARY_RESPONSE_KEY_EEXISTS, 0);
}
/* release our reference */
settings.engine.v1->release(settings.engine.v0, c, it);
} else {
write_bin_error(c, PROTOCOL_BINARY_RESPONSE_KEY_ENOENT, 0);
}
}
</quote>
From the perspective of physical structure, it is completely right.
However, the get() method is the only chance for the engine module to control
read access on the specified item, and we have no way to inform the engine
module whether this invocation is due to the preprocess of the delete operation,
or the get operation.
In other word, the engine module cannot distinguish "get" command and a part
of "delete" command.
Thanks,
KaiGai Kohei
> (2010/01/07 19:19), Toru Maesaka wrote:
>> Ahh, now I'm with you.
>>
>>> I don't deny it is niche. However, it is absolutely necessary feature
>>> for people who want to set up secure web application platform.
>>
>>> It seems to me there is no difference from what I said, except for name
>>> of the framework.
>>
>> Indeed. Apologies for not realizing this earlier... Good luck on
>> hacking your Access Control engine module :)
>>
>> - Toru
>>
>> P.S. It might save your time if you look at this first:
>> http://github.com/trondn/memcached/blob/engine/doc/engine-interface.xml
Sorry for this break of discussion.
I checked the engine implementation, and found out several matters to
implement access control feature on the framework.
So, it is still unavailable an example module based on the framework.
* Again, the engine framework is not designed to host multiple modules
in usual way. If a security module occupies the engine framework, it
also has to load another module that provide physical storage engine,
by itself. (Or, security module also has to provide i/o feature? it
is quite nonsense.)
* The engine framework does not provide a hook just after a connection
from client is established. It is not a must-requirement. However,
we want to obtain privilege of the client process, and cache it in
the userspace.
If it is not available, we have to invoke a system-call for each
access control decision, but it will be expensive.
* The security module needs its private storage different from the storage
engine module.
Now, the conn structure has "void *engine_storage" to store an engine
private information, but the right to use is closed for the second
module.
In addition, the hash_item structure does not provide any private
storage to store security context of the item (e.g this context will
record what process create this object).
* The deployment of some hooks makes hard to decide what permission
should be applied. For example, process_bin_delete() calls "get"
method to confirm the object to be removed is existing.
But the "get" method is only chance for the engine module to know
a certain item is accessed in reader context.
Of course, if we consider "delete" is a combination of "read" and
"delete", it will be resolved. But it will damage the flexibility
of security design.
Please note that I don't intend to disparage the engine framework.
It seems to me it well organizes the physical structures in memcached.
However, it is not designed to host a security module, so here are
several difficulties.
So, I would like to go back to the earlier idea which tries to implement
a pluggable framework for access control feature.
In the default, it does not apply anything, like air.
However, if administrator want to apply security checks in memcached also,
it will allow to apply its access controls in this level.
Thanks,
> I began to trace the engine implementation.
> In overview level, it seems to me the engine interface well organize the
> physical structures in memcached, like filesystem interface in the operating
> system.
> However, I could find several places unsuitable for infrastructure of
> the access controls, because it needs to handle logical schema.
>
> For example, process_bin_delete() calls the get() method at first to
> confirm existence of the item for the given key. Then, if exist,
> it calls remove() method to delete the specified item.
>
> <quote>
> static void process_bin_delete(conn *c) {
> :
> if (settings.engine.v1->get(settings.engine.v0, c,&it, key, nkey) == ENGINE_SUCCESS) {
Dustin
On Jan 21, 1:00 am, KaiGai Kohei <kai...@ak.jp.nec.com> wrote:
> * Again, the engine framework is not designed to host multiple modules
> in usual way. If a security module occupies the engine framework, it
> also has to load another module that provide physical storage engine,
> by itself. (Or, security module also has to provide i/o feature? it
> is quite nonsense.)
It's not *entirely* clear what you mean by this. I do use the
engine to load multiple sub-engines that are separate and provide
security isolation.
> * The engine framework does not provide a hook just after a connection
> from client is established. It is not a must-requirement. However,
> we want to obtain privilege of the client process, and cache it in
> the userspace.
> If it is not available, we have to invoke a system-call for each
> access control decision, but it will be expensive.
It certainly does and I use it. You can see it here:
http://github.com/northscale/bucket_engine/blob/master/bucket_engine.c#L433
> * The security module needs its private storage different from the storage
> engine module.
> Now, the conn structure has "void *engine_storage" to store an engine
> private information, but the right to use is closed for the second
> module.
The engine should be aware of what you're doing with that data, so
it can supply the API to manipulate it.
> In addition, the hash_item structure does not provide any private
> storage to store security context of the item (e.g this context will
> record what process create this object).
Well, no. Expanding the item structure is *very* costly. That
should should be avoided where possible.
That said, a given engine can define an item to look like whatever
it wants after the base stuff. You'll notice the base structure
doesn't even have a space for the key or data.
> * The deployment of some hooks makes hard to decide what permission
> should be applied. For example, process_bin_delete() calls "get"
> method to confirm the object to be removed is existing.
> But the "get" method is only chance for the engine module to know
> a certain item is accessed in reader context.
> Of course, if we consider "delete" is a combination of "read" and
> "delete", it will be resolved. But it will damage the flexibility
> of security design.
This is a good time to be talking about this, as the engine API
isn't dogma yet.
KaiGai Kohei
>
> On Jan 21, 1:00 am, KaiGai Kohei<kai...@ak.jp.nec.com> wrote:
>
>> * Again, the engine framework is not designed to host multiple modules
>> in usual way. If a security module occupies the engine framework, it
>> also has to load another module that provide physical storage engine,
>> by itself. (Or, security module also has to provide i/o feature? it
>> is quite nonsense.)
>
> It's not *entirely* clear what you mean by this. I do use the
> engine to load multiple sub-engines that are separate and provide
> security isolation.
What I wanted to say is, it is not an essential job for the security
module to load and manage sub-modules, although it is possible.
Basically, I don't think the security module should touch something
corresponding to physical i/o operations. As long as user have enough
privileges, it should call the secondary engine module without any
preventions.
If it does not need any additional properties on the session and
items to be accessed, the matter is not too hard.
The security module just track the entry points of secondary module,
and can call it on its invocations.
However, the secondary module may want to use conn->engine_storage,
although the security module also want to cache the user's privilege.
In addition, format of the item object is depending on the secondary
module, so the security module cannot modify it in discretionary.
If we focus on the security design that enforces simple isolation
between (authenticated) users, we don't need store any additional
security properties in item objects. But, it is not flexible.
>> * The engine framework does not provide a hook just after a connection
>> from client is established. It is not a must-requirement. However,
>> we want to obtain privilege of the client process, and cache it in
>> the userspace.
>> If it is not available, we have to invoke a system-call for each
>> access control decision, but it will be expensive.
>
> It certainly does and I use it. You can see it here:
>
> http://github.com/northscale/bucket_engine/blob/master/bucket_engine.c#L433
Oh, I could not find it.
>> * The security module needs its private storage different from the storage
>> engine module.
>> Now, the conn structure has "void *engine_storage" to store an engine
>> private information, but the right to use is closed for the second
>> module.
>
> The engine should be aware of what you're doing with that data, so
> it can supply the API to manipulate it.
The matter is both of the security module and engine module want to
use this private storage for their own purpose. :(
>> In addition, the hash_item structure does not provide any private
>> storage to store security context of the item (e.g this context will
>> record what process create this object).
>
> Well, no. Expanding the item structure is *very* costly. That
> should should be avoided where possible.
>
> That said, a given engine can define an item to look like whatever
> it wants after the base stuff. You'll notice the base structure
> doesn't even have a space for the key or data.
Yes, do_item_alloc() allocates sizeof(hash_item) plus various length
bytes of memory for each items. It is a well known technique to handle
various kind of data objects, not only memcached.
How much length do you expect for the security property of an item?
I plan to add a "void *" pointer which indicates security_id object
managed in libselinux. The security_id structure is a pair of
security context (cstring) and reference counter. The number of
security context is proportional to the number of users, not # of items,
it does not cousume local memory so much.
>> * The deployment of some hooks makes hard to decide what permission
>> should be applied. For example, process_bin_delete() calls "get"
>> method to confirm the object to be removed is existing.
>> But the "get" method is only chance for the engine module to know
>> a certain item is accessed in reader context.
>> Of course, if we consider "delete" is a combination of "read" and
>> "delete", it will be resolved. But it will damage the flexibility
>> of security design.
>
> This is a good time to be talking about this, as the engine API
> isn't dogma yet.
Now I lean toward the security purpose framework which provides
meaningful hooks for access controls, and capability to assign
a security property on the session and items.
Is it too early?
KaiGai Kohei
It enables to apply access controls on every client's request based on
the access control decision of SELinux.
In my plan, SELinux specific routines will be pluggable using modular
architecture, but, right now, this patch implements this feature as
a static logic to simplify the code.
All the functions are prefixed as "semcd_*" to avoid namespace conflicts.
It means "secure memcached".
From viewpoint of the core memcached routines, these functions perform as
security provider. It anyway returns its access control decision (whether
it should be allowed, or not) based on its own security policy, and the
caller has to control the code path according to the decision.
For example, process_get_command() calls semcd_get_command() before returning
data contents of the specified item, then it returns either true or false.
If false, the caller (process_get_command) returns error. It does not depend
on the way to make access control decision. (For example, we can assume
a security model based on the SASL authenticated user, not only SELinux.)
@@ -2543,6 +2584,11 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
stats_prefix_record_get(key, nkey, NULL != it);
}
if (it) {
+ if (!semcd_get_command(c, it)) {
+ out_string(c, "SERVER_ERROR access denied");
+ item_remove(it);
+ return;
+ }
This patch modified two data structure to contain security attributes.
The conn::semcd_private and item::semcd_private is used to store
security_id_t (which is a pointer for a pair of security context in text
representation and reference counter) object.
And the following hooks allows to make access control decision at the
strategic points in memcached.
extern bool semcd_get_command(conn *c, item *it);
extern bool semcd_update_command(conn *c, item *old_it, item *new_it, int comm);
extern bool semcd_delta_command(conn *c, item *it, bool incr);
extern bool semcd_delete_command(conn *c, item *it);
extern bool semcd_flush_command(conn *c);
When SELinux support is not enabled, these function calls are replaced by
(true). It means this code does not affect anything when security feature
is not enabled.
The "test_policy.te" is a development purpose security policy.
It can be built and installed as follows:
% make -f /usr/share/selinux/devel/Makefile test_policy.pp
# su
# semodule -i test_policy.pp
# setsebool semcd_enable_auditallow 1
It is a working example:
[kaigai@saba memcached]$ ./memcached -v -s /tmp/memcached.sock
selinux: semcd_initialize() done
selinux: new connection from 'unconfined_u:unconfined_r:unconfined_t:s0:c0'
avc: granted { create } for scontext=unconfined_u:unconfined_r:unconfined_t:s0:c0 \
tcontext=unconfined_u:object_r:semcd_item_t:s0:c0 tclass=db_blob
selinux: close connection from 'unconfined_u:unconfined_r:unconfined_t:s0:c0'
--> connection from "...:s0:c0", new item was also labeled as "...:s0:c0".
selinux: new connection from 'unconfined_u:unconfined_r:unconfined_t:s0:c1'
avc: granted { create } for scontext=unconfined_u:unconfined_r:unconfined_t:s0:c1 \
tcontext=unconfined_u:object_r:semcd_item_t:s0:c1 tclass=db_blob
selinux: close connection from 'unconfined_u:unconfined_r:unconfined_t:s0:c1'
--> connection from "...:s0:c1", new item was also labeled as "...:s0:c1"
selinux: new connection from 'unconfined_u:unconfined_r:unconfined_t:s0:c0'
avc: granted { read } for scontext=unconfined_u:unconfined_r:unconfined_t:s0:c0 \
tcontext=unconfined_u:object_r:semcd_item_t:s0:c0 tclass=db_blob
selinux: close connection from 'unconfined_u:unconfined_r:unconfined_t:s0:c0'
--> connection from "...:s0:c0" can read an item labeled as "...:s0:c0"
selinux: new connection from 'unconfined_u:unconfined_r:unconfined_t:s0:c0'
avc: denied { read } for scontext=unconfined_u:unconfined_r:unconfined_t:s0:c0 \
tcontext=unconfined_u:object_r:semcd_item_t:s0:c1 tclass=db_blob
selinux: close connection from 'unconfined_u:unconfined_r:unconfined_t:s0:c0'
--> but connection from "...:s0:c0" cannot read an item labeled as "...:s0:c1"
Note that we recommend to use unix domain socket for development purpose, because
it does not need to set up labeled network configuration between server and client.
Any comments are welcome.
Thanks,
Toru Maesaka
Thanks for your proof of concept.
From observing your patch, you should be able to develop this as an
engine once we make changes to the modular engine subsystem based on
your feedback. Your feedback on get() being called inside another
function is a good point (thanks!) and I believe some of us have come
to an agreement to refactor that area. As for getting your hands on
the connection socket, Trond is willing to add it to the API.
Toru
2010/2/4 KaiGai Kohei <kai...@ak.jp.nec.com>:
--
Toru Maesaka <d...@torum.net>
KaiGai Kohei
> Hi!
>
> Thanks for your proof of concept.
>
>> From observing your patch, you should be able to develop this as an
> engine once we make changes to the modular engine subsystem based on
> your feedback. Your feedback on get() being called inside another
> function is a good point (thanks!) and I believe some of us have come
> to an agreement to refactor that area. As for getting your hands on
> the connection socket, Trond is willing to add it to the API.
I have one more point to be issued here.
Memcached protects an item from concurrent updates using mutex lock
(cache_lock) in store_item. The store_item() acquires a mutex lock
during do_store_item(), so all the updates operations shall be
serialized.
However, it also means we cannot know whether the item to be updated
is already on the cache, or not, outside of the critical section.
The SET command may work on either creation of a new item or updates
of an existing item, depending on existence of the specified item.
The REPLACE, APPEND and PREPEND command just returns an error if the
specified item does not exist on the cache.
Even if the security stuff calls item_get() outside of the critical
section to check existence of the specified item, we have no guarantee
any other thread does not create/update the target item.
It is the reason why I put semcd_update_command() hook inside of the
do_store_item().
But, in the engine mechanism, it shall be handled inside of the pluggable
module. In fact, the do_store_item() is a part of the default engine
routines in the engine subtree.
Of course, locking mechanism is one of the significant factor which
affects the performance of storage engine. I'm skeptical to enforce
a unique locking mechanism for all the engine modules.
It is just an idea. If this efforts should be placed on the engine
project, it seems to me all the security hooks should be included
into server_interface_v1, rather than engine_interface_v1.
If I understand correctly, the server_interface_v1 structure is a set
of function pointers to be called from engine modules.
If we can make clear a set of interfaces that engine module to tell
an access control decision of security provider, we can implement
this feature much simpler. (Of course, the default security provider
should always return 'allowed' to keep backword compatibility.)
If necessary, I can introduce it in more details.
The idea of such a centralized security provider (independent from
implementation details) is called 'reference monitor' in security
research, with 30 years history. :-)
Thanks,
KaiGai Kohei
>
> Thanks for your proof of concept.
>
>> From observing your patch, you should be able to develop this as an
> engine once we make changes to the modular engine subsystem based on
> your feedback. Your feedback on get() being called inside another
> function is a good point (thanks!) and I believe some of us have come
> to an agreement to refactor that area. As for getting your hands on
> the connection socket, Trond is willing to add it to the API.
This is an off topic.
Is the current get() interface also problematic for the original engine
purpose? IIUC, the framework requires get() method two roles.
1. Ensure existence of the specified item on the storage system
2. Load data contents of the specified item into the local memory
As long as all the items are always on local memory (like default engine),
it is not a matter, because all the items are always on the local memory.
However, if the engine module supports backend storage, the second job
may require us unnecessary cost when client tries to delete an item.
It is not necessary to load all the data contents when we delete an item.
KaiGai Kohei
> (2010/02/04 18:32), Toru Maesaka wrote:
>> Hi!
>>
>> Thanks for your proof of concept.
>>
>>> From observing your patch, you should be able to develop this as an
>> engine once we make changes to the modular engine subsystem based on
>> your feedback. Your feedback on get() being called inside another
>> function is a good point (thanks!) and I believe some of us have come
>> to an agreement to refactor that area. As for getting your hands on
>> the connection socket, Trond is willing to add it to the API.
>
> This is an off topic.
And, one more off topic.
Is the do_add_delta() really thread safe?
When we can increment/decrement the specified item inplace, it modifies
the data field of the item, without replacement.
But it does not prevent any other thread read the item to be modified
during memcpy() and memset(). Then, it has a possibility that unlucky
thread read neither old nor new value.
For example, if the new value is "100\0" next to "99\0", and the working
thread is scheduled just after putting the first '1', other thread can
see "19\0" for a while.
For safety, the value should have binary representation and should be
incremented or decremented with atomic operation of CPU. But it is not
a portable way...
Trond Norbye
I have one more point to be issued here.
Memcached protects an item from concurrent updates using mutex lock
(cache_lock) in store_item. The store_item() acquires a mutex lock
during do_store_item(), so all the updates operations shall be
serialized.
However, it also means we cannot know whether the item to be updated
is already on the cache, or not, outside of the critical section.
The SET command may work on either creation of a new item or updates
of an existing item, depending on existence of the specified item.
The REPLACE, APPEND and PREPEND command just returns an error if the
specified item does not exist on the cache.
Even if the security stuff calls item_get() outside of the critical
section to check existence of the specified item, we have no guarantee
any other thread does not create/update the target item.
It is the reason why I put semcd_update_command() hook inside of the
do_store_item().
But, in the engine mechanism, it shall be handled inside of the pluggable
module. In fact, the do_store_item() is a part of the default engine
routines in the engine subtree.
Of course, locking mechanism is one of the significant factor which
affects the performance of storage engine. I'm skeptical to enforce
a unique locking mechanism for all the engine modules.
It is just an idea. If this efforts should be placed on the engine
project, it seems to me all the security hooks should be included
into server_interface_v1, rather than engine_interface_v1.
If I understand correctly, the server_interface_v1 structure is a set
of function pointers to be called from engine modules.
If we can make clear a set of interfaces that engine module to tell
an access control decision of security provider, we can implement
this feature much simpler. (Of course, the default security provider
should always return 'allowed' to keep backword compatibility.)
Trond Norbye
I'm changing the remove call to look like:
ENGINE_ERROR_CODE (*remove)(ENGINE_HANDLE* handle,
const void* cookie,
const void* key,
const size_t nkey,
const uint64_t cas);
Cheers,
Trond
Trond Norbye
On Feb 5, 2010, at 4:01 AM, KaiGai Kohei wrote:
> (2010/02/05 11:40), KaiGai Kohei wrote:
>> (2010/02/04 18:32), Toru Maesaka wrote:
>>> Hi!
>>>
>>> Thanks for your proof of concept.
>>>
>>>> From observing your patch, you should be able to develop this as an
>>> engine once we make changes to the modular engine subsystem based on
>>> your feedback. Your feedback on get() being called inside another
>>> function is a good point (thanks!) and I believe some of us have come
>>> to an agreement to refactor that area. As for getting your hands on
>>> the connection socket, Trond is willing to add it to the API.
>>
>> This is an off topic.
>
> And, one more off topic.
>
> Is the do_add_delta() really thread safe?
Yes it is thread safe in the context it is called.
Trond
KaiGai Kohei
>
> On Feb 5, 2010, at 3:28 AM, KaiGai Kohei wrote:
>>
>> I have one more point to be issued here.
>>
>> Memcached protects an item from concurrent updates using mutex lock
>> (cache_lock) in store_item. The store_item() acquires a mutex lock
>> during do_store_item(), so all the updates operations shall be
>> serialized.
>> However, it also means we cannot know whether the item to be updated
>> is already on the cache, or not, outside of the critical section.
>>
>
> That is an implementation detail in the default storage engine, and you
> are free to implement this the way you want in your own engine..
Yes, the author of engine module can choose its own locking mechanism.
The point is that I want to utilize the upcoming engine modules with
the security module. It maybe have the following structure, if it
is implemented as an engine module.
ENGINE_ERROR_CODE security_store(....) {
if (!permission_is_allowed(...))
return ENGINE_EACCESS;
return secondary_engine->store(....);
}
In this case, the item to be updated is correctly identical with the item
to be actually updated by the secondary engine module.
>> The SET command may work on either creation of a new item or updates
>> of an existing item, depending on existence of the specified item.
>> The REPLACE, APPEND and PREPEND command just returns an error if the
>> specified item does not exist on the cache.
>>
>> Even if the security stuff calls item_get() outside of the critical
>> section to check existence of the specified item, we have no guarantee
>> any other thread does not create/update the target item.
>> It is the reason why I put semcd_update_command() hook inside of the
>> do_store_item().
>>
>
> Not sure I understand what you are saying here.. do_store_item is called
> from store_item when we acquired the cache mutex. item_get will try to
> acquire the same mutex so it has to wait for do_store_item to return and
> release the mutex..
What I wanted say is....
For example, when we create a new item with SET command, a thread-A
calls settings.engine.v1->store() which checks permissions and calls
the secondary engine module which actually handles physical i/o.
In this case, a thread-B can execute the store() method between the
permission check and the secondary store.
See the following diagram,
Thread-A Thread-B
---o---------------------o-----
| |
V |
security_store(...) |
| |
V |
permission checks V
| security_store(...)
| |
| V
| permission checks
| |
| V
| store_item(...)
| - Lock mutex
| - do_store_item(...)
V - Unlock mutex
store_item(...) |
- Lock mutex |
- do_store_item(...) |
- Unlock mutex |
| |
V V
---------------------------------
In this case, the thread-A checks permission before store_item().
At that time, the specified item is not on the cached, it checks
permission to *create* a new item.
However, the thread-B may calls store_item() earlier than thread-A.
In this case, the thread-B also checks permission to *create* a new one.
In the result, the thread-A updates an existing item, although security
stuff allowed to create a new item.
(For example, thread-B assign different user-id from thread-A.)
But, after that, I found out that we can put a new mutex stuff in the
security_store() like:
security_store(...) {
pthread_mutex_lock(...);
if (!permission_is_allowed(...)) {
pthread_mutex_unlock(...);
return ENGINE_EACCESS;
}
retval = secondary_engine->store(...);
pthread_mutex_unlock(...);
return retval;
}
Right now, it seems to me this issues does not prevent us.
>> It is just an idea. If this efforts should be placed on the engine
>> project, it seems to me all the security hooks should be included
>> into server_interface_v1, rather than engine_interface_v1.
>> If I understand correctly, the server_interface_v1 structure is a set
>> of function pointers to be called from engine modules.
>> If we can make clear a set of interfaces that engine module to tell
>> an access control decision of security provider, we can implement
>> this feature much simpler. (Of course, the default security provider
>> should always return 'allowed' to keep backword compatibility.)
>>
>
> That would be possible, but I'm not sure if I see the big benefit of
> this. I have my doubts that the majority of engines will implement this
> kind of access control, and if we should create such interface exported
> from the memcached core it has to be designed in a way so that it works
> for other systems as well (such as Trusted Solaris). Personally I'm not
> convinced that this is a feature that a lot of people would use, that's
> why I believe that it would be great for a specialized engine (and by
> doing so, it can be tailored to the system you want instead of creating
> a generalized solution).
Yes, the assumption of reference monitor model is all the *internal* stuff
follows on the access control decision by the centralized security module.
If administrator loads an engine module which ignores the reference monitor,
it breaks the assumption.
> What kind of access control function did you
> want the core to provide, and what would be the required change in the
> core to support this?
At least, we needs three functionalities:
* Capability to load multiple (stackable) engine module.
I want the security feature to work independently from the type of
physical engine module. It may be possible to implement initialize()
method to load another module, but it is obviously reinvention of the
wheel. I think it should be a job of the core.
BTW, it will be valuable for non-security usage. We can expect a pseudo
engine module which distributes the given item into appropriate engine
module. For example, if the key is prefixed by "persistent.*", it is
stored in the module with backend storage, otherwise, the item is on
the default engine.
* Capability to store security attribute of the item and connection.
In SELinux model, any entities (such as items in memcached) have to be
labeled with security context, to identify them. It means the core stuff
has to provide a field to store a security context of item.
It may be a special purpose void * pointer, like as I did in my proof of
concept patch.
One other possibility is enlargement of the key/value field of hash_item
structure. For example, it may be possible to call the secondary allocate()
with enlarged nbytes, to allocate a region to store security attribute,
not only key/value pair.
The security attribute depends on the type of security model.
If it uses traditional user-id matching policy, each item will have user-id
which create itself.
* Hooks to make access control decision.
Right now, it is not clear whether the existing engine framework is enough
to host access control stuff. (Perhaps, the second proof of the concept
patch (on the engine branch) will make it clear.)
At least, it does not have an error code to notice "access denied".
KaiGai Kohei
Is it actually thread safe?
Also see the diagram.
Thread-A Thread-B
----o-----------------------------o-----
| |
V |
process_get_command() |
| |
V |
item_get() V
- Lock mutex() process_arithmetic_command()
- do_item_get() |
- Unlock mutex() V
| add_delta()
V - Lock mutex()
add_iov(c, ITEM_data(it),...) - do_add_delta()
| |
V V
transmit() memcpy()
| |
| V
| - Unlock mutex()
| |
V V
-----------------------------------------
The do_item_get() returns the specified item object with reference
counter increment. It is a reference to the item object, not a copy.
What I wanted to point out is there is a possibility transmit() can
be called during thread-B executes memcpy().
The &c->msglist[c->msgcurr] may refer the item to be modified in
do_add_delta(), and it may be sent the buffer in updating.
Am I missing something?
Dustin
On Feb 7, 10:59 pm, KaiGai Kohei <kai...@ak.jp.nec.com> wrote:
> However, the thread-B may calls store_item() earlier than thread-A.
> In this case, the thread-B also checks permission to *create* a new one.
> In the result, the thread-A updates an existing item, although security
> stuff allowed to create a new item.
> (For example, thread-B assign different user-id from thread-A.)
I don't see how this is different from what "add" has to do
already. Multiple things can happen concurrently. You define what is
correct, then do what you need to do to make your engine do it. If
you need to lock and check permissions before you take final action,
then that's what you need to do.
> * Capability to load multiple (stackable) engine module.
I do this in bucket engine.
> * Capability to store security attribute of the item and connection.
I do this in bucket engine (just the connection side, I have no need
to do anything on the item side, but the engine controls the shape of
the item completely).
> * Hooks to make access control decision.
I do this in bucket engine.
... I sent you pointers to this in the past. You seem to be
repeatedly asking for things I've already shown are working. You've
also brought up some good points where we need to make improvements.
I do think it'd be good if you aim for a version that sits within
its own project running entirely as an engine and try to isolate the
things that you can't possibly do (vs. paths that are less
desirable). Feel free to harvest as much of bucket engine as you can
to make it easier to hack something together.
KaiGai Kohei
Hmm, indeed, it is a good point to make clear and isolate what is possible
and impossible in the current engine framework.
OK, I'll try to implement the security feature on the engine module.
Please wait for a while.
KaiGai Kohei
>> http://github.com/memcached/memcached/tree/engine-pu
>>
>> Is it correct branch for the discussion base?
>
> http://github.com/trondn/memcached/tree/engine is the tip. engine-pu is
> ... not quite master yet.
BTW, does the --enable-default-engine option of configure script performs
correctly?
I did as follows:
$ git clone git://github.com/trondn/memcached.git -b engine memcached-engine
$ cd memcached-engine
$ ./config/autorun.sh
$ ./configure --enable-default-engine
$ make
$ ./memcached
Could not find symbol "create_instance" in self: ./memcached: undefined symbol: create_instance
$ find | grep .so\$
./.libs/default_engine.so
Is it really built-in? Or, am I misunderstanding something?
Dustin
On Feb 8, 11:52 pm, KaiGai Kohei <kai...@ak.jp.nec.com> wrote:
> BTW, does the --enable-default-engine option of configure script performs
> correctly?
It worked the last time I tried it, but it shouldn't matter much for
your uses. If you're writing an engine, you'll be specifying -E
anyway.
Does it work when you specify it?
./memcached -E .libs/default_engine.so
KaiGai Kohei
Yes, it works.
I expected --enable-default-engine allows to link default_engine in static.
Thanks
Dustin
On Feb 9, 12:46 am, KaiGai Kohei <kai...@ak.jp.nec.com> wrote:
> I expected --enable-default-engine allows to link default_engine in static.
It should (did last time I tried it). I'm not sure what issue
you're having with that.
Trond Norbye
Just a wild guess from me.. The create_instance function isn't referenced from the memcached.c source so it's not linked into your binary at all.. Just try to edit memcached.c and initialize the create function to point to create_instance to force the linker to include it into your binary...
The way the loading in memcached works is that it tries to dlopen(NULL, ...) and then use dlsym to resolve the function.. You could try to use nm on memcached to see if it got a symbol named create_instance?
Cheers,
Trond
KaiGai Kohei
>
> On Feb 9, 2010, at 6:07 PM, Dustin wrote:
>
>>
>> On Feb 9, 12:46 am, KaiGai Kohei<kai...@ak.jp.nec.com> wrote:
>>
>>> I expected --enable-default-engine allows to link default_engine in static.
>>
>> It should (did last time I tried it). I'm not sure what issue
>> you're having with that.
>
> Just a wild guess from me.. The create_instance function isn't referenced from
> the memcached.c source so it's not linked into your binary at all.. Just try to
> edit memcached.c and initialize the create function to point to create_instance
> to force the linker to include it into your binary...
Maybe, this estimation is right.
The manpage of dlopen(3) introduced an example:
| If this program were in a file named "foo.c", you would build the program
| with the following command:
|
| gcc -rdynamic -o foo foo.c -ldl
The manpage of gcc(1) also said:
| -rdynamic
| Pass the flag -export-dynamic to the ELF linker, on targets that
| support it. This instructs the linker to add all symbols, not only
| used ones, to the dynamic symbol table. This option is needed for some
| uses of "dlopen" or to allow obtaining backtraces from within a
| program.
Then, I tried as follows:
$ make CFLAGS='-rdynamic'
$ ./memcached -d -s /tmp/memcached.sock
$ mcdclient.php add aaa xxxxx unix:///tmp/memcached.sock
success to add [key:aaa, value:xxxxx]
$ mcdclient.php get aaa unix:///tmp/memcached.sock
'aaa' => 'xxxxx'
It works for me.
> The way the loading in memcached works is that it tries to dlopen(NULL, ...) and
> then use dlsym to resolve the function.. You could try to use nm on memcached to
> see if it got a symbol named create_instance?
But nm command shows the following result, even if without -rdynamic.
$ ./memcached
Could not find symbol "create_instance" in self: ./memcached: undefined symbol: create_instance
$ nm ./memcached | grep create_instance
08054be0 T create_instance
$ nm ./memcached | grep default_
08055190 t default_arithmetic
080554a0 t default_destroy
080550d0 t default_flush
080550f0 t default_get
08054ad0 t default_get_info
08054e90 t default_get_stats
080554f0 t default_initialize
08055400 t default_item_allocate
080553e0 t default_item_delete
08055170 t default_item_release
08054e30 t default_reset_stats
08055140 t default_store
08054ae0 t default_unknown_command
Thanks,