Devops: Security
Matt Ryanczak
maintenance of system and application security in a devops world.
On another list I follow the subject of selinux came up which got me to
wondering... Does anyone use a strict selinux profiles in a devops type
of environment? How do you keep system and application security policies
up to date in a rapidly evolving environment where servers are
provisioned on-demand, and application releases happen almost daily
which result in new application footprints that must be noted in selinux
policies?
It seems like having operations involved in the development process
(scrums, stories, etc.) at least opens the door to allowing for strict
policy based access control for applications and the OS they run on but
I'm curious if anyone actually does this. This is a two way street as
well, there are certainly system tools that application developers can
leverage to make their apps more secure, selinux is an example.
Does anyone out there implement mandatory access controls using selinux
(or apparmor) and update policy via a CM tool such as Chef? Do chroots,
jails or other security mechanisms (tripwire, whatever.) play a role
your deployments? If yes please share :) If not, is it because it is
just too difficult to keep up with?
Are we still in the hard-outer-shell-with-squishy-insides world or is
devops taking system and application security to the next level?
Chad Woolley
> I'm curious how others view the development, implementation and maintenance
> of system and application security in a devops world.
Everybody and every project has different security needs. It's all
about risk vs. return, and people love to go overboard on security.
Here's one "pretty good" approach:
* Lock down every port you aren't using at the box and firewall
* Make all your boxes only accessible only via SSH through a bastion host
* Have reasonable policies on SSH keys and password security
(reasonable again depends on your app domain and individual situation)
* Have easy access to all logs via a tool like Splunk (to have all
info easily accessible if you do suspect a security breach).
Even some of those are overkill for some situations.
Then, on your actual boxes, do The Simplest Thing That Could Possibly
Work, which means not going overboard on security.
In other words, do everything you can to keep people out of your
boxes. If they actually get into your boxes, you're already pwnd, so
it's pointless to make your day-to-day life harder in order to deal
with that scenario.
For me (my case is an interpreted app environment - Ruby/Rails), this
means running everything as a non-root user (except stuff that can't
by design, like nginx/apache), and setting up NOPASSWD sudoers for
that user.
For external monitoring or anything else which might represent a risky
but necessary attack vector, you can set up independent
security-restricted users that only have access to the files/resources
they need. E.g. your Nagios/scoutapp.com jobs might only have
read-only access to basic OS or database metrics, but have no access
to real data or any write access to anything.
That's my $.02. I am not a security professional, and I'm sure many
people disagree with me :)
-- Chad
Noah Campbell
- Teach the idea of least privilege.
- Explain how auditing works.
- If giving developers root access and then backing into least privilege is how you run your shop, explain what you're looking for (ports, file access, commands, etc.)
- If sudo is the preferred method of privilege escalation, give them some insight into how you manage sudo.
- If centralized authentication is encouraged, make sure they have easy access to create new accounts (easy being "within minutes").
- If you have centralized authorization, or use groups and group membership for authorization, explain it to them and make it easy to get.
...
The list goes on, but as I write it, two themes come to mind.
* Make it easy to get service credentials, access to group membership during development.
I think the worse case scenario is the developer builds their own authentication and authorization system because they couldn't get what they need. From a developers perspective, how is it they can provision 20 machines in 5 minutes, but it takes 5 days to get a login name.
* Give developers full access with audit in development.
This should test your audit capabilities. Let them loose and then tighten down least privilege. If a user keeps 'sudo su -' to get something done, go find out why 'sudo cmd' isn't working.
Once the app is built, deploying to production should put those policies into effect. If you require staging to test first, so be it. If you can test in production rolling the change out in a controlled way, great. Since security configuration is almost parallel to system configuration and both represent change management, make sure they are treated equally.
-Noah
Noah Campbell
415-513-3545
noahca...@gmail.com
Matt Ryanczak
> That's my $.02. I am not a security professional, and I'm sure many
> people disagree with me :)
Sounds pretty typical to me.
Does anyone treat security policy* as just another application in the
stack that has to pass through continuous integration along with other
applications?
* security policy: iptables rules, selinux/apparmor policy,
passwd/group/sudoers config, sshd_config+keys, ACLs, .htaccess, general
OS hardening, etc.
Justin Ellison
Chad Woolley
> Here's one "pretty good" approach:
>
> * Lock down every port you aren't using at the box and firewall
> * Make all your boxes only accessible only via SSH through a bastion host
> * Have reasonable policies on SSH keys and password security
> (reasonable again depends on your app domain and individual situation)
> * Have easy access to all logs via a tool like Splunk (to have all
> info easily accessible if you do suspect a security breach).
On, and:
* Keep up-to-date on security patches for OS, apps, and tools.
Radu Brumariu
which is fine, but it is also very ops-ish, and very little dev-ish.
For me, devops work is mostly on the application, everything else is
just infrastructure.
For example, here are some questions that, imo, would make more sense
for a devops shop ( again considering that due diligence has been done
towards securing the OS ) :
- do you scan your application for XSS ?
- does your site use input validation on _all_ possible parameters and
entry points ?
- do you do automatic security check on the code as it's committed?
- do you measure and keep track of abuse traffic on your application ?
- is security review an integral part of your application design process ?
- do you do all these in a consistent and automated fashion ?
Radu
Ohad Levy
> I see a lot of talk about iptables, sudo, permissions, patches etc,
> which is fine, but it is also very ops-ish, and very little dev-ish.
> For me, devops work is mostly on the application, everything else is
> just infrastructure.
>
> For example, here are some questions that, imo, would make more sense
> for a devops shop ( again considering that due diligence has been done
> towards securing the OS ) :
>
> - do you scan your application for XSS ?
> - does your site use input validation on _all_ possible parameters and
> entry points ?
Do you know where do you use mass assignment? - sorry I had to.
Matt Ryanczak
> I see a lot of talk about iptables, sudo, permissions, patches etc,
> which is fine, but it is also very ops-ish, and very little dev-ish.
> For me, devops work is mostly on the application, everything else is
> just infrastructure.
Isn't devops both infrastructure and code? Infrastructure as code? Given
that, OS and app security is fair game. Throw in network security as
well while your at it. All of this is interdependent imho.
Paul Graydon
> I see a lot of talk about iptables, sudo, permissions, patches etc,
> which is fine, but it is also very ops-ish, and very little dev-ish.
> For me, devops work is mostly on the application, everything else is
> just infrastructure.
first place that made devops necessary?
There is no such thing as "just infrastructure", just like there is no
"just code".
How code interacts with infrastructure, and how the infrastructure is
set up to handle the code is as significant as anything you actually write.
>
> For example, here are some questions that, imo, would make more sense
> for a devops shop ( again considering that due diligence has been done
> towards securing the OS ) :
>
> - do you scan your application for XSS ?
> - does your site use input validation on _all_ possible parameters and
> entry points ?
> - do you do automatic security check on the code as it's committed?
> - do you measure and keep track of abuse traffic on your application ?
> - is security review an integral part of your application design process ?
> - do you do all these in a consistent and automated fashion ?
It's absolutely critical to think holistically.
Security across every part of the infrastructure is everyone's concern
at every stage. If you don't have that, gaps appear. "Oh that's a QA
responsibilty." whilst QA thing "Oh that's a sysadmin responsibility"
who think "It's a dev & QA responsibility".
Security being intrinsic is as important as the idea that ops should be
an intrinsic part of dev, for much the same reasons. It's about doing
things right and correct from the get-go. No more ugly kludges to get
things working. No more compromises.
Security is significantly more than just what happens at a user
interaction level, even within the context of an application. From an
off-the-top-of-my-head,
sysadmin-thinking-what-he'd-like-a-dev-to-consider perspective:
- How does the application use resources on the system?, e.g. Do you
need temporary working space on the file system? Where? What are the
minimum permissions it needs? What quotas can be reasonably put in place?
- Are you using the language correctly (as designed/documented) or are
you taking advantage of unintended behaviour, because it works? (no
matter how common the approach is). The former is safer and less liable
to be affected by patches, the latter bodes the possibility that someone
could take advantage of other unexpected behaviour to compromise your
system.
- Speaking of patches, how, when? What procedures have you in place for
testing and confirming new patches work? Are patches automated? How do
you track patches not just at an OS level but modules you use in your
application? Your code can be as well written as you like and pass all
your security scans, code reviews etc, but if there is a vulnerability
in the language / module you're using your system will be insecure.
- Which version of the language are you developing in? New? Old?
Somewhere in between? If new it's probably not had enough time to iron
out bugs, if old you run the risk it's reaching the end of life cycle
and any vulnerabilities won't be patched.
- Can your app run suitably chrooted to reduce the possible attack
vectors should someone compromise it?
- What packages, specifically, do you need on the server to run the
application. Fewer the better. If it's not there on the server it
can't be compromised (simple concept that seems hard for people to
grok). Can we easily segregate the services, especially if it needs a
mix of resources?
There are more things to consider, I might add more tonight when I've a
chance to thinking longer on the subject (maybe blog post worthy for
someone to tackle?)
Paul
Radu Brumariu
infrastructure, I meant the space in which the application operates.
And I am not disconsidering it, and as I mentioned, those things are
good -- just didn't see any mentioning about securing the application
or even building security into the application rather than looking at
the host it runs on.
That's all.
And I still care only about my application. And if to make it
[secure|work|perform] means I have to [secure|install|tune] my
[apache|DNS|X] server, then so be it. See, it is holistic, I just
choose to frame everything in terms of my application, because that's
all my users care about.
And I strongly believe there are only _just users_. This you can take literally.
Best,
Radu
Paul Graydon
> And I strongly believe there are only _just users_. This you can take literally.
>
compromises come from inside jobs, people that won't necessarily be
using the application to get at things so they won't be "users".
Security is a much, much larger picture!
Paul
Ernest Mueller
DevOps/DevOps and Security at DevOpsDays Austin and then at AppSec
USA in Austin. Here's one from RSA from Gene Kim and Josh Corman:
http://www.tripwire.com/state-of-security/it-security-data-protection/it-at-ludicrous-speed-with-rugged-devops/
- I think it's basically the cutting edge topic this year in DevOps.
Model driven automation lets you build loads of security in. We used
our model to wrap every kind of box Amazon security groups explicitly
allowing only defined access, install ossec on each box at start,
configure onboard firewalls, etc. And just having your systems
created from a model makes them auditable by default. Never bothered
with selinux kind of stuff, but had app code scanning as part of the
build process (veracode). A lot of the extant app sec tools are hard
to automate, we used services like Rapid7 and White Hat but so far
they haven't caught the API bug. Security tooling is lagging pretty
bad on API/cloud/devops happy kinds of things.
Ernest