DevOps and WebSphere Environments

[GeekInSuit]

Hi all,

We manages a medium size WebSphere environments (in the low 100s), but things are getting tough and we are suffering with all the DevOps usual pains.

So IBM comes in and gives us a "just add water and lots of money" approach to our situation, and whilst it looks good. I'm looking at what else is out there? 

Does anyone has experience with WebSphere-centric environment and had implemented capabilities to:

- Automated provisioning of environment (from provisioning AppServer up)

- Configuration Management

- Automated deployment

Specifically I'm looking for advise on tools and lesson learnt from that area.

Thanks in advance

GeekInSuit

[ranjib dey]

not websphere specifically, but i have used tomcat/jboss . Which OS you are using? What are the exact pain points you have?

[Julian Simpson]

I wrote a bunch of Jython scripts to automate cluster configuration a few years ago.    We also did app deployment via Jython.  I found Jython a pleasure, but exposing WebSphere's innards via WebSphere was sometimes ... funky.

J.

Thanks in advance

GeekInSuit

--

Julian Simpson

The Build Doctor Ltd.

http://www.build-doctor.com

jul...@build-doctor.com

(+44) 207 183 0323

[John Vincent]

It's been at least 8 years or so but unless IBM removed JACL/Jython
scripting support, you can automate quite a bit of WAS. We had a fully
automated process that did the install from an answer file. We had
another utility that deployed all of our code automatically using the
wsadmin tool. I could probably dig it up. Websphere was actually very
nicely automatable.

--
John E. Vincent
http://about.me/lusis

[Louis Coilliot]

Hello,

for a customer I use :
- a custom PXE+Kickstart+VMware API+a Web customization frontend for
OS (AIX, Linux) and VM (Lpar,VMware) provisioning
(with lpar and vio this has been really painful to automatize)
- puppet for configuration management (Linux and AIX targets)
- mcollective for orchestration (Linux and AIX targets)
- kermit.fr as a web dashboard for applications (own dev., the
application evolved a lot since the screencasts)

The application servers are Jboss, Weblogic, Oracle AS (sorry, not Websphere).

This is for hundreds of servers, but only tens of application servers
because we drive also other components (databases, batch consoles,
infrastructure servers...)

So far the actions we have as mcollective agents for application servers are :
- inventories (instances, applications, ressource pools)
- stop/start instance
- deploy an application
- redeploy an application
- create an instance
- add a resource pool
- get some logs

Kermit let us chain some actions.

This does not replace the specific consoles of the app. servers but it
does the more common daily basis stuff.

For long/async. actions we use a custom development over mcollective
with local mini schedulers on the system nodes that we can communicate
with through mcollective. There should be something like this
mainstream is the future.

The installation of the application servers are scripted but not
automated (except for jboss, automated with puppet)
The archive of the application servers are huge and that just doesn't
fit with rpm/yum or similar tools.

This is improving every day.

Louis Coilliot

2012/3/27 GeekInSuit <twy...@gmail.com>:

[Justin Ellison]

We're weblogic here, but just another vote for:

1) PXE boot the machine, install minimal OS

2) CM (Puppet in our case) configures the box.  A shell script fires via puppet to install Weblogic, using a XML answer file that's an erb template from Puppet.  It also configures and starts node manager.

3) Custom jython scripts run (also erb templates via puppet) to create or join the domain, setup server containers, applications, etc.

All said and done, we can have new hardware without an OS up and processing requests in under 20 minutes.  Best of all, it didn't cost us anything in licensing.  It's not rocket science either -- that was my first deliverable after reading "Pro Puppet".  While it's certainly not neat and tidy (it's begging for a refactor), it works reliably.

As for automated deployment, we don't have it fully automated yet, but it is pretty streamlined.  We have a Jenkins job that will create an ear file given a git tag.  We then just take that ear and use the admin console to deploy it to the cluster.  With versioning built into the ear, it's a zero-downtime affair.  We could fully automate deployments with either jython scripts or maven, but don't trust the VM's to behave properly afterwards.  That darn permgen OOM issue with Java and hot deploys that seems to impact all appservers is something we bump up against.  We end up doing a rolling restart after the deploy to make sure the VM's are good to go.

Justin

[FishNiX]

+1

We are not a WAS shop either, but we use a mix of VM templates,
cfengine and jenkins to get our apps moving. Since we are edu, we
have a million different java apps, so..
- O/S provisioning happens via a VM template with guest customizations
bolted on top, JeOS
- cfe:
- comes along and installs required packages, including a rolled
JBoss rpm
- replaces a bunch of JBoss files to allow us to pass -D properties
to the init of JBoss for instance specificity
- drops in a sysconfig with all of that instance specificity (not a
template in our case, b/c cfe templating is weak-sauce) that gets
called by the jboss init (also supplied by cfe and has NO specificity
in it)
- Jenkins shells out, puts the artifacts (wars, jars, -ds.xml, etc) in
the java container and restarts the app server(s) (rolling)

As a side note, we have a Jenkins plugin that we wrote to basically
shell to a machine stop jboss, run mvn clean install with a properties
file built from the build properties, and start jboss. We create
simple, standard, template based "installers" for all of our java apps
based on mvn + the antrun plugin. This works well for us because we
have so many little apps and lots of people with their fingers in
Jenkins. It gives DEVs a lot of flexibility on what and how things
get deployed. Our Jenkins instance easily has over 300 jobs in it now
and many of those are "installer" jobs. DEVs are empowered to push
buttons for DEV environments, business types are allowed to push
buttons in higher environments.

-Camden

[GeekInSuit]

Hi all,

Thank you so much for your replies and ideas.

The sort of solution I'm looking for are:

- Automated provisioning of environment (from provisioning AppServer up)

Reason: We do not control the VM or the OS level. So when we get the server "hopefully" we get what was ordered. There's no way of controlling the golden copy of what's being deployed.

- Automated deployment

Reason: Automated deployment to such a way that we will be able to remote deployment from one interface.

Jython + bash is the way we've been doing it but managing multiple jython scripts has also been a pain. We've also found that whilst IBM WAS has been advertising that you can do everything with jython, there are still gaps.

- Configuration Management

Reason: As we run multiple environment of the same thing, the idea is to have the configuration of the AppServers, et al. be part of the configuration management, and be able to have version control around it. So there changes are traceable.


Additional: Running on the same interface, as we have different team running PROD SUP and non-PROD, so having this running on the same interface would allow us log what has been done to which environment. 

The IBM tools we were looking for is Build Forge, but naturally it cost an arm and a leg and my first born (although thinking about it the first born might be worth it.;)

So basically the scripts are all there but we have to tie it up together.


@rdeys: The pain points we have been facing includes:

- Building brand new environments are taking long time (30 mandays+) because each applications = multiple WebServices over a few servers
- Environment consistency is hard to manage and maintain. Also no logs to support which team had done what, so a common interface is important for transparency
- Deployment pain, deployment is taken too long.

@Louis: mcollective and http://kermit.fr/ is interesting. I'm reading on that now. This might be closest we can get to.

@FishNix: Jenkins was the old hudson right? So your build scripts includes deployment for continuous deployment? How does one control which environment the deployment should go to?

[dave.stauffer]

I've been managing WebSphere servers for about 8 years (5.x and up).
We started with Jython for simple stuff but have the same pain points
you did. With WAS 6.1 we started leveraging their 'templates' to
create new server instances. Automation seems better with each
version but the cost of build forge just doesn't seem worth it. Here
is what I would suggest.

(1) Look into puppet/chef. Even if you aren't going to use it to
control the OS/VM you can use it to report on it. This way you can
tell when something is not configured correctly and at least open a
ticket with your server team.
(2) get your automation scripts into version control (e.g. svn, git)
(3) get your websphere and application configuration files into
version control
(4) Look at using puppet/chef/mcollective or rundeck to automate
running/orchestrating the scripts and reporting on the results
(5) Look at using Jenkins (yes, old Hudson) or Bamboo to do your
software build and then write a plugin to call puppet/chef/mcollective/
rundeck to do the deploy including the WAS/App configuration OR
consider something like DeployIt by http://xebialabs.com/products
(6) take a look at what WebSphere is doing with JMX. In the past
(6.1) their JMX implementation was horrible. I haven't look at it
recently but if they have improved it you have the potential to both
change configuration as well as manage servers (stop/start/etc). This
will take some java coding but could be well worth it if you don't
have the budget for the IBM tools.

Dave Stauffer
The Hershey Company

> @Louis: mcollective andhttp://kermit.fr/is interesting. I'm reading on

[Stuart Charlton]

In our environment (class 1 railway)...

Automated provisioning (install, profile setup, app server creation, cluster creation) is done with Puppet for WebSphere 7 app server, WebSphere Extreme Scale, HAProxy, Nginx, and a few Tivoli products (LDAP, Webseal, and AM).

We also install and config standalone WebSphere Portal 6 on WAS 7; clustering that beast requires small animal sacrifice.

- Config management is mostly Puppet - we template the profile XML on each app server, or template the Jython script that does the work - sometimes this lags the demand though and we do it through a quick and dirty Jython that isn't integrated into Puppet

- Automated deployment is through a mix of Bamboo, Jython scripts, shell scripts (to move files). Mostly we orchestrate with the shell script, which will grab the release builds off Bamboo, copy it around, then puppet kick the various servers. It's not 100% but could be if the team weren't so buried.

new WAS clusters take maybe an hour or two to install and configure, maybe a day or two to shake down and verify. Portal clusters takes up to a week due to its large amounts of suck.

My general view is that between Puppet, the Jython WAS tool, and a system like Bamboo or Cruise , you can get to some form of sanity with Websphere...

Stu

Sent from my iPad

[Anthony Shortland]

We see the "automated deployment" console as an element in the tool-chain in its own right. Build consoles like Bamboo and Jenkins (and Cruise) extend to this functionality to some extent, of course, but there's nothing like a purpose built tool!

Have you looked at Rundeck (http://rundeck.org/) for deployment orchestration? There's a Bamboo plug-in (https://github.com/connaryscott/bamboo-rundeck-plugin) and a Puppet node provider (https://github.com/jamtur01/puppet-rundeck) that would tie the deployment scripting in your tool-chain together nicely ... 

As for Websphere cluster configuration, I'm guessing that it requires a mixture of distributed orchestration (juggling your nodes) and local orchestration (doing stuff on each box). Where is the pain that requires "small animal sacrifice" exactly? Lack of programmable interfaces? Coordinating the orchestration?

Anthony.

[Stuart Charlton]

I have looked at Rundeck ; it's one of the options to eventually replace the bash solution.

As for clustering - Websphere app server is a different beast from WebSphere Portal.  App server only requires the execution of a couple local node commands to have it join a cluster.  

Portal comes with a massive Ant-based ConfigEngine that is driven from properties files, and can be flaky, and is mandatory for use (ie. no manual workarounds).

For example:

- There are different pathways for creating the first node of the cluster, adding a node to a cluster, adding a VM on an existing mode to a cluster, and removing VMs/nodes.  Each of these has its own timing and synchronization issues:  a pathway usually involves 5 to 7 ConfigEngine commands, each ConfigEngine command takes 20 to 40 minutes to run, afterwards the IBM documentation says "wait 30 minutes" at certain spots while background processes are synchronizing state from the WAS Dmgr).  The ConfigEngine and Dmgr is not thread safe - one can't for example add several cluster nodes at once, it is capable of working on one node (or VM) at a time only.

- The property files are modal, ie. they require tweaks to certain properties depending on which ConfigEngine pathway you wish to run and the current state of the cluster.

- There are dependencies on external servers (the Websphere deployment manager, Tivoli LDAP, Tivoli policy server, Tivoli access manager) that also need to be in a predictable state or else errors can torpedo the configuration process requiring a rollback to a prior known state, which is difficult to determine in an automated fashion given the voodoo and under documentation of the ConfigEngine.

- There are tricky incompatibilities between certain WebSphere patch levels, Tivoli access manager plug-ins, and JVMs. You need to, for example, have two different IBM VM (or a Sun and IBM JVM) version level installed and know which to use for which command to avoid spurious NPEs.

- The ConfigEngine also is helped by having sysadmin access to Oracle; most tables it creates can be done under a particular schema with configurable tablespaces, but the JCR (content repository) module of portal has a subset of tables that (I kid you not) requires hard coded tablespace names.  This could be done externally by Puppet ahead of time of course assuming your DBAs are willing to house a sysadmin role there (luckily we have a dedicated Oracle instance).  The ConfigEngine then proceeds to copy the contents of the local OOTB Derby database into Oracle over 30-40 minutes, and recreating all the connection pools. 

- Oracle also needs to ensure it is ready for the onslaught of connections via the PROCESSES parameter, as each Portal server contains 6 connection pools that default to 10 to 50 connections (so, 60 x 4 server cluster is 240 sessions at startup.... This can grow to 900+ under load -- and we haven't even discussed the application-specific connection pools, just the built in modules with Portal!). 

You MUST have separate connection pools for I think 3 Portal modules, but it's a PITA to figure out what module is crapping out due to a oracle issues so I just give them all their own pool and then use a Jython script to drop down the connection pool settings for Portal modules we aren't using (but still are mandatorily installed and set up). 

Manually, setting up a 4 server/2 node Websphere portal cluster given the above takes best-case 12 hours - 8 of which are processing time, 3 are waiting time, and 1 hour is kicking off ConfigEngine tasks.  I've done it on average in 15 to 18 hours (usually straight).  Some take 4 or 5 days, including $300/hour IBM consultants.  

From an automation POV, we've done the happy path but it has never worked out fully - capturing and handling the exceptional conditions given the variety of them and difficulty to detect  has been frustrating our efforts to fully automate cluster configuration.   A combination of Splunk for log search and some manual intervention has been the solution thus far.   

Our new solution is the right one - throw Websphere Portal to the curb like the broken, stained and cigarette-burned sofa it is.  We are sticking with App Server for now.

Cheers

Stu

Sent from my iPad

[John Ryding]

Hi All,

I am looking at trying to improve the automated testing around one of our development projects and was wondering how others use git's pre-commit hooks to execute automated tests before code is even checked in?

What types of tests do you execute? Do you execute all of your project's unit tests, or just a subset of tests that do some sanity checking?

Some examples I found were:

- Execute tests that globally affect the product [1]
- Execute sanity checks to ensure that syntax is correct [2]

Thanks!

-John Ryding

============

[1] http://eng.wealthfront.com/2012/01/pre-commit-tests.html
[2] http://www.jedi.be/blog/2011/12/05/puppet-versioning-like-a-pro/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+jedi%2FIZwx+%28JEDI+%3A+Just+Enough+Developed+Infrastructure%29

[Jez Humble]

My rule is to execute as many tests as you can in a few minutes (your commit suite should under no circumstances be more than 10 minutes). Ideally that would be all your tests, not just unit tests (if that makes you raise your eyebrows, check out this: http://dan.bodar.com/2012/02/28/crazy-fast-build-times-or-when-10-seconds-starts-to-make-you-nervous/ ). Certainly you want to run at least all of your unit tests. If you can't run all your unit tests in a few minutes, then you need to fix them - unit tests should not touch the database, filesystem or network, and each one should only test one or two methods / functions. This is precisely so you can run them all in a few minutes before pushing to trunk.

For more information on organizing and pipelining test suites, check out this chapter from my book: http://www.informit.com/articles/article.aspx?p=1621865 - skip to page 4 of the article if you just want to know about commit tests. The book has a whole chapter on the commit suite, and if you want more on writing good tests I recommend this book: http://www.amazon.com/Growing-Object-Oriented-Software-Guided-Tests/dp/0321503627

Thanks,

Jez.

--
Jez Humble

Co-author, Continuous Delivery

http://continuousdelivery.com/

http://jezhumble.net/

[Brian Akins]

On Jun 8, 2012, at 5:45 PM, John Ryding wrote:

>
> I am looking at trying to improve the automated testing around one of our development projects and was wondering how others use git's pre-commit hooks to execute automated tests before code is even checked in?

I'm a commit early, commit often type of guy. I tend to encourage frequent (possibly broken) commits. This can be limited to topic branches, etc. I would only run tests on things being merged into master, etc. But that's me...

--Brian

[R. Tyler Croy]

On Fri, 08 Jun 2012, John Ryding wrote:

>
> Hi All,
>
> I am looking at trying to improve the automated testing around one of our
> development projects and was wondering how others use git's pre-commit
> hooks to execute automated tests before code is even checked in?
>
> What types of tests do you execute? Do you execute all of your project's
> unit tests, or just a subset of tests that do some sanity checking?

Personally, I think managing a pre-commit would be a pain, especially as the
team grows. I've been quite a fan of Gerrit+Jenkins
(https://code.google.com/p/gerrit/) which I've deployed successfully at the
last two companies I've worked at.

In the case where you don't want to enforce code reviews in Gerrit, you can
still set up Gerrit + Jenkins to do pre-testing of commits before they hit the
main tree. THis is what Wikimedia does with their Puppet manifests: blog-post
<http://blog.wikimedia.org/2011/09/19/ever-wondered-how-the-wikimedia-servers-are-configured/>,
production Gerrit <https://gerrit.wikimedia.org>


If you do want to go the pre-commit route, I recommend running something that's
less than 10 seconds, which likely means that you'd be left with puppet-lint or
foodcritic. Even then it's a last-ditch sanity check, you should probably bake
running-tests-prior-to-commit into your development workflows and culture
anyways.


Cheers
- R. Tyler Croy
--------------------------------------
Code: http://github.com/rtyler
Chatter: http://twitter.com/agentdero
rty...@jabber.org

[Jez Humble]

Assuming OP is talking about software development not managing server configuration,

Personally, I think managing a pre-commit would be a pain, especially as the
team grows.

This just isn't the case (or at least, not if you do it right). Having worked on large teams with really enormous test suites, they pay-off is massive. I always ask how many people are doing unit testing when I give talks, and the answer is always <25%, which I find appalling. I just don't see how you can call yourself any kind of professional if you're not writing unit tests. And if you're doing it right, you should be able to have a comprehensive suite of unit tests that runs in less than a minute. And if that's true, there's just no excuse not to run it every time you commit.

I've been quite a fan of Gerrit+Jenkins
(https://code.google.com/p/gerrit/) which I've deployed successfully at the
last two companies I've worked at.

One of the problems with code review is that it doesn't help you at all with regressions. Also, while I think it's OK to do code review post-commit, gating pushing to master on code review is a Bad Idea, because it delays getting feedback on whether your code integrates with the rest of the system (that's the point of continuous integration). This problem really does become very serious once your team gets bigger. Code reviews are a good idea, but it's better to do them using a tool like Crucible, which lets you review after you push to master.

Thanks,

Jez.

[R. Tyler Croy]

On Mon, 11 Jun 2012, Jez Humble wrote:

> Assuming OP is talking about software development not managing server
> configuration,
>
> > Personally, I think managing a pre-commit would be a pain, especially as the
> > team grows.
>
> This just isn't the case (or at least, not if you do it right). Having
> worked on large teams with really enormous test suites, they pay-off is
> massive.

I was actually referring, rather poorly, to Git pre-commit hooks. Managing Git
hooks in general has been a thorn in my side since there's no way to enforce it
from the source repo's perspective, they're all local in .git/hooks (or
~/.git-template IIRC).

> I always ask how many people are doing unit testing when I give
> talks, and the answer is always <25%, which I find appalling. I just don't
> see how you can call yourself any kind of professional if you're not
> writing unit tests. And if you're doing it right, you should be able to
> have a comprehensive suite of unit tests that runs in less than a minute.
> And if that's true, there's just no excuse not to run it every time you
> commit.

I 100% agree, except for the implication that "you're not doing it right if
your unit tests are taking too long." With Rails in particular, the default
state of the world seems to love scaffolding buckets of crap, so a lot of times
loading the Rails environment alone takes a 10s of seconds to load (much to my
dismay).

As far as I'm concerned, whether through Gerrit, a local pre-commit hook or
something else [1], I think running the test before your code is "exposed" to
the other team is invaluable.

> > I've been quite a fan of Gerrit+Jenkins
> > (https://code.google.com/p/gerrit/) which I've deployed successfully at
> > the
> > last two companies I've worked at.
>
>
> One of the problems with code review is that it doesn't help you at all
> with regressions. Also, while I think it's OK to do code review
> post-commit, gating pushing to master on code review is a Bad Idea, because
> it delays getting feedback on whether your code integrates with the rest of
> the system (that's the point of continuous integration). This problem
> really does become very serious once your team gets bigger. Code reviews
> are a good idea, but it's better to do them using a tool like Crucible,
> which lets you review after you push to master.

Gerrit and Jenkins supports the workflow you describe. Let's say you have a
commit to push into the master branch, when the commit is pushed to Gerrit and
event is triggered in Jenkins, the job that is kicked off that will pull your
change in with the latest code from the master branch and run whatever tests
you've configured.

I /think/ this meets the criteria about getting integration feedback on your
commit. I you don't need to gate commits to master on code review, I've seen
folks use Gerrit as a means of doing commit validation before merging changes
into the "gold" Git repo's master branch (you can +2 and submit your own
changes if you configure it to allow that).



[1] <http://kohsuke.org/2012/06/05/push-changes-directly-into-buildhive-and-never-run-tests-again/>

[Jez Humble]

Hi Tyler

Thanks - that's very helpful. It's good to know that Gerrit lets you do review post-commit.

With Rails in particular, the default state of the world seems to love scaffolding buckets of crap, so a lot of times loading the Rails environment alone takes a 10s of seconds to load (much to my dismay).

Although I haven't used Rails in a few years now, I seem to remember it does badness like loading ActiveRecord to run unit tests. I interwebbed and came up with this: http://blog.webbynode.com/2012/05/unit-tests-in-ruby-on-rails/ - seems sensible to me based on my very superficial knowledge.

Cheers,

Jez.

[sla...@nullserv.com]

Hi!
My team manages an all WebSphere-based application farm of approximately 2400+ instances (JVMs).
Most are clustered.
Our solution for provisioning is based on shell script wrapper for Jython scripts.
Over the years i've built quite a library of those scripts.
Its a simple menu based shell script that asks input data and then executes on the target DM remotely.

As far as deployments go - almost same deal: shell script wrapper > small Java property file parser that also produces a Jython script for a particular app that's being deployed (we have around 350 apps) > remote wsadmin execution of said Jython script on a target Deployment Manager.

All these things run on one designated server (Solaris 10 local zone). Makes it easier for our team to deploy 30-40 apps on any given Friday night.

We don't currently have a config management solution though. Actively looking for one. IBM tried to push it's Rational Automation Framework, but i've found it to be worse than BMC's tool - BMA (a.k.a Phurnace).
The only tool i really did like was called mValent Integrity Server which was later purchased by Oracle and is now a part of their integrated offering. And since this is an IBM shop - there goes that idea...
With about 240 WAS ND cells its becoming a real PITA to mitigate "config drift"...

Sal.

[jyang]

Check out www.metaforsoftware.com.  We created a light-weight SaaS tool to detect drift in cluster environments.  It can be scheduled to check for server changes hourly or daily and get email notification when changes are detected.  Can also be used to detect drift between cloned servers in a cluster. We're currently in free beta.  Please check us out, would love to know what you think.

[Slavka Baytalskiy]

Will certainly check it out. Thanks for a tip!

Slava Baytalskiy

sla...@nullserv.com

718-569-8979

--
You received this message because you are subscribed to a topic in the Google Groups "devops-toolchain" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/devops-toolchain/4oqe4snWlpg/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to devops-toolcha...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.