ControlTier performance concerns ( ... and a Puppet module)

[Al @ Lab42]

Good evening everybody,
I'm new to Control Tier and to this list and I would like to
understand if ControlTier can be the right tool for the environment
where I'm working.

I've installed the server (version 3.6.0) via the "official" rpm on a
Centos 5.5 Vmware guest, with VMware tools installed, 4 Gb of RAM and
4 Virtual CPUs and the default OpenJDK rpm.
I've made some test projects with the "biggest" one of them having
just about 20 nodes.
I find the software very promising and interesting, to the point that
I'd like to collaborate to it sharing the "modules" and components
I'll write for my company, trying to make them as more generic as
possible.

BUT, I find the web interface VERY slow and -almost- unusable, at
least for real life production environments.
For this reason I'd like like to know if there are known performance
issues or hints that can help me in making it response faster. This is
a crucial element for its adoption in my place.

As a side node, to who may be interested, I've written a simple Puppet
module that:
- Installs the client (only via package, at the moment)
- Sets the client (places authorized keys and runs ctl-setup)
- Has a define for joining a project
The module has still to be polished and needs better documentation,
but it works and has been tested on Ubuntu .04, 10.40 / Debian 5 (with
custom packages) and Centos/RedHat 5:
http://git.example42.com/?p=example42modules/.git;a=tree;f=controltier

Thanks in advance to whoever will help.

Best regards,
Alessandro Franceschi

[Anthony Shortland]

Cool ... we've long had rudimentary PuppetClient and PuppetMaster modules included in the Elements module library, but it seems you've taken that a step further, so it'll be interesting to look at integrating your work with the project.

The project welcomes the contribution of modules from the community, and there are some ideas in the offing to make that easier.

As for performance, while the Workbench UI is usable for 100s of resources we don't usually interact with it to maintain project's directly. Are you using ProjectBuilder to maintain the resource model via project XML? We find that bulk-loading resources via the ProjectBuilder load-resources command is the key to productivity in non-trivial sized environments.

We've got some Jena/RDF upgrades backed up on the feature queue that we know will yield more than an order of magnitude improvement of performance of Workbench.

In truly enormous environments, (i.e. many 1000s of nodes) you can even run the ControlTier server without Workbench, though at that point you're limited to some extent in the richness of the resource model you can employ.

Anthony.

> --
> You received this message because you are subscribed to the Google Groups "ControlTier" group.
> To post to this group, send email to contr...@googlegroups.com
> To unsubscribe from this group, send email to controltier...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/controltier?hl=en
> http://wiki.controltier.org

[Noah Campbell]

Hm...I'm using 3.5.5-2 (which was the last release before it was cut to 3.6.0). I have ~2-3000 nodes across 10 projects. The performance is fine across the install. The main screen is a bit slow, but that's due to the number of jobs, not the number of projects or resources. The project drop down helps speed that up by filtering out the jobs not relevant to that project.

That being said, I'm curious if your VM may be contributing to the slowness. Did you preallocate the entire disk? I've found this to have a *significant* impact on performance. Also...are you sharing the host server with other guests (Is this an ESX install?) as you may be in contention for other resources.

-Noah

On Sep 27, 2010, at 10:57 AM, Al @ Lab42 wrote:

[Noah Campbell]

Anthony...I don't think he's referring to a module similar to modules in ModuleForge. Looking at the source, these are puppet classes.

@Alessandro

I like your puppet classes, thanks for the contribution.

I noticed in your project class (http://git.example42.com/?p=example42modules/.git;a=blob;f=controltier/manifests/project.pp;h=912eb96ad256d96f9bcc2cefd05ab592ec835400;hb=HEAD) that you check for the inclusion of the project.properties file. You'll likely want to keep track of a similar file, resources.xml. The docs are lagging behind, but this file will keep track of all the nodes in that project. I'm not an expert in puppet, but if you could generate the list of nodes that match a given class you could easily keep this file up to date.

-Noah

[Anthony Shortland]

Oh ... that's interesting ... something complementary to what we've done before ... a Puppet module to install and configure the ControlTier client ... where we've used ControlTier modules to manage Puppet!

As Noah notes ... its all about tracking the node list ultimately.

Anthony.

[Al @ Lab42]

On Sep 27, 9:13 pm, Anthony Shortland <anth...@controltier.com> wrote:
> Cool ... we've long had rudimentary PuppetClient and PuppetMaster modules included in the Elements module library, but it seems you've taken that a step further, so it'll be interesting to look at integrating your work with the project.
>
> The project welcomes the contribution of modules from the community, and there are some ideas in the offing to make that easier.
>
> As for performance, while the Workbench UI is usable for 100s of resources we don't usually interact with it to maintain project's directly. Are you using ProjectBuilder to maintain the resource model via project XML? We find that bulk-loading resources via the ProjectBuilder load-resources command is the key to productivity in non-trivial sized environments.
>
> We've got some Jena/RDF upgrades backed up on the feature queue that we know will yield more than an order of magnitude improvement of performance of Workbench.
>
> In truly enormous environments, (i.e. many 1000s of nodes) you can even run the ControlTier server without Workbench, though at that point you're limited to some extent in the richness of the resource model you can employ.

Thanks Anthony,
I'm going to investigate better the command line usage (for the moment
I've just run some commands via CLI), up to now I've been more
attracted by the (really nice) Web UI. Consider that anyway my setup
is really minimal for the moment that's why I was surprised by the
slowness of the web interface, even for normal operations like logging
in, changing the project or listing elements.
I'm going to investigate this more, since it seems is not a normal
behaviour.

Alessandro

[Al @ Lab42]

On Sep 27, 9:22 pm, Noah Campbell <noahcampb...@gmail.com> wrote:
> Hm...I'm using 3.5.5-2 (which was the last release before it was cut to 3.6.0).  I have ~2-3000 nodes across 10 projects.  The performance is fine across the install.  The main screen is a bit slow, but that's due to the number of jobs, not the number of projects or resources.  The project drop down helps speed that up by filtering out the jobs not relevant to that project.
>
> That being said,  I'm curious if your VM may be contributing to the slowness.  Did you preallocate the entire disk?  I've found this to have a *significant* impact on performance.  Also...are you sharing the host server with other guests (Is this an ESX install?) as you may be in contention for other resources.

VM are thin provisioned, so the disk is not preallocated. That can
actually be a performance bottleneck, but mostly on write operations
AFAIK, and there shouldn't be much I/O when moving around the Web UI.
The guest is on a cluster of different ESX hosts that manage several
virtual machines, but there's no lack of resources.
I've got to make some further tests, since it seems that the beahviour
of my server seems not to be normal.
Thanks for the input, anyway.
Alessandro

[J. Ryan Earl]

On Mon, Sep 27, 2010 at 3:26 PM, Al @ Lab42 <lab42.it@gmail.com> wrote:

is really minimal for the moment that's why I was surprised by the
slowness of the web interface, even for normal operations like logging
in, changing the project or listing elements.

 

1. You should running ControlTier on bare-metal.  That's a general rule of thumb with all performance issues when virtualization is involved, test bare-metal.

2. I've found allocating the Java heap from hugepages reduces ctier CPU usage and startup time by ~25% on my setup.  Add "-XX:+UseLargePages" to your JAVA_OPTIONS and put something like "hugepages=650" on your kernel line in grub.conf.  You can also put "vm.nr_hugepages = 650" it in sysctl.conf, but each page has to be contiguous memory which you're only guaranteed at bootup which is why I do it as a kernel argument.  On a VM, this may have no effect.

/etc/default/ctier (with RPM install):

export JAVA_OPTIONS="-XX:+UseLargePages -XX:MaxPermSize=256m -Xmx1024m -Xms256m $CONFIG_PROPS"

/boot/grub/grub.conf:

title CentOS (2.6.18-194.11.3.el5)

        root (hd0,0)

        kernel /vmlinuz-2.6.18-194.11.3.el5 ro root=/dev/VG00/root hugepages=650

        initrd /initrd-2.6.18-194.11.3.el5.img

-JR

[Al @ Lab42]

On Sep 27, 9:41 pm, Noah Campbell <noahcampb...@gmail.com> wrote:
> Anthony...I don't think he's referring to a module similar to modules in ModuleForge.  Looking at the source, these are puppet classes.

Right.

> @Alessandro
>
> I like your puppet classes, thanks for the contribution.
>

> I noticed in your project class (http://git.example42.com/?p=example42modules/.git;a=blob;f=controltie...) that you check for the inclusion of the project.properties file.  You'll likely want to keep track of a similar file, resources.xml.  The docs are lagging behind, but this file will keep track of all the nodes in that project.  I'm not an expert in puppet, but if you could generate the list of nodes that match a given class you could easily keep this file up to date.

That's an interesting info, but maybe is not relevant for my module.
I check project.properties just to know if the node has already
"created" the project.
The logic of the module is to quickly setup a ctier client node with
whatever is necessary to make it usable from the server, including the
"joining" of a project (sorry for being approximative with ControlTier
naming conventions).
Basically, with that Puppet module, in a node you want as ctier client
to have to:
include controltier
make the client member of one or more projects (the project should
already be present on the server) with the define:
controltier::project { project_name: }

So I don't need the list of nodes of a project, because I actually
define in Puppet what nodes are part of what project.
Consider that I might miss some important steps for a proper client
setup, and maybe there's something more that can be done with Puppet.
It's also important for me to understand what is better to do with
Puppet and what is better to do with ControlTier.
I found that, if you don't use a bulk procedure, to make a node part
of a project is a rather repetive task that Puppet can automize well

Any suggestion for the module (still a "beta" version) is highly
welcomed.

Alessandro

[Al @ Lab42]

On Sep 27, 10:42 pm, "J. Ryan Earl" <o...@jryanearl.us> wrote:

Wonderful. Thanks for the info. I'm going to check'em out.
Alessandro

[Al @ Lab42]

On Sep 27, 9:53 pm, Anthony Shortland <anth...@controltier.com> wrote:
> Oh ... that's interesting ... something complementary to what we've done before ... a Puppet module to install and configure the ControlTier client ... where we've used ControlTier modules to manage Puppet!

:-) Right. The tools can well behave together and integrate.
I still have got to "find the right measures" and define who has to do
what, but I think that Puppet should just install and make "usable" a
Control Tier node. The rest, included possibly custom scripts used by
Control Tier on different nodes, should be left to ControlTier.
But considering my total lack of experience with ControlTier I'm
really open to alternative approaches.

Alessandro

[Noah Campbell]

In ControlTier, without a resources.xml file you won't be able to dispatch any commands to any nodes. If you're maintaining that information in puppet, you need to share it with ControlTier so it can dispatch. That place to share is <project>/etc/resources.xml

-Noah

[Noah Campbell]

Puppet can provide a list of nodes right (I'm not a user of puppet)? If that's there then controltier can control it's resources.xml file. It will require translating that list to resources.xml, but we do that routinely. Answer the first question and I can guide you through the rest.

-Noah

[Al @ Lab42]

On Sep 27, 11:01 pm, Noah Campbell <noahcampb...@gmail.com> wrote:
> In ControlTier, without a resources.xml file you won't be able to dispatch any commands to any nodes.  If you're maintaining that information in puppet, you need to share it with ControlTier so it can dispatch.  That place to share is <project>/etc/resources.xml

Uhm sorry for the question, but maybe I'm missing some key step.
In order to setup a client, with my puppet module I just:
- Install the ctier-client package
- Place the server ctier's public dsa key in client's authorized_keys
- Run in the client the command (as ctier user) "ctl-setup -n
$client_fqdn -s $server_fqdn"
- AFTER this, When I want the client to be in a project that I've
already created on the server, I execute the command (as ctier user)
"ctl-project -a create -p $project_name"

Are these the right and necessary steps?
I don't manage (and don't intend to) with Puppet resources.xml and I
expect to have it created when issuing the ctl-project command (and, I
suppose, modified when I do other ctl operations).

Thanks for any clarification,
Alessandro

[Anthony Shortland]

Yes ... we've integrated ControlTier client installation in a number of places before ... e.g. Yum install from the Kickstart postinstall script on Redhat/CentOS Linux or even having ControlTier bootstrap itself (via ctl-exec - over ssh).

If you're using Puppet to manage "platform" packages then its a good tool to use to install the ControlTier client.

Anthony.

[Anthony Shortland]

You can make a node available for use with ControlTier in a number of ways:

• Register it with a Workbench project using ctl-project from an installed client (as you describe).
• Add the node directly to the project via the Workbench UI (no client install needed).
• Create nodes in Workbench via ProjectBuilder load-resources (using project XML ... again no client install needed).

Beyond this, Noah is describing a method (under 3.6.0) of using a resources.xml file outside of Workbench (though Workbench generates that file too).

So to answer your question: those are the "right and necessary steps" when you follow the client-installation method and they will implicitly handle resources.xml management via node registration with the project as you deduced ... no need to explicitly manage resources.xml unless you want to.

Anthony.

[Al @ Lab42]

On Sep 27, 11:19 pm, Anthony Shortland <anth...@controltier.com>
wrote:

> You can make a node available for use with ControlTier in a number of ways:
>
> Register it with a Workbench project using ctl-project from an installed client (as you describe).
> Add the node directly to the project via the Workbench UI (no client install needed).
> Create nodes in Workbench via ProjectBuilder load-resources (using project XML ... again no client install needed).
>
> Beyond this, Noah is describing a method (under 3.6.0) of using a resources.xml file outside of Workbench (though Workbench generates that file too).
>
> So to answer your question: those are the "right and necessary steps" when you follow the client-installation method and they will implicitly handle resources.xml management via node registration with the project as you deduced ... no need to explicitly manage resources.xml unless you want to.

Ok, perfect. Thank you for the clarification.
BTW, I'm going to the Puppet Camp in San Francisco, at the end of next
week.
It can be a good occasion to discuss integration between Puppet and
ControlTier.
Anyone in this list going there?
Alessandro