I'm not really testing cf-serverd itself, but rather, testing how it behaves on top of a couple of different filesystems. The details beyond that aren't really important to my question, but might be useful to someone else. So:
Initially, I was just using a file copy promise on the secondaries to keep in sync with a single master. But that didn't scale well as the number of files increased, so I switched to a locally checked-out copy of the SVN repo, but large svn working directories are like horses - they look pretty, but everything seems to kill them. So, ongoing maintenance of the svn thing got too tedious - and it was slow to do an svn update across ~100K files (and the asosciated .svn directories). A while back, I moved to Gluster as the means of synchronizing several secondaries with a fail-over pair of primary servers, and a post-commit hook to update the GlusterFS on change. My policy consists of around 30-ish files which go to all systems, and another seven or so files which are specific to each system. It's that set which are system-specific that cause the problem (for most everything); I have tens of thousands of systems, and those files are very small.
The first iteration of the Gluster solution was a replicated volume across two bricks, and the CFEngine secondaries serving files directly from a gluster mount. The gluster client is implemented using FUSE, so it has all the drawbacks of being a userspace filesystem - including really bad performance with a bunch of small files. In addition, with a replicated gluster volume, a client doing a stat() call needs to contact all of the servers which have a replica to ensure that the data it's getting is accurate. So, with two replicas, the client first has to essentially lock "hey, youz guys, I need to know about this file" on two systems and wait for a response from both - which is slow on the same network, but is particularly bad across a slow WAN link between geographically distributed sites. While it performed ok with a couple of systems, the combined impact of both issues caused cf-serverd to be unable to keep up with the client load when I ramped it up to the full ~3-5K per site. CPU usage was low, but load average went through the roof because the system was constantly waiting on I/O, and clients would time out before cf-serverd could respond. Since then, I've moved to using Gluster replication, as Gluster 3.4 can replicate from a volume to a local filesystem. The replication is neat; since the filesystem knows what has changed, it can basically do a smart rsync of just what's changed recently (or since the last sync, if a node is temporarily down). So, I replicate from the central Gluster filesystem to a local XFS filesystem on each secondary. However, the amount of activity and number of small files causes some memory issues on the Gluster client, which eventually causes the cf-serverd process to keel over when the system starts running low on memory and connections start to back up. Also, Gluster 3.4 has poor status tracking on replication. Gluster 3.5 can show the replication status of replicated volumes with more granularity and is supposed to have a couple of tweaks which should fix the memory issue I'm seeing, but it can't replicate to a local filesystem any more.
So, the reason for wanting to generate a load on cf-serverd is to see how the underlying synchronization mechanism handles the load it will be subjected to. I'm either going to implement a Gluster plugin to export from a replicated volume to a local filesystem, replicate to a volume with a brick local to each CFEngine secondary and then use an NFS loopback mount to each local replicated volume (this is the easiest solution and removes the FUSE drawbacks, so I hope it works), implement a FUSE filesystem to provide the per-host configuration files using an SQL view rather than "real" files (I implemented this a while back, and it seems awesome, but haven't tested to validate under a load), or go with some other solution if none of those behave like I want. But in order to determine if they behave like I want, I either have to temporarily shift the production load over to a test system, or generate a synthetic load. I'd rather generate a synthetic load. :)
Amusingly enough, the "how might you solve this problem of distributing config files to a bunch of machines" question comes up in your SRE phone screening process, leading to a minor ethical dilemma for someone who happens to follow this mailing list and be familiar with the BitTorrent solution that the phone screener is probably most familiar with. :)
--Danny