Re: Lost after mvn package

[James Waldrop]

Hi there -- the documentation is still being improved, so this is totally understandable. What exactly are you trying to do? I hope to add some examples in the very near future, just getting a config change out the door first.

James

On Mon, Jul 9, 2012 at 11:11 AM, md1327 <132...@gmail.com> wrote:

After I generate the jar file using mvn package I am not sure what else I need to do in order to run a test in Iago

[md1327]

James,

I am trying to run the provided Java example in CentOS. Here's what I've done:

   I first downloaded the twitter-iago package

   used mvn package to generate the jar file

   used mvn install to install in root directories.

Now I am not sure how to use the Java Example file and configuration file to run a test on my server. I am new to Apache maven and I am not really sure of where to "put" the example test file and what not. Anything helps

Matt

[md1327]

If you can just give me instructions on how to run the application after you download it that would be great.

[James Waldrop]

Ok, at the risk of writing down something that's in the process of changing, here's the basic workflow:

1) get to the mvn package step that you're at

1.5) unpack the zip you packaged up -- in a dist directory or something:

mkdir target/dist; cd target/dist; unzip ../*.zip

2) create a log file. This is worth spending more time on than I can write about at this point, but here's an example:

yes "/" | head -n 1000 > sample.log

That will create a 1000-line file with just / in it. Ie, it will be hitting the server you configure and asking for the root doc. You want at least 1000 lines in your log file for performance reasons that aren't worth going into here. Also, it should be obvious, but you're going to be doing an HTTP load test with a log file like this. We're taking advantage of the fact that there's a record processor for HTTP that just expects your log file to be a bunch of path parts. This is the default HTTP processor as of 0.5.1, but that will change in my next release (0.6.0 -- I'm testing it now). So keep in mind that this is somewhat subject to change.

3) create a config file. Here's an example that uses that sample log:

cat > config.scala

---- PASTE THIS ---

import com.twitter.parrot.config.ParrotLauncherConfig

new ParrotLauncherConfig {

  localMode = true

  jobName = "testrun"

  port = 80

  victims = "twitter.com"

  scheme = "http"

  log = "sample.log"

  requestRate = 5

  parser = "http"

}

--- SIHT ETSAP ---

This is executable Scala, if that's not clear, so you can do some fairly creative things here, but I designed it to support simple behavior if that's all you need. In this case we're setting the parser to "http" which signals that we want the default HTTP record processor as I described above. We're hitting twitter.com with 5 rps (you run the risk of getting rate limited if you try to do more than this, which will make using Twitter hard, so I recommend experimenting with higher rates against your own site, probably not in production -- Iago can and will take down a service without much effort).

4) Run things:

java -jar iago*.jar -f config.scala

You're probably going to run into some roadblocks at one or more of these steps. Think of this as a high-level map and ask more questions as needed.

James

[md1327]

James,

Everything is running well with one exception; this error continuously shows up in my parrot-server.log file

ERR [20120709-15:45:28.707] server: unexpected error: com.twitter.finagle.WriteException: java.net.ConnectException: connection timed out

which I suspect is a network issue on my end.

Thank you for your detailed response and help.

Matt

[James Waldrop]

I actually think that's my issue. I am debugging what's going in now, it's some issue with the latest Finagle I expect. It seems requests are never making it onto the wire.

Sent from my iPhone

[md1327]

James,

Sorry to bother you again. I am an intern at Hewlett-Packard and my supervisor has requested that I ask you a few questions about Iago.

Other than testing a servers ability to handle a constant rate of requests, are there any other uses for Iago?

Does a complete source of documentation exist for Iago?

What are the typical requests serviced by twitter's servers? How can we mimic these using Iago?

If you would like to, I can be reached at matthew....@hp.com.

Thanks for all of your help,

Matt

[James Waldrop]

Ok, figured out the issue. I'm used to running Iago inside the datacenter with sub millisecond latencies, and of course hitting external sites like Twitter when you're not *at* Twitter takes longer than the Finagle default of 10ms to establish a connection. I've pushed a fix -- you'll want to pick it up and rebuild.

James

[James Waldrop]

Responses inline...

On Mon, Jul 9, 2012 at 3:25 PM, md1327 <132...@gmail.com> wrote:

Other than testing a servers ability to handle a constant rate of requests, are there any other uses for Iago?

There's a list of use cases mentioned in our blog post: http://engineering.twitter.com/2012/06/building-and-profiling-high-performance.html

But, I don't mind talking about why Iago rocks. :)  Here's a few reasons we felt the need to write it:

1) It's not just about supporting a constant rate, but that it even supports expressing load as a rate at all. All of the somewhat programmable load generators I've used express load in terms of threads or concurrent users. This is fine if you're modeling a closed system, where a client issues a request, waits for a response and then issues the next request. This description of a system is not very accurate for any service that's on the Internet, and at a certain level of scale isn't even a great model for most enterprise systems that sit behind a firewall. Unless you have a well-specified queueing system that is in between you and the system whose performance characteristics you care about, and that queue uses a thread pool or some other concurrency-limiting mechanism to dispatch messages / operations, you probably don't have a closed system. 

It's also worth pointing out that constant rate is only one of the ways to express load using Iago. You can also express RPS as a function, and one of my roadmap items that will be fairly trivial is to describe load as a sum of functions, so that you can do things like describe the rate per endpoint and then Iago would handle the summation for you. Commercial tools are fairly strong in this area, so I don't necessarily consider this a point of distinction, but open source tools are horrible at it.

Lest I seem too academic about this, I should add that the outcome of trying to model a system as if it's closed when it's really open is that it's substantially more difficult to reason about the results you get from performance tests. So, for instance, it can be very hard to pinpoint whether a throughput change is caused by a change to the system or your load generator, because your load generator is effectively coupled with the system under test. It also means that you have a much harder time testing a system to failure in a way that replicates the real world -- as the system degrades because more concurrent users are hitting it, every thread waits longer for a response, and you end up with very smooth reductions in throughput past some optimum. In fact anyone who's ever watched this happen in the real world can tell you that you hit sharp cliffs in your capacity, past which most of your users will receive no useful response at all. In the real world, your users don't care that you've hit your capacity limit, their requests arrive independent of your ability to service them. Iago is the only system I know of that can model this besides tools like ab that are very hard to customize.

2) Speaking of customization, Iago has a point of view in the load generation game that is very distinctive from everything else I've used. In my experience, you have two different users of load generators -- people who don't consider themselves developers who are probably generally happy with tools like ab and JMeter, and developers who get frustrated with those tools and end up spending an hour writing a for() loop that executes some number of threads hitting their service. Their load "test" is deeply flawed and they know it, but they don't want to spend a ton of time learning some system that expresses requests in terms of a GUI, XML, or something else other than actual code. Iago is aimed at helping that second group, the developers who really just want to be able to write some code and get load. Engineers at Twitter typically need about 30 minutes to go from having no load to having realistic traffic shapes at real scale using Iago.

3) Speaking of real scale, most load generators are kind of pitiful in terms of the amount of load you can actually push with them. Again, a tool like ab is the exception because it's so restricted, but sending 1000 rps is a big deal for many load generators, and with some of them you have to pay a lot of money to do that (*cough* LoadRunner *cough*). Iago is designed using the same technologies we use to scale Twitter, so it's asynchronous, non-blocking, and built on top of the JVM -- a highly-tuned platform that is capable of providing sub millisecond response time in real world conditions. We routinely run load tests that generate well in excess of 100,000 rps, and I've had serious conversations about needing to replay 1,000,000 rps worth of production traffic that I fully expect to be able to achieve with a minor bit of optimization in how requests are generated and handed off to the underlying dispatch mechanism. Keep in mind that there are very few sites that have problems on this scale, and it starts to make sense that we would need to invent something here.

4) Another outcome of the fact that Iago is so programmable is that you can do things like run the tests you would normally run testing for regressions at real scale. One of my teammates took Iago and built a system on top of it that understands the nouns and verbs of Twitter and is able to express actions plus expected results and then run a whole suite of tests at whatever rate makes sense for the environment the test is targeted at. This means that replicating problems which only happen under load becomes very doable, and my personal experience is that this is something people will be really excited about once they understand it.

Does a complete source of documentation exist for Iago?

I wish! Documentation and examples are the area that we're working on the hardest right now. The documentation we have is the README.md that ships with the project which is fairly complete, plus the very-much-work-in-progress material at http://twitter.github.com/iago plus of course the source code. There are a number of examples internally at Twitter that I will be moving into the repo as I sanitize them (our security team is rightly concerned that we not expose too much in terms of our internals that might help people who would like to attack us -- and knowing what we're focused on testing might give people ideas about where to look for problems). One of the things I'm hoping folks like yourself will be able to do is contribute your examples back to the project so that others will have a useful set of proven solutions to real-world problems to refer to. At any rate, I expect to spend the most time in this area over the next few months until things like not knowing what to do after you build the project is a non-issue. :)

 

What are the typical requests serviced by twitter's servers? How can we mimic these using Iago?

I'm not sure I understand this question. Are you asking how you can use Iago to test Twitter? If so, you'll find that you can't do this except at very low (uninteresting) traffic levels, due to mechanisms we have in place to prevent denial of service attacks. At Twitter our tests run inside the datacenter, almost never against production systems, so the sorts of limits you have as an external consumer of Twitter's services don't apply.

If you're asking what variety of traffic you can simulate with Iago -- the answer is basically that the sky's the limit in the case where you're a developer able to extend the system. I originally wrote support for HTTP and Thrift, the other protocols that Iago supports have been contributed by people here at Twitter that needed to test those sorts of traffic. It didn't take them long to add support for their protocols, either. If you're not a developer, there's basic support for the common HTTP cases, but you probably will want to investigate other solutions that are more mature. At some point a non-developer will be able to use Iago and get everything they want from it, but that day is probably a year away.

If you want to have a direct conversation, I'm reachable at j...@twitter.com, and I expect to present Iago at some user group meetings at some point as well.

James

[md1327]

Any idea what:

Exception in thread "testrun" java.lang.InterruptedException: sleep interrupted
    at java.lang.Thread.sleep(Native Method)
    at com.twitter.parrot.server.RequestConsumer.waitForNextRequest(RequestConsumer.scala:101)
    at com.twitter.parrot.server.RequestConsumer.run(RequestConsumer.scala:70)

indicates?

[James Waldrop]

Did this happen about 5 minutes in? 5 minutes is the default test run duration -- and when the server gets a shutdown, it will do this.

[md1327]

Yes it was 5 mins in. I still don't understand what is wrong, I repackaged the updated master, which was supposed to fix the finagle issue, yet I am still getting the error in the log file. I am just running a simple HTTP test, yet it seems like my requests are not even making it to the wire. I've sent to twitter and local host. :/

Thanks

[James Waldrop]

That error isn't actually a problem. It just means your test is over...

The server doesn't actually print a lot of stuff in the normal course of a successful run. If it printed something every time we sent a message, at higher rates you'd end up filling up your disks pretty quickly. So, at the risk of asking a dumb question -- how do you know you aren't sending requests?

If you want to put things into debug mode, you can -- edit the file:

src/main/resources/templates/template-server.scala

And change the INFO to DEBUG near the top of the file... Then repackage and you'll get a debug message every time something is sent. If you want an excessive amount of debugging, look at line 50 in:

src/main/scala/com/twitter/parrot/server/FinagleTransport.scala

and uncomment it. That will give you a LOT of output, probably way more than you can really interpret, but it will help me diagnose what's going on.

James

[md1327]

The issue is the same as yesterday's, and the day befores. In my parrot-server.log I continually see:

ERR [20120709-15:45:28.707] server: unexpected error: com.twitter.finagle.WriteException: java.net.ConnectException: connection timed out

I will follow your advice and run a debug session and see whats going on.

Thank you

[md1327]

For five minutes this message was repeatedly printed to the screen, no other message occured

java.net.ConnectException: Connection timed out
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:399)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:361)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:277)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)

Thanks for your time and efforts

[James Waldrop]

The debugging will be in parrot-server.log

Can you confirm with me that you have the latest version? Specifically, show me what FinagleTransport says at line 47 and 48 -- and then do a mvn clean package and replicate?

If you want to get on a chat to make this faster, you can email me at jw [at] twitter.com.

[md1327]

sent an email from my personal account, it will be quicker this way.