How can I load test a websocket server?

[Rayland]

I have a websocket server written in Go and I want to load test it.

I don't know how to approach this problem, I don't have a good understanding of websockets. 

Do I use a pool of ports that clients can use or use as many as possible?

Do I close the connection after I've used it or reuse it? Can I check that that specific connection is not being used?

How do I parallelise this?

[Shawn Milochik]

I'm also very interested in any replies to this. In the meantime, hopefully what I've done and learned will help you get started.

1. The first thing you're going to run into is a "too many open files" error. This is due to the soft limits, and can checked with "ulimit -S -n" and can be modified by passing a new limit, such as "ulimit -S -n 65536." You will run into permissions problems changing the limits to the highest possible values, in which case you can modify /etc/security/limits.conf to help. Do a Google search, I'm not an expert in this. You should at least be able to bump up the default (probably 1024) to 2048 or 4096 without resorting to any sudo commands.

2. You are going to want to use multiple computers to do this. One running your server, and many running clients. You can easily create thousands of websocket connections from a single client, but you won't be able to really stress-test your server with only the connections from one machine. At least, I haven't been able to. Write a simple client app that just connects and starts sending messages, and make it configurable via command-line flag so you can tell it how many simultaneous connections to create.

3. Take this opportunity to profile your server! Use net/http/pprof (see the golang.org docs), and dump your CPU and RAM usage SVGs to disk. Let it run for hours (or days) and check for performance bottlenecks and memory leaks. This will be very valuable. Also, capturing the traceback will help you find your deadlocks.

For example, I run this every minute during load testing:

#!/usr/bin/env bash

stamp=$(date +%Y-%m-%d_%H%M%S)

wget -O traceback_$stamp.txt 'http://localhost:6060/debug/pprof/goroutine?debug=2'

go tool pprof -svg http://localhost:6060/debug/pprof/heap > RAM_${stamp}.svg

go tool pprof -svg http://localhost:6060/debug/pprof/profile > CPU_${stamp}.svg

4. If your server will be making HTTP request (such as authenticating users against some other backend), your server will begin to fail badly without setting the MaxIdleConnsPerHost on your http.Client. Read this: https://paperairoplane.net/?p=556 This tip alone would have saved me too many hours of frustration!

You will probably want to automate this, maybe with something like Ansible, so you can build up and tear down many servers as-needed. If you have an account with Digital Ocean, they have a good API for this, and I've created a rudimentary command-line tool in Go that lets me create, destroy, and list my droplets. I can clean it up a bit and open-source it if there's interest.

To answer your specific questions:

Pool of ports: You shouldn't have to specify ports. The ws:// or wss:// connections work like HTTP connections -- they connect at 80 or 443, but they then are assigned a higher port for their session. You shouldn't have to worry about it. 

Close or re-use: Your test client should behave badly, at least sometimes. Real-world clients aren't going to call Close() all the time. They may just fall off the internet. I don't think you can re-use a connection. I could be wrong, but in any case I suspect it's irrelevant since they will be garbage collected.

Is a connection being used?: Use the SetWriteDeadline and/or SetReadDeadline values for your sockets, so if a socket goes bad you will know, either by a write timing out or no new data for some period of time.

Hope this helps.