New Perspectives test server to evaluate nginx - please try it!

[Dave Schaefer]

Hey everyone,

I've created a new notary to test out a potential nginx configuration.
I'll run this for a few days, and if it works well I'd like to set up
nginx on the default notaries, to improve performance.

If you'd like to help out it would be great if you could add this
notary to your list. The server is:

http://ec2-54-66-132-58.ap-southeast-2.compute.amazonaws.com/

-----BEGIN PUBLIC KEY-----
MIHKMA0GCSqGSIb3DQEBAQUAA4G4ADCBtAKBrAGa3NpsLJPJt4+D2NoidbZq0sx4
ALtpaczKGdjgvtddQRq2ZC6vtdFKRtNy2Ml3pmLIVA2mndCQh1UCiG9OdJz2ZiZ2
BXWb82gyL3qpsSxUG41SyxZxvN4A7ABFASOolq3udNWr7ufIRVT2pCr0dV6IJOF/
/L6BJ8WzBeLVjID8SpnTc9wbwpGcvEWr0U6eEiJ/AyiClMYn2jtOnAW9l+rz1rR8
SJW79GNd6WUCAwEAAQ==
-----END PUBLIC KEY-----


The notary hasn't been primed with many sites, so you may see several
404s for the first while as it starts gathering data. My hope is that
this notary will have better performance once it gets running. If you
add it, please give me a shout about your experience and how it's
working for you!

The machine is still under testing, so I won't make any guarantees
about availability, but I'll try to keep it stable and running. Let me
know if you experience any issues.

Cheers!

[Aaron Bugher]

On Sunday, December 28 at 14:MPM, Dave Schaefer wrote:
> Hey everyone,
>
> I've created a new notary to test out a potential nginx configuration.

Thank you for your work.

...

> If you'd like to help out it would be great if you could add this
> notary to your list.

...

> If you add it, please give me a shout about your experience and how
> it's working for you!

I added it. Got a consistent result for google. I don't see
performance data to attach, but very little time passed, subjectively.

--
Aaron Bugher

[Dave Schaefer]

Thanks Aaron. I'm glad it's working for other people also ;)

If anyone wants to see the nginx config file being used, I have added
it to the official git repo here:
https://github.com/danwent/Perspectives-Server/blob/master/doc/guides/nginx/notary.nginx.conf

The only difference is that this new server also listens on port 8080,
just in case.

There is also a very brief guide to setting up nginx with Perspectives
server under Ubuntu -
https://github.com/danwent/Perspectives-Server/blob/master/doc/guides/nginx/guide_nginx.md


Cheers!

[Carl Antuar]

The new notary is added and working for me (with very little history, as mentioned).

However, all my quorum checks are failing, because I'm not getting any responses from notaries 2 through 7.

[Dave Schaefer]

Hey Carl, thanks for chiming in. I'm glad it's working for you.

Yes, my goal with setting up this test notary is to evaluate the
stability and performance of having nginx as a proxy in front of the
notaries. Using nginx would be an easy way to dramatically improve
performance and stability without having to update any of the code or
upgrade the notaries. We could then perform upgrades in the background
once things were stable, but get the notaries working again so
everyone can get replies.

@DanWent - Hey Dan I've pinged you privately about granting me ssh
access to the notary2-notary7 machines; let me know if you would be
comfortable with this. Otherwise we should be able to set up nginx for
the default notaries using the steps in the nginx guide[1]. I'm happy
to provide help or whatever it takes to set this up.


[1] https://github.com/danwent/Perspectives-Server/blob/master/doc/guides/nginx/guide_nginx.md

[Dave Schaefer]

Okay, to share some test results:

I ran some load tests on an Amazon ec2 notary with and without nginx.
My test methodology is below for any one that's interested but the
results are good:

Regular version 2.0 notary, with no caching:
12 Requests/s average (744 valid responses served), 33% request
timeouts (4037 timeouts), 1178ms average response time.

Exact same notary, with nginx caching:
128 Requests/s average (7,725 valid responses served), 0.4%
timeouts (57 timeouts), 516ms average response time.

So adding nginx in front of the machine allows it to serve 10x as many
requests per second, with a 1/3 response time.

This is pretty fantastic. These tests were using a t2.micro ec2
instance, which has limited processing and network capabilities, but
they are still good results. I'd like to run more extensive tests to
compare various other notary settings, and we may be able to squeeze
even more performance out of nginx by changing the configuration, but
I'm comfortable using these tests results to say: using nginx as a
caching proxy improves performance - we should add it to all of the
default ec2 notaries.


Test Methodology
---

1. Create a list of 250 hosts, 240 of which are valid domains and 10
which are invalid
2. Set up a test using blitz.io that will hammer a given server with
notary requests: 200 requests/second for 60 seconds (60 seconds is the
longest test time available under their free account tier).
3. Make sure the notary has valid database entries for all of the valid domains.
4. Run the test 10 times against each notary configuration.
5. Take the average of the results

[Carl Antuar]

Why can't you run the Python server in Nginx directly?

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "perspectives-dev" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/perspectives-dev/LfGrJHcPCOc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> perspectives-d...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Dave Schaefer]

Carl asked:

> Why can't you run the Python server in Nginx directly?

Hey Carl, if this is possible I'm all ears! Contributions are always welcome :)

As it stands it may take some effort to get all of the notary
functionality into an nginx module. Since we already have the notary
server software that does everything we need - accept requests, find
the right data, convert the data to XML, sign it, and send data out -
it seemed much easier to simply add nginx as a proxy and cache in
front of the already-existing system. nginx works really well as a
proxy and cache, and this way it doesn't have to know anything about
the rest of our set up.

If you have ideas though, please shout!

[Carl Antuar]

OK, are the existing notaries running in eg Apache using mod_wsgi, or are they straight Python?

If they're using WSGI, then Nginx could do the same. If straight Python, then never mind, proxying is probably the best approach. Although you might want to look into uWSGI, rather than a plain HTTP proxy.

Simple uWSGI configuration sample: https://www.lucasrolff.com/nginx/using-nginx-as-a-proxy-to-uwsgi-applications/

Suggested usage pattern for uWSGI: http://cramer.io/2013/06/27/serving-python-web-applications/

uWSGI caching cookbook: https://github.com/unbit/uwsgi-docs/blob/master/tutorials/CachingCookbook.rst

[Dave Schaefer]

Hey Carl, the existing notaries are all using straight python, with
CherryPy as a webserver, and no apache or proxying of any kind.

Thanks for the links! I will have a look. It would be good to do some
performance comparisons.

[mic...@yanovich.net]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Also, I recently set up the de.yano.nu notary server to use the nginx
proxy as well.

Information for it is available at: http://de.yano.nu/



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJUvFfYAAoJEKBpQL3CDq4dfaIQAK8AzkfcGwzmIX1ldDDvD811
ICdNiYVlyqDwGH43qBnFZyEBponyAGD9T1EXiqBrtiYfI+6q/t9YZYw8pvlf7AR7
05RB9GfGcICO9yBT/lJFE20q90kHVgnkqVN3U11ax1CGeYzjCeTIGYLgxNaF4k1/
d6TvF96DHElU3xOkDQt2kZGqmfZTgmSXI0G4lCewyhBgaWtYxzvmPj1FLqPw9ZQX
bDZ7acyCRM7bDdADLI9SRHLFMAWvS32xv50tYjJeGfoIw1ALrhCu+tmkUTMWWkVS
WVtwAPYSpt9vFq9wWaLKH3x2l3339xQZzReuS83i4P7KodrOGEMh/4v3PJhDtaiR
TSfvs3Jhcy56D7sFNCE/edAYZK3zp0Ulf430jkNX61meyo/QRu7LjjvD/9XdRR0T
rYstfeuRfoiHIk7KSF7ckpfhziPsLy2OFhNzECH5ujdjXobhofj0ee5Zhj6b9jpn
+g8wOcCmSHv9iGQHC3dmyz6bfZNTi2muOx9r3x4C85svic29fHfMSzRP7Wj/PmS0
0SdXCUDiaI1ZmyYv/8ppznEHANo/b8RF0j/LWsRRS2Gn2JVXGBvEacs6F2rsHym4
AFIjb+G7SP6o6TKW31RDqlpzg/0LfdjIY/cGMn9iYwb9Yi24KrneaKZ5nkkgIuMp
vUdUWz3AelPHVzbTPDvO
=7OkN
-----END PGP SIGNATURE-----