[Release] Perspectives Server v3.5

[Dave Schaefer]

I'm happy to announce that Perspectives Server version 3.5 has been released!

This update adds a --logfile switch to control proper output to log files, and places a limit on how large log files can grow. This fixes a longstanding issue that has prevented the default notaries from remaining stable long-term.

This update also fixes a threading issue with the scanner, and performs a lot of code cleanup and refactoring.

With this update complete we are all set to bring the default notaries back online (and hopefully add https as well).

This update contains no data changes, only code changes, so upgrading is easy:

1. Stop your server

2. Sync new code with 'git pull'

3. Start your server.

These steps have also been included in the doc/upgrades directory.

As always, if you have any questions or comments just ask!

Full changelog

---

3.5

-----

+ Add --logfile switch for scanner and server to handle logging to a file on disk.

This lets us limit how much disk space is used for logging, so it doesn't run out of control.

* Change default admin scripts to use new --logfile switch

* Clean up admin scripts

* Fix multiple sources of logging, printing, and traceback to use logging module

* Fix print statements to use print function

* Fix threading bug with scanner: properly lock objects when recording scan results

* Organize unit tests: use one test module for each code module being tested

This makes it much easier to see what is being tested.

* Convert old-style classes to new-style classes

* Add missing license header

* Determine whether sockets block based on OS (nonblocking sockets don't work on Windows)

* Remove some global variables and state from threaded_scanner

* Fix list_services code to be testable, and add minimal tests

* Minor code formatting and whitespace cleanup

* Add some docstrings and documentation

[Chris]

Thanks for all your hard work on this.

I've updated my 4 public notary servers (listed at https://noc.schulte.org/perspectives.html) to this version.  Since the upgrade, they all appear to respond much more quickly as I browse to secure sites, including sites that already have current cached observations.

Well done.
Chris

[Dave Schaefer]

Hey Chris, thanks for the note! I'm glad its working well for you :)

That's fascinating if performance has changed. Do you run with nginx, pycaching, or any other type of caching? In particular, do your notaries write to local files on disk? Do you use the default admin shell scripts?

I did not do any performance testing for this release; I was only worried about correctness ;) But I wonder if changing the code to use the logging module and constraining log file size had made a noticeable difference.

[Chris]

Dave, yes the unexpected performance boost was welcome.  I'm using stock notary code that directly binds to TCP/8080, keeping the sqlite DB on local disk and using the suggested cron for daily updates.

I haven't measured the actual service's response time to a GET request for /?host=foo.com, so my perceived change could also be something on the client (plugin) side.  I do note that the plugin was recently updated, so perhaps some interaction between the server and client code is just working better for my particular setup.

Regardless of the reason, I wanted to chime in to let you know and say thanks.  I'm looking forward to seeing where the project goes, such as adding TLS support for notary<-->client communications and expanding the plugin to support to other browsers.

Chris

[Dave Schaefer]

Hey Chris, thanks I really appreciate it :) It's kind words like that that keep me going ;)

Yes, upgrading the default notaries is ready to go and can be done at any time, but I don't currently have access to the networknotary.org DNS controls, so I'm not able to easily spin up new machines and set them running. I'm trying to work with Dan to get that sorted out so we can make the notaries responsive again.

When we bring the default notaries back online I'm leaning strongly towards *not* carrying forward their old data and starting them fresh. I plan to post a full writeup with details, but the short version is that a combination of bugs in the version2 notary software means we can't vouch 100% for the accuracy of the data. It would be best to start over.

In the meantime I'm working towards refactoring the scanner[1] so it works correctly can can be unit tested. The steps toward that involve some other internal fixes to logging and data handling. I'll likely post a small release or two soon on my way to the scanner work.

Have a great week!

Dave

[1] https://github.com/danwent/Perspectives-Server/issues/45