Perspectives add-on never trusting anything

[Will Dormann]

Hi folks,

I love the concept of the project and it's sorely needed.  However,
Perspectives is never trusting any site that I visit.  From any network
or on any platform.

See the attached screenshot for example. There is 100% agreement between
the 4 contacted notaries and the thumbprint that my browser is seeing.
However:

- The Perspectives icon is a red X
- The dialog says "Warning: Perspectives has NOT seen this certificate
consistently"

The attached screenshot is from Perspectives 4.5.2 on Firefox 36, but I've confirmed that Perspectives 4.6 behaves the same way.
Is this a bug?   


Thanks
-WD

[Carl Antuar]

No, it's not a bug. The problem is that over half your notaries are not responding, which is treated as a 'No' vote.

[Will Dormann]

So is there a problem with the availability of the notaries?  I've tried from three different networks, and I've never received a "yes" vote for a single site that I've visited.

[Dan Wendlandt]

Hi folks,

I've cleaned up some of the notaries, which had full disks.  It will now take the a bit of time to chew through doing an observation (or two or three, depending on your quorum duration) before they start showing up as fully green again.   Sorry for the issues.

Dan

--
You received this message because you are subscribed to the Google Groups "perspectives-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to perspectives-d...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
650-906-2650
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Will Dormann]

Hi Dan,

I'm still generally not seeing responses from the networknotary.org
notaries (and subsequently no quorum) using the browser add-on. Are
those machines too heavily loaded?


-----
$ time curl
"http://perspectives7.networknotary.org:8080/?host=www.reddit.com&port=443&service_type=2&"
<notary_reply
sig="ACD83RVcOptQAe/U2Ao+uI4KFplsbl4P/dfJqEsTpSGY58CsP+arUWJvJN1fIWOcKIY3xlAeX6O6CaIm1wrGp8x0L7VfXrfxptHOljhuyfIj1yivSUWUxIhCwR6REyLQNO1TrNAOql7u1bIFR1gyBPL2BwOqPBqAT0V/h8yAhdTK6UmSoTXB1tUpLOWMtsh1F0l/0dsM6BBbaKe2s1mJ4CpvFB5A4dU63+pp2A=="
sig_type="rsa-md5" version="1">
<key fp="01:95:04:0f:22:87:23:6c:f5:ad:9e:f7:29:82:1c:b9" type="ssl">
<timestamp end="1375801030" start="1346116122"/>
</key>
<key fp="2b:d8:d1:db:44:d1:e3:6c:28:40:d2:db:8d:61:44:6d" type="ssl">
<timestamp end="1383019344" start="1375801031"/>
<timestamp end="1383105639" start="1383063424"/>
<timestamp end="1383192063" start="1383149014"/>
<timestamp end="1383541474" start="1383407853"/>
</key>
<key fp="31:98:03:ef:04:c5:80:c8:56:fd:fc:4e:b1:cc:fb:18" type="ssl">
<timestamp end="1316123108" start="1309249182"/>
</key>
<key fp="4c:04:54:a0:27:00:1d:65:bc:92:5e:e1:f5:4f:72:f8" type="ssl">
<timestamp end="1346116121" start="1316123109"/>
</key>
<key fp="d1:88:0a:39:00:fe:93:f8:fc:25:0f:fd:25:f8:44:63" type="ssl">
<timestamp end="1383063423" start="1383019345"/>
<timestamp end="1383149013" start="1383105640"/>
<timestamp end="1383407852" start="1383192064"/>
<timestamp end="1427849255" start="1383541475"/>
</key>
<key fp="e6:80:8b:5b:7a:c9:5c:0c:6b:40:b7:00:3a:cd:8e:44" type="ssl">
<timestamp end="1427849256" start="1427849256"/>
</key>
</notary_reply>

real 2m22.821s
user 0m0.007s
sys 0m0.011s
-----

If a notary takes close to 2.5 minutes to respond with results, that's
that's really not going to work, right?

I can't seem to consistently reproduce it, but I've also seen this:

$ telnet perspectives2.networknotary.org 8080
Trying 177.71.234.231...
telnet: connect to address 177.71.234.231: Connection refused
telnet: Unable to connect to remote host


Thanks.
-WD

On 4/1/15 1:42 AM, Dan Wendlandt wrote:
> Hi folks,
>
> I've cleaned up some of the notaries, which had full disks. It will now
> take the a bit of time to chew through doing an observation (or two or
> three, depending on your quorum duration) before they start showing up
> as fully green again. Sorry for the issues.
>
> Dan
>
>
> On Tue, Mar 31, 2015 at 7:46 PM, Will Dormann <wdor...@gmail.com
> <mailto:wdor...@gmail.com>> wrote:
>
> So is there a problem with the availability of the notaries? I've
> tried from three different networks, and I've never received a "yes"
> vote for a single site that I've visited.
>
>
> On Tuesday, March 31, 2015 at 10:28:30 PM UTC-4, Carl Antuar wrote:
>
> No, it's not a bug. The problem is that over half your notaries
> are not responding, which is treated as a 'No' vote.
>
> On Wednesday, 1 April 2015 05:58:47 UTC+10, Will Dormann wrote:
>
> The attached screenshot is from Perspectives 4.5.2 on Firefox 36, but I've confirmed that Perspectives 4.6 behaves the same way.
> Is this a bug?
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "perspectives-dev" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to perspectives-d...@googlegroups.com

> <mailto:perspectives-d...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> 650-906-2650
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "perspectives-dev" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/perspectives-dev/BAhXv9vY4EY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> perspectives-d...@googlegroups.com
> <mailto:perspectives-d...@googlegroups.com>.

[Dave Schaefer]

Hi Will, thanks for the note. I am glad you are interested in Perspectives!

You have indeed identified the current primary problem with the Perspectives project - the default notaries are in sore need of repair. This is not a bug per se (or at least, not the result of one single bug), but it is *definitely* something we need to address. We have been working towards fixing this for quite some time.

One issue is that the original default notaries are running a very old version of the software. The old version was not optimized for performance and simply cannot keep up with a large number of requests. As you note, it can take a *very* long time for those notaries to reply, which is not good.

The fact that you're using 'curl' and 'telnet' suggests that you are good with tech and comfortable with technical details :) I can give a brief overview of our current issues in case you are interested. Let me know if you prefer more or fewer details.


The fastest way to fix the default notaries is likely to add caching so they can quickly return results without re-doing the certificat calculations each time. I have both implemented some in-memory caching and experimented with setting up nginx as a cache. Either of these should work.

One currently blocking issue is that I do not have access to change the DNS entries for the notary machines; Dan and I are still working out account permissions to enable that. So it is currently not possible for me to spin up a new notary and switch it over.

Another issue is that simply cleaning up the existing notaries is not a stable, long-term solution, because the notary software currently does not limit the growth of its logs. Even if a machine is fixed, it quickly fills its hard drive and goes down again. This is definitely something we need to patch and address. There is an existing ticket on GitHub - https://github.com/danwent/Perspectives-Server/issues/25

A third issue is that the old version of the notary software (anything before version 3.2) has a very bad bug: when it scans websites for certificates and finds something new, the scanner automatically backfills old observation data with no limit, regardless of whether that data can be validated. This was fixed as GitHub bug #23 - https://github.com/danwent/Perspectives-Server/issues/23

Because of #23, I believe that the data from the old/existing default notaries should likely be scrapped, and we should start over with a new set of machines and data. If, for example, a notary's hard drive filled up and it was out of commission for 100 days, if it starts up again and makes an observation, it would backfill the observation data and claim that it had seen a given certificate for the past 100 days, when in fact it had not. This is very bad.

@DanWent - because of the above, I believe our best bet is to either scrap the old data and then upgrade the default notaries, or to create some new notaries to use as the defaults and swap them over. Thoughts?




At any rate, I really do appreciate your interest in the project. I believe it can be made to be funtional and useful! But we are currently in the process of fixing several things.

There is also a Roadmap outlined on the wiki here - https://github.com/danwent/Perspectives/wiki/Perspectives-Roadmap . The summary is that there are also some blocking client issues to fix so we can publish new versions on addons.mozilla.org.


Unfortunately, some personal commitments prevent me from making these changes for the next couple of weeks. I will definitely return to Perspectives work and continue to fix our setup as soon as I can.


I hope that helps to at least explain our current situation. My apologies that we are not in a better state.

Feel free to continue asking questions!
Dave