DNS event

[Richard Penner]

It seems like there was an event which resulting in DNS failures with the metamath.org domain, with us2.metamath.org being directed to some New York cable subscriber and other addresses failing outright.

dig us2.metamath.org @a0.org.afilias-nst.info  +authority

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.2.rc1.fc15 <<>> us2.metamath.org @a0.org.afilias-nst.info +authority

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29757

;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 6, ADDITIONAL: 12

;; WARNING: recursion requested but not available

;; QUESTION SECTION:

;us2.metamath.org. IN A

;; ANSWER SECTION:

us2.metamath.org. 86400 IN A 209.150.45.36

;; AUTHORITY SECTION:

org. 86400 IN NS a0.org.afilias-nst.info.

org. 86400 IN NS c0.org.afilias-nst.info.

org. 86400 IN NS d0.org.afilias-nst.org.

org. 86400 IN NS b2.org.afilias-nst.org.

org. 86400 IN NS b0.org.afilias-nst.org.

org. 86400 IN NS a2.org.afilias-nst.info.

;; ADDITIONAL SECTION:

a0.org.afilias-nst.info. 86400 IN A 199.19.56.1

a2.org.afilias-nst.info. 86400 IN A 199.249.112.1

b0.org.afilias-nst.org. 86400 IN A 199.19.54.1

b2.org.afilias-nst.org. 86400 IN A 199.249.120.1

c0.org.afilias-nst.info. 86400 IN A 199.19.53.1

d0.org.afilias-nst.org. 86400 IN A 199.19.57.1

a0.org.afilias-nst.info. 86400 IN AAAA 2001:500:e::1

a2.org.afilias-nst.info. 86400 IN AAAA 2001:500:40::1

b0.org.afilias-nst.org. 86400 IN AAAA 2001:500:c::1

b2.org.afilias-nst.org. 86400 IN AAAA 2001:500:48::1

c0.org.afilias-nst.info. 86400 IN AAAA 2001:500:b::1

d0.org.afilias-nst.org. 86400 IN AAAA 2001:500:f::1

;; Query time: 120 msec

;; SERVER: 199.19.56.1#53(199.19.56.1)

;; WHEN: Wed Oct 23 10:44:06 2019

;; MSG SIZE  rcvd: 452

dig us2.metamath.org @ns2.systemdns.com  +authority

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.2.rc1.fc15 <<>> us2.metamath.org @ns2.systemdns.com +authority

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10599

;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; WARNING: recursion requested but not available

;; QUESTION SECTION:

;us2.metamath.org. IN A

;; ANSWER SECTION:

us2.metamath.org. 300 IN A 173.76.107.169

;; Query time: 72 msec

;; SERVER: 64.98.148.16#53(64.98.148.16)

;; WHEN: Wed Oct 23 10:46:25 2019

;; MSG SIZE  rcvd: 50

[David A. Wheeler]

On Wed, 23 Oct 2019 11:23:56 -0700 (PDT), Richard Penner <richar...@gmail.com> wrote:
> It seems like there was an event which resulting in DNS failures with the
> metamath.org domain, with us2.metamath.org being directed to some New York
> cable subscriber and other addresses failing outright.

Yes. I can't reach us.metamath.org or www.metamath.org.

Norm: Do you need help?

--- David A. Wheeler

[Alexander van der Vekens]

http://us2.metamath.org is also not working.

May this also be the reason why the Travis build for my pull request failed?

[vvs]

There is some problem with metamath.org domain registration: https://icann.org/epp#clientHold

[Richard Penner]

That is correct -- the bad record for us2.metamath.org is probably just exposed by this cilentHold breaking the domain which Norm has had registered since Feb 2001.

Probably, only Norm can resolve this issue at this point.

This status code tells your domain's registry to not activate your domain in the DNS and as a consequence, it will not resolve. It is an uncommon status that is usually enacted during legal disputes, non-payment, or when your domain is subject to deletion.

Often, this status indicates an issue with your domain that needs resolution. If so, you should contact your registrar to resolve the issue. If your domain does not have any issues, but you need it to resolve, you must first contact your registrar and request that they remove this status code.

Ref: https://www.icann.org/resources/pages/epp-status-codes-2014-06-16-en#clientHold

On Wed, Oct 23, 2019 at 2:10 PM vvs <vvs...@gmail.com> wrote:

There is some problem with metamath.org domain registration: https://icann.org/epp#clientHold

--
You received this message because you are subscribed to a topic in the Google Groups "Metamath" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/metamath/Gfr7EUxfjAY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to metamath+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/metamath/9f5542a5-13ad-4ae1-92ce-920893c1fb63%40googlegroups.com.

[Norman Megill]

David has alerted me to this.

I opened a ticket with the DNS provider.  I have no idea what is happening. I checked the DNS setup form on their site and everything looks fine.  The service is provided by domainmonger.com, and metamath.org is paid for through 02/04/2020.

The last change I made was yesterday morning at 8am when I updated the IP address of cn.metamath.org with no other changes.  I believe it was working fine yesterday, could some one confirm?  When is the last time that anyone has successfully accessed the site?

I will post here when they respond.

P.S. It looks like my email is also not working; I have received nothing since 9AM this morning.  It is forwarded to me from MIT via the us.metamath.org server.

Norm

[Norman Megill]

An alternate email that I normally don't check is nme...@yahoo.com.

[Norman Megill]

Forget that.  Yahoo won't let me log in without verifying via n...@alum.mit.edu, which doesn't work now.
I'll check this group for any messages for me.

Norm

[Norman Megill]

I have tried to reach domainmonger.com via phone, both of their numbers, and left urgent messages about my support ticket asking them to call me on the phone.  There is no way to reach a human.  They will close in 40 minutes based on the hours posted on their site.

David, could you see if http://173.76.107.169/index.html works for reaching the site?  That is the direct IP of us2.metamath.org.  I can't test it meaningfully because it's the IP of my house.  If it works, could you temporarily change the Travis script to retrieve the metamath program from there?

Also, other people may feel free to use the direct IP until this problem is resolved.

[Mario Carneiro]

the IP works for me

--
You received this message because you are subscribed to the Google Groups "Metamath" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metamath+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/metamath/e5a5e144-48eb-4c7c-8d79-22b84a7a71d3%40googlegroups.com.

[David A. Wheeler]

>David, could you see if http://173.76.107.169/index.html works for
>reaching
>the site? That is the direct IP of us2.metamath.org. I can't test it
>meaningfully because it's the IP of my house. If it works, could you
>temporarily change the Travis script to retrieve the metamath program
>from
>there?

It works. I will change it once I get to a computer with a keyboard.


--- David A.Wheeler

[Richard Penner]

After two phone calls, I have emailed tech support at Afilias.Net and have BCCed NM and DW in case they want to be added to the discussion/incident ticket.

--
You received this message because you are subscribed to a topic in the Google Groups "Metamath" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/metamath/Gfr7EUxfjAY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to metamath+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/metamath/cc30ec87-92e0-420a-8a42-f04a42ff8738%40googlegroups.com.

[David A. Wheeler]

On Wed, 23 Oct 2019 15:27:32 -0700 (PDT), Norman Megill <n...@alum.mit.edu> wrote:
> David, could you see if http://173.76.107.169/index.html works for reaching
> the site? That is the direct IP of us2.metamath.org. I can't test it
> meaningfully because it's the IP of my house. If it works, could you
> temporarily change the Travis script to retrieve the metamath program from
> there?

Done.

All: If you have outstanding pull requests, "git checkout develop; git pull" to sync your develop branch.
Then "git checkout YOUR_BRANCH; git merge develop" to bring in the recent change
to remove DNS names. Then "git push" to push out the change, which should now pass.

--- David A. Wheeler

[Richard Penner]

WHOIS reports a record change at 2019-10-23T14:19:04Z with Z standing for UTC. Possibly that was the timestamp of the record change that put the clientHold on the record.

The time is now 2019-10-24T09:00:00Z and while I having found a WHOIS copy newer than 2019-10-23T17:34:45Z I see that www.metamath.org and us2.metamath.org are resolving for me. Even cn.metamath.org works. 

And dig us2.metamath.org @a0.org.afilias-nst.info  +authority points to the correct name servers for the domain and no longer gives the bogus NY IP.

WHOIS update -- checking with PIR gives me a WHOIS lookup closer to the source of truth: https://thenew.org/get/find-my-org/whois/?domain=metamath.org

Updated Date: 2019-10-24T04:41:07Z and no clientHold. Happy Computing!

[Norman Megill]

The DNS is working again.

Here is the message from the DNS provider:

Norman Megill (Norman Megill),

The issue was that the Registry suspended the domain because they sent an email to the admin account, nm+meta...@alum.mit.edu, and it bounced. We just changed the email to ours to unsuspend it, and it is now working. What email address would you like to use instead?

Domainmonger
Technical Support

The problem is that the alum.mit.edu server has started bouncing emails with a "+" field, so I'll take out the "+metamath.org" in the Registry record.  This is unfortunate because I used that to identify where spam originated, making it easier to filter.  I guess that also explains why I've had less spam in the last few months.

Thank all of you for your help.

Norm

[Alexander van der Vekens]

us.metamath.org is OK, but us2.metamath.org/ is still not working ...

On Thursday, October 24, 2019 at 3:25:08 PM UTC+2, Norman Megill wrote:

The DNS is working again.

Here is the message from the DNS provider:

Norman Megill (Norman Megill),

The issue was that the Registry suspended the domain because they sent an email to the admin account, n...@alum.mit.edu, and it bounced. We just changed the email to ours to unsuspend it, and it is now working. What email address would you like to use instead?

[vvs]

Works fro me. It's just probably needs time to propagate through all DNS caches.

[Benoit]

Now that it's back to normal, I have two naive questions:

1) Isn't the purpose of mirror sites to serve as replacements when an event like this occurs ?  However, during the event, it seems the mirror sites were unavailble either.  Can we do something about it ?

2) Because of the event, the Travis verification was unavailable, because every time Travis verifies a PR, it needs to download things from us(2).metamath.org.  Would it be possible to make local copies to avoid this? (for the metamath program, it is already on GitHub)

Thanks,
Benoît

[vvs]

They serve their purpose as host mirrors. But there is no DNS domain mirror and there is no e-mail mirror either.

There is always room for improvement of course, but everything has their costs as well.

[Richard Penner]

In the initial post in this thread, the rogue us2.metamath.org IP had a Time-To-Live of 86400 seconds (1 day).  DNS servers which honor that will need that long before they fetch the new value.

[Norman Megill]

On Thursday, October 24, 2019 at 11:21:58 AM UTC-4, Benoit wrote:

Now that it's back to normal, I have two naive questions:

1) Isn't the purpose of mirror sites to serve as replacements when an event like this occurs ?  However, during the event, it seems the mirror sites were unavailble either.  Can we do something about it ?

They were unavailable because the entire domain metamath.org was suspended by the Registry.  That includes all subdomains (the mirrors).  The only way I see to prevent this is to have a separate domain name for each mirror (e.g. us--metamath.org) but that would cost more money.

Keep in mind that this is the first time this has happened in 18 years.

It bothers me that a simple bounced email can trigger the Registry to take down a domain without notice, with no effort to contact me.  The Registry record has my phone number and worst case my snail mail address.  There are many reasons an email can bounce.  Imagine if amazon.com or google.com was suspended because they forgot to update the contact email for an employee who left.

2) Because of the event, the Travis verification was unavailable, because every time Travis verifies a PR, it needs to download things from us(2).metamath.org.  Would it be possible to make local copies to avoid this? (for the metamath program, it is already on GitHub)

It doesn't matter to me, but if someone wants to do that it's fine with me.  I hope this problem will never happen again.

Norm

[vvs]

BTW, there is no need to put the burden of maintaining redundancy on Norman. Anyone could register alternative domain names and maintain it themselves. Because this is once in lifetime event they might even get away with this using free services, e.g. they can use alternative DNS roots: https://en.wikipedia.org/wiki/Alternative_DNS_root

But this will require dedicating some network, computing and human resources for this task. So, if anyone really wants such redundancy guarantees they should put their money where their mouth is and do this as a community service. Or there always is a possibility to just use IP addresses in case of emergency. For me this is an easy choice.

[Benoit]

Just in case this wasn't clear enough: these were two naive questions, as I wrote.  Maybe I should have written "genuinely naive" or "naively genuine" to be clearer.  Anyway, thanks Norm for the explanations.

Benoît

[vvs]

No offense either. This was just my genuine attempt to explain to casual reader what it might take to make a redundant service. And who knows, may be someone will do this eventually.

P.S. Internet is such a place where strangers never know for sure how to interpret real intentions of everyone else. But is this any different from a real world? I wonder to which extent this could explain interest to formal logic in ordinary people.

[Alexander van der Vekens]

us2.metamath.org/ works for me now, too!