Problems accessing 8.8.8.8 (and 8.8.4.4) from within the Spark network (and N4L)

[Pete Mundy]

Just a heads-up for those who may be affected. We've seen widespread inability to ping and lookup DNS from 8.8.8.8 ad 8.8.4.4 since 12:42pm today from a bunch of nodes on the Spark (retail) network, as well as on N4L.

Access via other ISPs appears to be unaffected.

I'm sure it will be resolved soon enough, but in the meantime if this is causing you pain then changing to another DNS server should resolve (eg 1.1.1.1 is working fine).

Pete

[Tim Harper]

Working ok at Mt Aspiring for 8.8.8.8 and 8.8.4.4.

regards,

Tim Harper


Phone 03 443 5167 (messages cannot be left on this number)
Mobile 027 443 1236

t...@mtaspiring.school.nz
www.mtaspiring.school.nz

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Simon Wright]

We are having weird issues not being able to access some sites.

We can ping 8.8.8.8 but we can't ping spark/n4l dns 127.56.237.1 or 210.55.111.1

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

DISCLAIMER
This e-mail is intended for the addressee only and may contain information which is subject to legal privilege. This e-mail message and accompanying data may contain information that is confidential and subject to privilege. Its contents are not necessarily the official view Otago Boys’ High School or communication of the Otago Boys’ High School. If you are not the intended recipient you must not use, disclose, copy or distribute this e-mail or any information in, or attached to it. If you have received this e-mail in error, please contact the sender immediately or return the original message to Otago Boys’ High School by e-mail, and destroy any copies. Otago Boys’ High School does not accept any liability for changes made to this e-mail or attachments after sending.

[Jonathan Webster]

Hi, looks to be an issue in Spark's core network - they have identified it and are working towards a resolution... no exact details or ETA at this stage sorry

Simon, I'm guessing that second was a typo, it's  122.56.237.1 :) 

--

Jonathan Webster

Solution Architect

The Network for Learning Ltd

M +64 22 040 3300  P 0800 LEARNING

A Suite 306, Geyser Building, 100 Parnell Road, Parnell, Auckland 1052

A PO Box 37118, Parnell, Auckland 1151  n4l.co.nz

[Andrew Godfrey]

All sorts of weirdness since midday.

🤪

[Pete Mundy]

I think your first IP there is wrong Simon. You said 127.56.237.1, but I think it should be 122.56.237.1?

I just tested 122.56.237.1 & 210.55.111.1 from one of the sites in which we're seeing the issue and we're seeing problems to those servers as well. We can't ping either of them, and only the second one (ns2.xtra) is answering DNS.

Here's some output from a test device to illustrate our view:


ladmin@test-box ~ $ ping -c 1 -W 4 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

ladmin@test-box ~ $ dig apple.com @8.8.8.8 +short

; <<>> DiG 9.9.5-9+deb8u6-Raspbian <<>> apple.com @8.8.8.8 +short
;; global options: +cmd
;; connection timed out; no servers could be reached



ladmin@test-box ~ $ ping -c 1 -W 4 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=13.5 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 13.571/13.571/13.571/0.000 ms

ladmin@test-box ~ $ dig apple.com @1.1.1.1 +short
17.142.160.59
17.172.224.47
17.178.96.59



ladmin@test-box ~ $ ping -c 1 -W 4 127.56.237.1
PING 127.56.237.1 (127.56.237.1) 56(84) bytes of data.
64 bytes from 127.56.237.1: icmp_seq=1 ttl=64 time=0.123 ms
--- 127.56.237.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.123/0.123/0.123/0.000 ms

ladmin@test-box ~ $ dig apple.com @127.56.237.1

; <<>> DiG 9.9.5-9+deb8u6-Raspbian <<>> apple.com @127.56.237.1
;; global options: +cmd
;; connection timed out; no servers could be reached



ladmin@test-box ~ $ ping -c 1 -W 4 210.55.111.1
PING 210.55.111.1 (210.55.111.1) 56(84) bytes of data.
--- 210.55.111.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

ladmin@test-box ~ $ dig apple.com @210.55.111.1 +short
17.172.224.47
17.178.96.59
17.142.160.59


Definitely some weirdness going on there!

[Pete Mundy]

It just simultaneously came right across all of the test nodes at the same time for us. Including the weirdness to ns1/ns2.xtra.

It must have been escalated relatively quickly!

I hope it's smooth sailing for others now too :)

Pete

Begin forwarded message:

From: Pete Mundy <pe...@fiberphone.co.nz>

Subject: Re: [techies-for-schools] Problems accessing 8.8.8.8 (and 8.8.4.4) from within the Spark network (and N4L)

Date: 22 May 2018 at 2:17:31 PM NZST

To: "techies-f...@googlegroups.com schools" <techies-f...@googlegroups.com>

<snip>

[Mike Etheridge]

Can’t see them from here (PNGHS Manawatu) either. Not happy.

EGs-Laptop-2:~ EG$ ping 122.56.237.1

PING 122.56.237.1 (122.56.237.1): 56 data bytes

Request timeout for icmp_seq 0

Request timeout for icmp_seq 1

Request timeout for icmp_seq 2

Request timeout for icmp_seq 3

Request timeout for icmp_seq 4

Request timeout for icmp_seq 5

Request timeout for icmp_seq 6

^C

--- 122.56.237.1 ping statistics ---

8 packets transmitted, 0 packets received, 100.0% packet loss

EGs-Laptop-2:~ EG$ ping 210.55.111.1

PING 210.55.111.1 (210.55.111.1): 56 data bytes

Request timeout for icmp_seq 0

Request timeout for icmp_seq 1

Request timeout for icmp_seq 2

Request timeout for icmp_seq 3

Request timeout for icmp_seq 4

Request timeout for icmp_seq 5

^C

--- 210.55.111.1 ping statistics ---

7 packets transmitted, 0 packets received, 100.0% packet loss

[Peter Lambrechtsen]

I've had an email back from an ex-colleague saying it should be sorted as of a few mins ago.

Still having generic weirdness now?

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsub...@googlegroups.com.

[Simon Wright]

Arh, let me double check.

We seem to be working fine now. 

[Graeme Lee]

I changed my DNS forwarders from Spark to Google to get around the problem. Seemed to only affect Google Chrome and not Internet Explorer…

 

Were you given a technical reason for the outage?

 

Regards,

 

Graeme Lee

ICT Director

KingsWay School

100 Jelas Road, Orewa

Auckland 0932, New Zealand.

P: +00  64 09 427 0991 | F: +00 64 09 427 0919 |W: www.kingsway.school.nz

If you are not the intended recipient you must not disclose or use the information in this email or attachment.

If you have received this email in error, please notify me immediately and delete the email and any copies.

 

From: pe...@crypt.co.nz <pe...@crypt.co.nz> On Behalf Of Peter Lambrechtsen
Sent: Tuesday, 22 May 2018 2:29 p.m.
To: techies-f...@googlegroups.com
Subject: Re: [techies-for-schools] Problems accessing 8.8.8.8 (and 8.8.4.4) from within the Spark network (and N4L)

 

I've had an email back from an ex-colleague saying it should be sorted as of a few mins ago.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.

[Andrew Godfrey]

All seems good at the moment here. Including the N4L website !

Andrew Godfrey  |  Network Manager

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Clayton Hubbard]

Hi All. We waiting for official clearance however looks like incident is resolved

Clayton

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Clayton Hubbard
Solution Architect
0220430155

[Pete Mundy]

Hi Clayton

There used to be a live network status & issues page on the N4L website, but I can't for the life of me find it now.

Has it been moved or removed? Can you point us to a direct link?

Cheers,
Pete

[Pete Mundy]

Opps, ignore that! I found it right in front of me. It's called "alerts" :)

https://www.n4l.co.nz/managed-network-5/managed-network-status/

Pete

Begin forwarded message:

From: Pete Mundy <pe...@fiberphone.co.nz>

Subject: Re: [techies-for-schools] Problems accessing 8.8.8.8 (and 8.8.4.4) from within the Spark network (and N4L)

[Peter Lambrechtsen]

I wasn't given specifics unfortunately as I just forwarded this email trail onto my ex-colleagues and he said it was under investigation from the NOC and then emailed me again once it had been resolved.

I doubt I will be able to find out the root cause as I am no longer with Spark and thus outside the inner circle these days.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsub...@googlegroups.com.

[Andrew Godfrey]

@Pete Mundy:

However, in this case I couldn't even get to that server to view the alert page.

Andrew Godfrey  |  Network Manager

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Pete Mundy]

This sort of situation is certainly where an alternative form of access on a different network comes in very handy :)

(eg a cellphone with wifi turned off)

[Andrew Godfrey]

Unless you were using a spark SIM or trying to access anything that you had to go through the spark network to get to.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.

[Clayton Hubbard]

Hi Pete,

It is here. https://www.n4l.co.nz/managed-network-5/managed-network-status/

All confirmed as resolved now.

Clayton Hubbard

Solution Architect

The Network for Learning Ltd

M +64 22 043 0155  DDI +64 9 972 2906  P 0800 LEARNING

A Suite 306, Geyser Building, 100 Parnell Road, Parnell, Auckland 1052

A PO Box 37118, Parnell, Auckland 1151  n4l.co.nz

[Pete Mundy]

Oh sorry, I obviously wasn't clear. I meant that the Spark SIM could be used for accessing the N4L network-status page on their website. No more than that.

From what I saw, ns1.xtra and ns2.xtra remained accessible from within the Spark retail network and so a 3G/4G SIM should have still worked fine for accessing the network status page.

[Clayton Hubbard]

Just another note we do push the alerts via Twitter and Facebook as well.

Question for the Group: Would it be beneficial to have a dedicated outage channel for these type of events or scheduled outages etc? 

Clayton Hubbard

Solution Architect

The Network for Learning Ltd

M +64 22 043 0155  DDI +64 9 972 2906  P 0800 LEARNING

A Suite 306, Geyser Building, 100 Parnell Road, Parnell, Auckland 1052

A PO Box 37118, Parnell, Auckland 1151  n4l.co.nz

[Gerard MacManus]

yes, it would be beneficial.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsub...@googlegroups.com.

[Andrew Godfrey]

A dedicated channel for managed network alerts would be better than having to follow all the alerts in @NL4NZ

Andrew Godfrey  |  Network Manager

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-schools+unsub...@googlegroups.com.