Last weeks Issue with Google Chrome not being able to access Google sites

[ve...@mahurangi.school.nz]

Hi all

This is the post I tried to send yesterday. Posting it more for getting any feedback whether you have seen the issue mentioned below. Yesterday we had no student's device issues which was wonderful so the issue appears to be resolved.

Last week we had major issues with Google Chrome browser not being able to access Google sites. Chrome could access any sites except google sites e.g. Classroom and search. ….weird!

Firefox and Safari could access Google sites

We managed to solve the majority of problems by getting standers and teachers to update Chrome to the latest version so by Friday morning most were able to access Google sites

However last period Friday, we had a number of Chromebooks and Windows laptops that were up to date but unable to access google search and classroom

Chrome could search using Bing and we tried to download Firefox on these devices

Query

Could this be a filter issue as we think it is not an issue at home for these devices . Could it be an device age issue

Any thoughts would be appreciated

Ngā mihi nui

Vern

Vern Dempster | Mahurangi College - Manager Information Services and L3Web Teacher | Ph. +64 9 425 8039 ext 721 | 021 769 314

[d.keen...@gc.ac.nz]

Yes, we ran into this issue hard during last week, and early Monday; I'm still dealing with periodic fallout this week, but yesterday was much better.

N4L finally assigned an engineer around 5 pm yesterday, but I've yet to talk to them.

I was told that Spark made some changes around 00:53 or 00:58 on Monday last week. This seems suspiciously like it was probably the cause.

Another area I investigated, and it is a likely candidate due to the issue affecting Google specifically, is this: https://www.bleepingcomputer.com/news/security/google-chromes-new-post-quantum-cryptography-may-break-tls-connections/

My thought is the upstream device, Spark or N4L's, was unable to cope with the increased processing requirements from this change; this also implies the Google SSL traffic is being decrypted...

I also set up a set of DNS proxies for additional caching beyond normal to see if there was a DNS rate-limiting issue further upstream, and when I did that, things settled down for the rest of yesterday.

P.S:  Thanks for posting, I was thinking after a while it was something we were doing wrong.

Regards,

David Keenleyside, BSc CS & IS, CTech

ITP Associate

EFF Member

ICT Technician

Glenfield College

PO Box 40176 (Kaipatiki Rd)

Glenfield, Auckland City 0629

Ph:       +64 9 444 9066 ext 677

DDI: +64 9 441 9779

Email:    d.keen...@gc.ac.nz

https://itp.nz/CTech/NZ160799

https://www.linkedin.com/in/david-keenleyside-626871/

The Three O’s of Backup: Online, Offline, Off-site.

The Three RA’s of Cloud: Run Anywhere, Run Anytime, Run Agnostic.

[Alistair Baird]

We've had no issues last week, hundreds of chromebooks here, as well as windows and macbooks.

--
You received this message because you are subscribed to the Google Groups "Techies for schools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/techies-for-schools/d92f3130-368b-4f7a-b5da-3a0ba9a2b924n%40googlegroups.com.

--

Kind regards,
Alistair Baird
IT Manager

P  06 354 4198 stpeterspn.school.nz   @stpeterspn 1 Holdsworth Avenue, Milson Palmerston North, 4414

[d.keen...@gc.ac.nz]

For clarification, we have our internal traffic divided amongst several gateways, for monitoring/logging purposes.

Regarding the student gateways, we observed our labs were unaffected on the local device gateway; traffic levels were similar to, and sometimes higher than, the BYOD gateway.

However, the BYOD gateway was impacted. We could move a device from the local device gateway to the BYOD gateway and replicate the exact same behaviour; all sites, except Google ones, worked.

Both gateways are Virtual machines on the same host going to N4L. At the time, thinking it might be a faulty VM, I replaced the BYOD one entirely [pure routing only] with a freshly minted VM, OS, and dedicated host; we encountered the same issue, showing something spooky upstream was happening and likely some form of overloading.

Monday was a test with the freshly rolled DNS proxies to see if DNS queries were being rate-limited somehow.  Now, given things might have stabilised anyway, I cannot be sure if it actually did anything; but today, Tuesday, is still young.

Regards,

David Keenleyside, BSc CS & IS, CTech

ITP Associate

EFF Member

ICT Technician

Glenfield College

PO Box 40176 (Kaipatiki Rd)

Glenfield, Auckland City 0629

Ph:       +64 9 444 9066 ext 677

DDI: +64 9 441 9779

Email:    d.keen...@gc.ac.nz

https://itp.nz/CTech/NZ160799

https://www.linkedin.com/in/david-keenleyside-626871/

The Three O’s of Backup: Online, Offline, Off-site.

The Three RA’s of Cloud: Run Anywhere, Run Anytime, Run Agnostic.

[Vern Dempster]

Hi David and others

I have just talked to our network support isometrics and have found out that this issue has been seen in lots of schools they look after and as you have said appears to be focused on overloaded DNS servers on the BYOD network, ie it breaks trying to get DNS info. ie its immediate and not a searching vaguely and then says not working.

Which is a pity as the Chrome upgrade seemed to fix it until it didnt. 

e.g. we had a about 10 students come up period 3 today Tuesday with their devices, unable to access google in English and Social science block and got to our office and then turned around as all of them seemed to be able to access google again. By swapping access points it seemed to fix things but as Conrad says this could just be luck or N4L/Spark improved something around 11:30 today or the wind was coming in a better direction.

He also told me that google.co.nz appeared to work while google.com didnt

And Safari, Firefox, and Edge seemed to keep working- didnt see issues. Only windows and Chromebooks - our significant MacBook brigade continued to worked on  well :-)

So I hope we can have some confirmation from N4L that it is all sorted

Thanks for your feedback

Vern Dempster | Mahurangi College - Manager Information Services and L3Web Teacher | Ph. +64 9 425 8039 ext 721 | 021 769 314

--
You received this message because you are subscribed to a topic in the Google Groups "Techies for schools" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/techies-for-schools/h2JmhbJkemI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to techies-for-sch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/techies-for-schools/85c5d189-dcc9-494b-b07d-c5d2ac4f9c53n%40googlegroups.com.

[Stephen Moran]

This isn’t a great option especially for colleges, but I can confirm disabling N4L’s Proxy/VPN filtering resolves the issue. We did so for an afternoon and had a peaceful time. Hopefully that test gives N4L more data to work with. 

--

You received this message because you are subscribed to the Google Groups "Techies for schools" group.

To unsubscribe from this group and stop receiving emails from it, send an email to techies-for-sch...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/techies-for-schools/73207374-CEAC-4794-A5C9-61D6F4086507%40mahurangi.school.nz.

[d.keen...@gc.ac.nz]

Unfortunately, this problem is hitting us hard again at the moment. I suspect I'll be running some experiments to see if there is a semi-sensible workaround.

It makes me wonder what they have done to make this happen, given we were operating fine before this.

Regards,

David Keenleyside, BSc CS & IS, CTech

ITP Associate

EFF Member

ICT Technician

Glenfield College

PO Box 40176 (Kaipatiki Rd)

Glenfield, Auckland City 0629

Ph:       +64 9 444 9066 ext 677

DDI: +64 9 441 9779

Email:    d.keen...@gc.ac.nz

https://itp.nz/CTech/NZ160799

https://www.linkedin.com/in/david-keenleyside-626871/

The Three O’s of Backup: Online, Offline, Off-site.

The Three RA’s of Cloud: Run Anywhere, Run Anytime, Run Agnostic.

[Vern Dempster]

Hi David

yes I have not had any contact from N4L about this issue I emailed them yesterday

 I will email them again I think

Ngā mihi nui

Vern

Vern Dempster | Mahurangi College - Manager Information Services and L3Web Teacher | Ph. +64 9 425 8039 ext 721 | 021 769 314

To view this discussion on the web visit https://groups.google.com/d/msgid/techies-for-schools/5d8a201b-302d-4926-9cfe-516f059b6bb7n%40googlegroups.com.

[Alistair Baird]

Just wondering what DNS your students are using, and whether Chrome tries 8.8.8.8, 1.1.1.1 or 8.8.4.4 first. We block all these DNS entries on our student firewall, forcing them to use our internal DNS server, and have not had any issues, although our internal DNS forwards to ns1.xtra.co.nz, ns2.xtra.co.nz then 8.8.8.8

To view this discussion on the web visit https://groups.google.com/d/msgid/techies-for-schools/E2B9438C-EA02-445F-B8F2-3F0659FCBE66%40mahurangi.school.nz.

[d.keen...@gc.ac.nz]

Over here, we use a fair mix for the internal DNS servers:

122.56.237.1
210.55.111.1
1.1.1.3
1.0.0.3
8.8.8.8

This is so there is at least some filtering should the Xtra DNS servers fall over, and finally, Google; it's a bit dangerous, though, as the filtering is minimal on Google's.

It's also why I'm rather puzzled over the situation, as our lab machines, using the same lookups, are unaffected; they do, however, use a different gateway.

I'll run some more testing using dig when we next get hit, as the timing is never right ;)

Regards,

David Keenleyside, BSc CS & IS, CTech

ITP Associate

EFF Member

ICT Technician

Glenfield College

PO Box 40176 (Kaipatiki Rd)

Glenfield, Auckland City 0629

Ph:       +64 9 444 9066 ext 677

DDI: +64 9 441 9779

Email:    d.keen...@gc.ac.nz

https://itp.nz/CTech/NZ160799

https://www.linkedin.com/in/david-keenleyside-626871/

The Three O’s of Backup: Online, Offline, Off-site.

The Three RA’s of Cloud: Run Anywhere, Run Anytime, Run Agnostic.

[Scott Simpson]

Hi all, we had this issue last week at one of our larger high schools.

 

TLDR & YMMV; N4L FortiGate was blocking TLSv1 packets as they were being categorized as “X-VPN” traffic under the Proxy Avoidance category.

 

The issue was related to v124 of Chrome where X25519Kyber768 was being enabled by default (see here: https://chromestatus.com/feature/5257822742249472). When running packet captures, we could see that TLS version negotiating wasn’t getting above v1 when attempting to connect to sites that were serving certs with X25519Kyber768 enabled (i.e. https://mail.google.com). Strangely, the test site https://pq.cloudflareresearch.com/ was working fine and was telling us that we were “post-quantum secure”.

 

Our best guess, based on reading: https://tldr.fail/, was that the TLSv1.3 connection was being broken somewhere on the N4L side. This possibly led to the TLS negotiating packets dropping down to v1 which subsequently got caught up incorrectly as VPN traffic.. and blocked.

 

As soon as the N4L put their firewall into transparent mode, the issue stopped immediately. Current filtering at this school is done on their own FortiGate.

 

This school’s internal network is setup like the below:

 

Internal School Network  à School FortiGate à N4L FortiGate à Internet

 

I hope this points you in the right direction or gives you somewhere else to look. If anyone would like a hand troubleshooting this.. more than happy to share some more details on what we were seeing with Wireshark captures and N4L FortiGate logs.

 

Thanks,

Scott

 

	Scott Simpson Director
	0210565463   sc...@contrast.nz   contrast.nz   PO Box 33434, Barrington, Christchurch 8244

To view this discussion on the web visit https://groups.google.com/d/msgid/techies-for-schools/0615e75d-4a7c-45a0-8361-1ad97a035d2en%40googlegroups.com.

[Pete Mundy]

Just FWIW as general background knowledge - I understand that N4L actively re-write outbound UDP to 122.56.237.1 and 210.55.111.1 from all (downstream) school networks and redirect to their own caching/filtering proxies. So your responses from those IPs at school may differ to the responses from those IPs at home (eg on a Spark UFB link).

To view this discussion on the web visit https://groups.google.com/d/msgid/techies-for-schools/0615e75d-4a7c-45a0-8361-1ad97a035d2en%40googlegroups.com.

[Hayden Brown]

Just confirming from the N4L side that we're reviewing this issue and be in touch shortly with affected schools.

[d.keen...@gc.ac.nz]

Just be aware this very same issue could rear its ugly head again: https://www.bleepingcomputer.com/news/security/chrome-switching-to-nist-approved-ml-kem-quantum-encryption/

"The update is to be implemented in Chrome 131 (current version is 128), scheduled for release on November 6, 2024."

Regards,

David Keenleyside, BSc CS & IS, CTech

ITP Associate

EFF Member

ICT Technician

Glenfield College

PO Box 40176 (Kaipatiki Rd)

Glenfield, Auckland City 0629

Ph:       +64 9 444 9066 ext 677

DDI: +64 9 441 9779

Email:    d.keen...@gc.ac.nz

https://itp.nz/CTech/NZ160799

https://www.linkedin.com/in/david-keenleyside-626871/

The Three O’s of Backup: Online, Offline, Off-site.

The Three RA’s of Cloud: Run Anywhere, Run Anytime, Run Agnostic.

[Jono Green]

Good to see quantum considerations beginning to be implemented. On a similar note, worth schools thinking about how they address passwords in a quantum world.