Testing web pages with POSH

[Kurt Buff]

I've got a snippet that I'm putting in a while loop:

$example = [Net.WebRequest]::Create("https://example.com")
try { $example.GetResponse() } catch {}
$cert = $example.ServicePoint.Certificate
$bytes = $cert.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytes -encoding byte -path
"C:\temp\tshark-capture\$(Get-Date -Format
"yyyy-MM-dd-HHmm")-example.cer"

I'm using PowerShell 5.1. I've noticed that when I browse the site
using Chrome, it's using TLS 1.3, but this snippet uses TLS 1.2.

Is there a way to get PowerShell to use TLS 1.3?

Kurt

[Kurt Buff]

Forgot to ask what is perhaps the most important part:

Is there a way to detect when the certificate found doesn't match the SNI sent by the client?

Kurt

[Michael B. Smith]

To get PS to use TLS 1.3, you have to upgrade to at least .NET 4.8, verify that it’s enabled in the registry (it isn’t by default), and set it to be used.

 

Option:

 

[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls13

 

Only:

 

               [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls13

 

Also, I honestly don’t know how much of .NET 4.8 was updated to work with TLS 1.3. Most of that work went into .NET (Core) 6.3 and above.

 

I have been unable to get SNI level of detail using PS. I use nmap for that purpose.

 

You might be able to get it by creating an SSL stream object.

 

Thanks.

 

Regards,

Michael B. Smith

Managing Consultant

Smith Consulting, LLC

--
You received this message because you are subscribed to the Google Groups "ntpowershell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntpowershell...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/CADy1Ce70ZAF5mMuYtysOu921LG6CkEq6nP8Z_K9RRj2Ad1UfuQ%40mail.gmail.com.

[Micheal Espinola]

I know we are talking about PowerShell here, but is there a specific need to use PoSH to test a page instead of a specialized tool like cURL? Specifying minimum and maximum TLS versions are command flags. 

--
You received this message because you are subscribed to the Google Groups "ntpowershell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntpowershell...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/CADy1Ce4SUAtbcvsO1kG-g88Lxskipgbec8tJbxpRYhVkrahZxw%40mail.gmail.com.

--

Espi

[Kurt Buff]

More detail on the problem and what I've tried.

BTW - I don't think I'll need TLS 1.3. I've opened a ticket with the firewall vendor to see if the problem is with their https inspection - it seems to be a bit erratic.

We have 4 sites on the same host, at a hosting provider. We have staff who randomly get the browser errors when visiting any of those 4 sites. Our CDN is Fastly. They state that if the SNI isn't received in the TLS request, they provide an alternate certificate from Let's Encrypt. Our firewalls are set to bypass https inspection for the 4 sites, and probably 80+% of the time it does just that

I have tried to use nmap to harvest the certs, like so, but had two problems - 1) the loop only executes a small number of times (less than 20 rather than 101) and 2) it never saw the bad certificate even after multiple runs. For the latter, perhaps I'm just impatient, but the firewall logs show 21% rate of https inspection in the past 24 hours, when it should be bypassing inspection entirely:
----------
$count = 0
while ($count -le 101) {
$site1 = nmap -v -p 443 --script ssl-cert site1.com
set-content -value $site1 -encoding ascii -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site1.txt"
$site2 = nmap -v -p 443 --script ssl-cert site2.com
set-content -value $site2 -encoding ascii -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site2.txt"
$site3 = nmap -v -p 443 --script ssl-cert site3.com
set-content -value $site3 -encoding ascii -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site3.txt"
$siter = nmap -v -p 443 --script ssl-cert site4.com
set-content -value $siter -encoding ascii -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site4.txt"
clear-variable site1
clear-variable site2
clear-variable site3
clear-variable site4
start-sleep -seconds 5
ipconfig /flushdns
$count++
}
----------

I've been successful using this, but it's requires manual inspection of the browser, and doesn't capture the certificate, and the correlation with the firewall inspection logs is a bit more tedious.
----------
ipconfig /flushdns
stop-process -name chrome
Remove-Item -path "$env:LOCALAPPDATA\Google\Chrome\User Data\Default\Cache\*" -Recurse -Force -EA SilentlyContinue -Verbose
Remove-Item -path "$env:LOCALAPPDATA\Google\Chrome\User Data\Default\Cache2\entries\*" -Recurse -Force -EA SilentlyContinue -Verbose
Remove-Item -path "$env:LOCALAPPDATA\Google\Chrome\User Data\Default\Cookies" -Recurse -Force -EA SilentlyContinue -Verbose
Remove-Item -path "$env:LOCALAPPDATA\Google\Chrome\User Data\Default\Media Cache" -Recurse -Force -EA SilentlyContinue -Verbose
Remove-Item -path "$env:LOCALAPPDATA\Google\Chrome\User Data\Default\Cookies-Journal" -Recurse -Force -EA SilentlyContinue -Verbose
Start-Process -FilePath "C:\Program Files\Google\Chrome\Application\chrome.exe" -ArgumentList '"https://site1.com"'
Start-Process -FilePath "C:\Program Files\Google\Chrome\Application\chrome.exe" -ArgumentList '"https://site2/com"'
Start-Process -FilePath "C:\Program Files\Google\Chrome\Application\chrome.exe" -ArgumentList '"https://site3.com"'
Start-Process -FilePath "C:\Program Files\Google\Chrome\Application\chrome.exe" -ArgumentList '"https://site4.com"'
----------

I've also tried this, but 1) it's so slow that I normally have to kill it because it seems to hang after about 3 times through the loop, and 2) I haven't seen it produce the error yet (again, I might be impateitn) and 3) I have to manually examine the certs it writes to disk 0 they aren't text.
----------
$count = 0
while ($count -le 101) {

$site1 = [Net.WebRequest]::Create("https://site1.com")
try { $site1.GetResponse() } catch {}
$certsite1 = $site1.ServicePoint.Certificate
$bytessite1 = $certsite1.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytessite1 -encoding byte -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site1.cer"

$site2 = [Net.WebRequest]::Create("https://site2.com")
try { $site2.GetResponse() } catch {}
$certsite2 = $site2.ServicePoint.Certificate
$bytessite2 = $certsite2.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytessite2 -encoding byte -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site2.cer"

$site3 = [Net.WebRequest]::Create("https://site3.com")
try { $site3.GetResponse() } catch {}
$certsite3 = $site3.ServicePoint.Certificate
$bytessite3 = $certsite3.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytessite3 -encoding byte -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site3.cer"

$site4 = [Net.WebRequest]::Create("https://site4.com")
try { $site4.GetResponse() } catch {}
$certsite4 = $site4.ServicePoint.Certificate
$bytessite4 = $certsite4.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytessite4 -encoding byte -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site4.cer"

clear-variable site1
clear-variable certsite1
clear-variable bytessite1
clear-variable site2
clear-variable certsite2
clear-variable bytessite2
clear-variable site3
clear-variable certsite3
clear-variable bytessite3
clear-variable site4
clear-variable certsite4
clear-variable bytessite4

ipconfig /flushdns

$count++
}
----------

Kurt

    To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/CAAfzEuzwvqHHZqn%3D_uX1Ybwh4iuy2mvyh8BY3t_yFDO7dUz37g%40mail.gmail.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/CAAfzEuzwvqHHZqn%3D_uX1Ybwh4iuy2mvyh8BY3t_yFDO7dUz37g%40mail.gmail.com.

[Michael B. Smith]

What exactly are you wanting to look at in the certificate? There is no reason to save it to disk before you examine it. But you might have to map it, depending on desired attributes.

 

These are hanging because you aren’t closing them and I think the pool is only 20 sites:

 

$site1 = [Net.WebRequest]::Create("https://site1.com")
try { $site1.GetResponse() } catch {}
$certsite1 = $site1.ServicePoint.Certificate
$bytessite1 = $certsite1.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytessite1 -encoding byte -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site1.cer"

S/b:

 

$site1 = [Net.WebRequest]::Create("https://site1.com")

try { $response = $site1.GetResponse() } catch {}

if( $response )

{

$certsite1 = $site1.ServicePoint.Certificate
$bytessite1 = $certsite1.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytessite1 -encoding byte -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site1.cer"

$response.Close()

}

Thanks.

 

Regards,

Michael B. Smith

Managing Consultant

Smith Consulting, LLC

 

From: ntpowe...@googlegroups.com <ntpowe...@googlegroups.com> On Behalf Of Kurt Buff

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/CADy1Ce5wpEux0Yb30BBmNKQynU5GCBuR1vsEX8%2BaYbeSqCRd3Q%40mail.gmail.com.

[Kurt Buff]

Each of our sites has its own Sectigo wildcard cert. The default/backup certs from Fastly (the CDN) have a subject name that doesn't match the site name (they have m.sni-m311.global.fastly.net), and Chrome (and FF, and I'm sure Edge also) won't allow the user to proceed.

Kurt

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/6992494d314a413182aa33f01cd45313%40smithcons.com.

[Michael B. Smith]

$site1.ServicePoint.certificate.Subject will give you the subject name of the cert.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/CADy1Ce5BFvoA%3DHBCq2uPMkobGF4Kd5crvOaVCoyEcPGZ4zKO%2BA%40mail.gmail.com.

[Kurt Buff]

Thanks.

That should help.

Kurt

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/44fbe72f7c244bdfb2d7d5ba88d94f51%40smithcons.com.

[Kurt Buff]

I am trying to optimize a bit, and only write the cert to disk if the subject isn't correct. Here's what I came up with, but it doesn't work. I also tried substituting Issuer, and that doesn't work either.  I'm guessing there's an encoding transform that I'm missing for both the host display and the if statement, but I'm not sure of which terms to search on to figure this out.

$site1 = [Net.WebRequest]::Create("https://site1.com")
try { $response = $site1.GetResponse() } catch {}
if( $response )
{
$certsite1 = $site1.ServicePoint.Certificate

write-host $certsite1.ServicePoint.certificate.Subject
if ( $certsite1.ServicePoint.certificate.Subject -notlike "*site1*" ) {

   $bytessite1 = $certsite1.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
   set-content -value $bytessite1 -encoding byte -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site1.cer"
   }

Else { Write-Host "Correct cert for site1.com" }
$response.Close()
}

On Mon, Mar 20, 2023 at 3:24 PM Michael B. Smith <mic...@smithcons.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/6992494d314a413182aa33f01cd45313%40smithcons.com.

[Michael B. Smith]

Overspecification.

 

All you need is $certsite1.Subject

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/CADy1Ce4tHa75nFWs4kGHA6be1rtKrs3KSOvGz96G%2BXbtXTmcaA%40mail.gmail.com.

[Kurt Buff]

Working!

I wish I could wrap my head around this better.

Thanks ever so much.

Kurt

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/6e4bad6e454b410f99ebb7b3f62c89ea%40smithcons.com.

[Kurt Buff]

Well nuts. I think I ran into a snag.

It looks like the site has set session Max-Age to 36000 seconds - 10 hours.

PHPSESSID=465eb7d6427f82c22d04e77f11f97b83; expires=Thu, 23-Mar-2023 05:32:03 GMT; Max-Age=36000; path=/; domain=site1.com; secure; HttpOnly; SameSite=Lax

Currently as configured the script is using TLS 1.2, as I noted earlier, but the certificate isn't coming through. Instead, I'm seeing this in Wireshark:

Is there a way to force a new session?

At the end of the script I do clear variables, like so:

$site1 = [Net.WebRequest]::Create("https://site1.us")

try { $response = $site1.GetResponse() } catch {}
if( $response )
{
$certsite1= $site1.ServicePoint.Certificate

write-host $certsite1.Subject
if ( $certsite1.Subject -notlike "*fly*") {
   $bytessite1= $certsite1.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
   set-content -value $bytessite1-encoding byte -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-site1.cer"
   $badcerts++
   }
Else { Write-Host "Correct cert for site1.us" }
$response.Close()
}

clear-variable certsite1
clear-variable site1

Kurt

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/6e4bad6e454b410f99ebb7b3f62c89ea%40smithcons.com.

[Kurt Buff]

Ok - with a little testing, I see that if I close out PowerShell and re-open, then run the script again, the first runthrough does see the certificate, and subsequent passes just re-negotiate.

That's not a good solution, but it's good to know.

I'm wondering now if abusing the user agent string by incrementing a variable and stuffing that in the request would work, or if there's a better way.

I get it that most web dev wants continuation, but that's not what's needed in this situation.

Kurt

[Michael B. Smith]

Now you’ve exceeded me. 😊

 

But I suspect it’s due to caching.

 

But an idea. “Clear-Variable x” is equivalent to “$x = $null”. So it doesn’t do anything in .NET, just in the PS host. You might try this instead:

 

               Clear-variable x

               Remove-variable x

               [GC]::WaitForPendingFinalizers()

               [GC]:Collect()

 

Thanks.

 

Regards,

Michael B. Smith

Managing Consultant

Smith Consulting, LLC

 

From: ntpowe...@googlegroups.com <ntpowe...@googlegroups.com> On Behalf Of Kurt Buff
Sent: Wednesday, March 22, 2023 3:57 PM
To: ntpowe...@googlegroups.com
Subject: Re: [ntpowershell] Testing web pages with POSH

 

Well nuts. I think I ran into a snag.

 

It looks like the site has set session Max-Age to 36000 seconds - 10 hours.

PHPSESSID=465eb7d6427f82c22d04e77f11f97b83; expires=Thu, 23-Mar-2023 05:32:03 GMT; Max-Age=36000; path=/; domain=site1.com; secure; HttpOnly; SameSite=Lax

 

Currently as configured the script is using TLS 1.2, as I noted earlier, but the certificate isn't coming through. Instead, I'm seeing this in Wireshark:

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/CADy1Ce4_HwviXOiZH8-Z9DPAJG83vL0hnDPJn%2BRv3fNF9kQp7w%40mail.gmail.com.

[Michael B. Smith]

Or changing the cache timeouts in the net.webrequest.

 

Thanks.

 

Regards,

Michael B. Smith

Managing Consultant

Smith Consulting, LLC

 

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/e87b39e6c2de4325b6189dc7396db778%40smithcons.com.

[Kurt Buff]

As you suspected, using CV and RV and doing garbage collection didn't work for this purpose.

I'm scratching my head over this, and will continue tinkering.

Kurt

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/e87b39e6c2de4325b6189dc7396db778%40smithcons.com.

[Aakash Shah]

I haven’t tested this, but since you indicated that opening a new PS window worked, what if you spawned a new PowerShell window from the script to perform each connection test? I’ve spawned additional windows in the past to help run multiple robocopy jobs that I can monitor and I wonder if a similar approach may help here. I recognize it’s not as elegant though.

 

-Aakash Shah

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/CADy1Ce4BpgcV6wdjo8LbJJD54dfzAY3NNXUOUnSumXC5YrGSaA%40mail.gmail.com.

[Kurt Buff]

You have captured my thoughts.

Not elegant, but it might work.

I'll update this post once I have some results.

Kurt

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/SJ0P221MB073708515BF62D85DD4CC709F2879%40SJ0P221MB0737.NAMP221.PROD.OUTLOOK.COM.

[Kurt Buff]

If it's stupid, but it works, it isn't stupid. (maybe).

This works:

Open powershell, run this:

#count = 0
While ( $count -ne 300 ) {
start-process powershell.exe -ArgumentList C:\temp\tshark-capture\Get-WebCerts.ps1
$count++
start-sleep -seconds 15
}

----------Get-WebCerts.ps1----------

$sites = "https://site1.com", "https://site2.com", "https://site3.com", "https://site4.com"

Foreach ( $site in $sites ) {
   $Request = [Net.WebRequest]::Create("$site")
   try { $response = $Request.GetResponse() } catch {}
   if( $response ) {
      $cert = $Request.ServicePoint.Certificate
      write-host $cert.Issuer
      if ( $cert.Issuer -notlike "*Sectigo*") {

         $bytes = $cert.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)

         set-content -value $bytes -encoding byte -path "C:\temp\tshark-capture\$(Get-Date -Format "yyyy-MM-dd-HHmm")-badcert.cer"
         }
      Else { Write-Host "Correct cert for $site" }
      $response.Close()
   write-host " "
   }
}

----------Get-WebCerts.ps1----------

On Thu, Mar 23, 2023 at 5:25 PM Aakash Shah <aakas...@uci.edu> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/SJ0P221MB073708515BF62D85DD4CC709F2879%40SJ0P221MB0737.NAMP221.PROD.OUTLOOK.COM.