There has got to be a smarter way to do this

[Kurt Buff]

To accommodate our SEIM, I've configured our DCs to do DNS log
rollover, and capped (deleted via a scheduled task) log retention to 2
days.

I've used the default log size of 500k, and 3 of the 4 DCs in our HQ
currently generate 20-24 separate log file in a day, though it's
diminishing as we roll out the new DCs..

I'm trying to document several things using those logs, such as
- decrease in DNS queries to the HQ DCs as we roll out the remote DCs
- misconfigurations the DNS server settings of equipment in the remote sites
- what kind of traffic the firewalls are seeing
- etc.

Below my signature is the sanitized script I'm using to parse out the logs.

My problem is that parsing a single log is taking upwards of 8 hours,
so it's taking many time longer to parse the logs than it is to
generate them. I'll never be able to keep up.

Is there a way to speed this up?

Thanks,
Kurt

$site01 = " 10.1."
$site02 = " 10.1."
$site03 = " 10.3."
$site04 = " 10.4."
$site04_vpn = " 172.16."
$site05 = " 10.5."
$site06 = " 10.6."
$site07 = " 10.7."
$site08 = " 10.8."
$site09 = " 10.9."
$site10 = " 10.10."
$site11 = " 10.11."
$site12 = " 10.12."
$site13 = " 10.13."
$site14 = " 10.14."
$site15 = " 10.15."
$site16 = " 10.16."
$fw_site01 = "172.20.20.34"
$fw_site02 = "172.20.20.42"
$fw_site03 = "172.20.20.50"
$fw_site05 = "172.20.20.10"
$fw_site06 = "172.20.20.2 "
$fw_site07 = "172.20.20.46"
$fw_site08 = "172.20.20.30"
$fw_site09 = "172.20.20.18"
$fw_site10 = "172.20.20.38"
$fw_site11 = "172.20.20.22"
$fw_site12 = "172.20.20.14"
$fw_site13 = "172.20.20.54"
$fw_site14 = "172.20.20.26"
$fw_site15 = "172.20.20.6 "
$fw_site16 = "172.20.20.58"
$site04_gw1 = "172.20.20.67"
$site04_gw2 = "172.20.20.68"


$dirs = "c:\temp\site04-dnslogs\dc0"

foreach ( $dir in $dirs ) {
cd -path $dir
New-Item -Path . -Name "site01-out.txt" -ItemType "file"
New-Item -Path . -Name "site02-out.txt" -ItemType "file"
New-Item -Path . -Name "site03-out.txt" -ItemType "file"
New-Item -Path . -Name "site04-out.txt" -ItemType "file"
New-Item -Path . -Name "site04-vpn-out.txt" -ItemType "file"
New-Item -Path . -Name "site05-out.txt" -ItemType "file"
New-Item -Path . -Name "site06-out.txt" -ItemType "file"
New-Item -Path . -Name "site07-out.txt" -ItemType "file"
New-Item -Path . -Name "site08-out.txt" -ItemType "file"
New-Item -Path . -Name "site09-out.txt" -ItemType "file"
New-Item -Path . -Name "site10-out.txt" -ItemType "file"
New-Item -Path . -Name "site11-out.txt" -ItemType "file"
New-Item -Path . -Name "site12-out.txt" -ItemType "file"
New-Item -Path . -Name "site13-out.txt" -ItemType "file"
New-Item -Path . -Name "site14-out.txt" -ItemType "file"
New-Item -Path . -Name "site15-out.txt" -ItemType "file"
New-Item -Path . -Name "site16-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site01-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site02-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site03-out.txt" -ItemType "file"
New-Item -Path . -Name "site04-gw1-out.txt" -ItemType "file"
New-Item -Path . -Name "site04-gw2-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site05-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site06-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site07-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site08-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site09-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site10-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site11-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site12-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site13-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site14-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site15-out.txt" -ItemType "file"
New-Item -Path . -Name "fw-site16-out.txt" -ItemType "file"

$dnslogs = gci -file *.log | select -expand name

foreach ( $dnslog in $dnslogs ) {
foreach ( $line in Get-Content $dnslog ) {
if ( $line.Contains($site01) ) {
add-content -value $line -encoding ASCII -path site01-out.txt
}
if ( $line.Contains($site02) ) {
add-content -value $line -encoding ASCII -path site02-out.txt
}
if ( $line.Contains($site03) ) {
add-content -value $line -encoding ASCII -path site03-out.txt
}
if ( $line.Contains($site04) ) {
add-content -value $line -encoding ASCII -path site04-out.txt
}
if ( $line.Contains($site05) ) {
add-content -value $line -encoding ASCII -path site05-out.txt
}
if ( $line.Contains($site06) ) {
add-content -value $line -encoding ASCII -path site06-out.txt
}
if ( $line.Contains($site07) ) {
add-content -value $line -encoding ASCII -path site07-out.txt
}
if ( $line.Contains($fw_site08) ) {
add-content -value $line -encoding ASCII -path site08-out.txt
}
if ( $line.Contains($fw_site09) ) {
add-content -value $line -encoding ASCII -path site09-out.txt
}
if ( $line.Contains($site10) ) {
add-content -value $line -encoding ASCII -path site10-out.txt
}
if ( $line.Contains($site11) ) {
add-content -value $line -encoding ASCII -path site11-out.txt
}
if ( $line.Contains($site12) ) {
add-content -value $line -encoding ASCII -path site12-out.txt
}
if ( $line.Contains($site13) ) {
add-content -value $line -encoding ASCII -path site13-out.txt
}
if ( $line.Contains($site14) ) {
add-content -value $line -encoding ASCII -path site14-out.txt
}
if ( $line.Contains($site15) ) {
add-content -value $line -encoding ASCII -path site15-out.txt
}
if ( $line.Contains($site16) ) {
add-content -value $line -encoding ASCII -path site16-out.txt
}
if ( $line.Contains($fw_site01) ) {
add-content -value $line -encoding ASCII -path fw-site01-out.txt
}
if ( $line.Contains($fw_site02) ) {
add-content -value $line -encoding ASCII -path fw-site02-out.txt
}
if ( $line.Contains($fw_site03) ) {
add-content -value $line -encoding ASCII -path fw-site03-out.txt
}
if ( $line.Contains($site04_gw1) ) {
add-content -value $line -encoding ASCII -path site04-gw1-out.txt
}
if ( $line.Contains($site04_gw2) ) {
add-content -value $line -encoding ASCII -path site04-gw2-out.txt
}
if ( $line.Contains($fw_site05) ) {
add-content -value $line -encoding ASCII -path fw-site05-out.txt
}
if ( $line.Contains($fw_site06) ) {
add-content -value $line -encoding ASCII -path fw-site06-out.txt
}
if ( $line.Contains($fw_site07) ) {
add-content -value $line -encoding ASCII -path fw-site07-out.txt
}
if ( $line.Contains($fw_site08) ) {
add-content -value $line -encoding ASCII -path fw-site08-out.txt
}
if ( $line.Contains($fw_site09) ) {
add-content -value $line -encoding ASCII -path fw-site09-out.txt
}
if ( $line.Contains($fw_site10) ) {
add-content -value $line -encoding ASCII -path fw-site10-out.txt
}
if ( $line.Contains($fw_site11) ) {
add-content -value $line -encoding ASCII -path fw-site11-out.txt
}
if ( $line.Contains($fw_site12) ) {
add-content -value $line -encoding ASCII -path fw-site12-out.txt
}
if ( $line.Contains($fw_site13) ) {
add-content -value $line -encoding ASCII -path fw-site13-out.txt
}
if ( $line.Contains($fw_site14) ) {
add-content -value $line -encoding ASCII -path fw-site14-out.txt
}
if ( $line.Contains($fw_site15) ) {
add-content -value $line -encoding ASCII -path fw-site15-out.txt
}
if ( $line.Contains($fw_site16) ) {
add-content -value $line -encoding ASCII -path fw-site16-out.txt
}
}
}
}

[Kurt Buff]

I should have included some sample data, so here a small bit:

2/25/2023 11:31:14 PM 0774 PACKET 0000021F02E4C9D0 UDP Rcv
172.20.20.54 e74d Q [0001 D NOERROR] A
(7)avalara(3)com(0)
2/25/2023 11:31:14 PM 0774 PACKET 0000021F02E4C9D0 UDP Snd
172.20.20.54 e74d R Q [8081 DR NOERROR] A
(7)avalara(3)com(0)
2/25/2023 11:31:14 PM 0620 PACKET 0000021F7993A070 UDP Rcv
172.20.20.54 3a91 Q [0001 D NOERROR] A
(3)www(7)example(3)com(0)
2/25/2023 11:31:14 PM 0620 PACKET 0000021F7993A070 UDP Snd
172.20.20.54 3a91 R Q [8081 DR NOERROR] A
(3)www(7)example(3)com(0)
2/25/2023 11:31:14 PM 0620 PACKET 0000021F0A68ED80 UDP Rcv
172.20.20.54 47c3 Q [0001 D NOERROR] A
(3)www(12)vekenbattery(3)com(0)
2/25/2023 11:31:14 PM 0620 PACKET 0000021F0A68ED80 UDP Snd
172.20.20.54 47c3 R Q [8081 DR NOERROR] A
(3)www(12)vekenbattery(3)com(0)
2/25/2023 11:31:14 PM 0620 PACKET 0000021F7DA504B0 UDP Rcv
172.20.20.54 fb4a Q [0001 D NOERROR] A
(3)www(23)1-744473-1592991327-304(2)rt(6)yammer(3)com(0)
2/25/2023 11:31:14 PM 0620 PACKET 0000021F7DA504B0 UDP Snd
172.20.20.54 fb4a R Q [8081 DR NOERROR] A
(3)www(23)1-744473-1592991327-304(2)rt(6)yammer(3)com(0)

Kurt

[Michael B. Smith]

[1] those add-contents are KILLING you. For each line matched, you are eating the overhead of open, seek-to-EOF, write, flush, close. Use a stringbuilder instead and do just one I/O:

[System.Text.StringBuilder] $sb = New-Object System.Text.StringBuilder( 1024 ); ## create SB object - top of function

$sb.AppendLine( $line ); ## append to SB object - each line

Out-File -FilePath <filename> -Append -InputObject $sb.ToString(); ## output the SB - end of function

[2] $line.Contains() [which is string.contains()] is too slow for hundreds of thousands/millions of comparisons. Use a hashtable and ContainsKey()

Eg:

$ht = @{}
$ht[ $site01 ] = 1
$ht[ $site02 ] = 1
...etc...

Then split out the specific field(s) you are interested in, and:

If( $ht.ContainsKey( $field1 ) ) ....

I just took a look, you must only be looking for a specific field:

## 3/2/2023 1:09:21 PM 15A0 PACKET 000000E4A89CB220 UDP Rcv 172.17.0.212 daa6 Q [0001 D NOERROR] A (3)emc(3)com(0)

Easy to find either with regex or IndexOf().

These two changes will give you an order-of-magnitude speed-up (as an estimate).

Thanks.

Regards,
Michael B. Smith
Managing Consultant
Smith Consulting, LLC

--
You received this message because you are subscribed to the Google Groups "ntpowershell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntpowershell...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ntpowershell/CADy1Ce7tr7nh%2BGy%2BszKhT0M-DhiTFK4PJmCxNK6kDLdo_Yz4CA%40mail.gmail.com.