unable to get youtube strict blocking working

[austinfria...@googlemail.com]

7.5 explicit proxy, AD auth.  

Youtube guard v3 api key added.  API key has youtube API data v3 "enabled" (API appears under google youtube API compatible authentication).  Daemon log enabled, shows:

Jan 19 12:29:57 diladele-ub16 wsytgd[8093]: 2021/01/19 12:29:57 Got signal to reload configuration, reloading the application...

Jan 19 12:29:57 diladele-ub16 wsytgd[8093]: 2021/01/19 12:29:57 Reloading application object configuration...

Jan 19 12:29:57 diladele-ub16 wsytgd[8093]: 2021/01/19 12:29:57 Config object is &{APIKey:XXXXXXXXXXXX DaemonPort:18891 Verbose:true SeedFile:/opt/websafety/var/spool/youtube_cache/videos.json CacheExpiration:432000}

Jan 19 12:29:57 diladele-ub16 wsytgd[8093]: 2021/01/19 12:29:57 New configuration was read successfully, proceeding to reload...

Jan 19 12:29:57 diladele-ub16 wsytgd[8093]: 2021/01/19 12:29:57 Application object successfully reloaded configuration.

Jan 19 12:29:57 diladele-ub16 wsytgd[8093]: 2021/01/19 12:29:57 Configuration reload signal handled successfully.

(I removed the API key from above). Webfilter youtube policy for the user has "strict restrictions" and all ticks enabled.  Action is "block videos by default".  Blocked categories - almost all especially "entertainment".

Client navigates to youtube and selects a video.  Video is verified to be in a blocked category by viewing page source for   "category listing.  Webfilter log shows correct "policy", correct "user", URL is along the lines of "https://r4---sn-bvvbax-ac5e.googlevideo.com/videoplayback" for method GET and with initial method of CONNECT for "r4---sn-bvvbax-ac5e.googlevideo.com:443" in category video_sharing

Client is not blocked from viewing any youtube page at all.  API metrics do not show activity (but I dont know if these are updated nightly - I have only started this today)

How can I fault find this?

[austinfria...@googlemail.com]

I should note that HTTPS is fully enabled.  There are some exclusions but only ONE youtube related one:

accounts.youtube.com 

for our chromebook.  Apart from that there are no other youtube exclusions.

[Rafael Akchurin]

The filter looks for video id in the request - may be chromebook does it differently. Are you sure the video id requests to you tube is visible as decrypted in the access log?

Best regards,

Rafael Akchurin

On 19 Jan 2021, at 15:20, 'austinfria...@googlemail.com' via Diladele Web Safety <web-s...@googlegroups.com> wrote:

I should note that HTTPS is fully enabled.  There are some exclusions but only ONE youtube related one:

--
You received this message because you are subscribed to the Google Groups "Diladele Web Safety" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web-safety+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/web-safety/7484e4a1-9f5e-47c8-843e-866faba72138n%40googlegroups.com.

[austinfria...@googlemail.com]

testing from a windows client - im not sure how I can check this.  I dont seem to get a hit for "youtube" in my logs at all.  PLenty of hits for googlevideo.  I cannot see any reference to video id in the access log

I did a refresh on a youtube page, this is the initial log output

1611067669.224     72 10.1.5.130 TCP_MISS/200 529963 GET https://r3---sn-bvvbax-ac5e.googlevideo.com/videoplayback? xxxxxxxxxxxxxxxxxxxx HIER_DIRECT/208.117.250.20 audio/webm "ws-iid=51377" "ws-mac=00:00:00:00:00:00" "ws-duration=72" "ws-timing=0" "ws-mtime=0" "ws-scanflags=63" "ws-categories=2251799813685248" "ws-trusted=0" "ws-level=2" "ws-verdict=0" "ws-policy=pupils_(non_senior_school)" "ws-member=Pupils" "ws-module=2" "ws-msgtype=3" "ws-param1=None" "ws-param2=None" "ws-debug=None" "squid-gt-st=1476"

1611067671.463     12 10.1.5.130 TCP_MISS/204 328 GET https://r3---sn-bvvbax-ac5e.googlevideo.com/generate_204 xxxxxxxxxxxxxxxxxxxx  HIER_DIRECT/208.117.250.20 text/html "ws-iid=51389" "ws-mac=00:00:00:00:00:00" "ws-duration=205" "ws-timing=0" "ws-mtime=0" "ws-scanflags=63" "ws-categories=2251799813685248" "ws-trusted=0" "ws-level=1" "ws-verdict=0" "ws-policy=pupils_(non_senior_school)" "ws-member=Pupils" "ws-module=2" "ws-msgtype=3" "ws-param1=None" "ws-param2=None" "ws-debug=None" "squid-gt-st=551"

1611067671.464     13 10.1.5.130 TCP_MISS/204 328 GET https://r3---sn-bvvbax-ac5e.googlevideo.com/generate_204? xxxxxxxxxxxxxxxxxxxx  HIER_DIRECT/208.117.250.20 text/html "ws-iid=51393" "ws-mac=00:00:00:00:00:00" "ws-duration=1" "ws-timing=0" "ws-mtime=0" "ws-scanflags=63" "ws-categories=2251799813685248" "ws-trusted=0" "ws-level=1" "ws-verdict=0" "ws-policy=pupils_(non_senior_school)" "ws-member=Pupils" "ws-module=2" "ws-msgtype=3" "ws-param1=None" "ws-param2=None" "ws-debug=None" "squid-gt-st=557"

1611067671.710     43 10.1.5.130 TCP_MISS/200 67589 GET https://r3---sn-bvvbax-ac5e.googlevideo.com/videoplayback? xxxxxxxxxxxxxxxxxxxx  HIER_DIRECT/208.117.250.20 audio/webm "ws-iid=51398" "ws-mac=00:00:00:00:00:00" "ws-duration=74" "ws-timing=0" "ws-mtime=0" "ws-scanflags=63" "ws-categories=2251799813685248" "ws-trusted=0" "ws-level=2" "ws-verdict=0" "ws-policy=pupils_(non_senior_school)" "ws-member=Pupils" "ws-module=2" "ws-msgtype=3" "ws-param1=None" "ws-param2=None" "ws-debug=None" "squid-gt-st=1469"

[Rafael Akchurin]

Can you please start playing a video in the browser and send me the url that is then in the browsers address box?

Best regards,

Rafael Akchurin

On 19 Jan 2021, at 15:50, 'austinfria...@googlemail.com' via Diladele Web Safety <web-s...@googlegroups.com> wrote:

testing from a windows client - im not sure how I can check this.  I dont seem to get a hit for "youtube" in my logs at all.  PLenty of hits for googlevideo.  I cannot see any reference to video id in the access log

To view this discussion on the web visit https://groups.google.com/d/msgid/web-safety/837923af-f816-44ed-8737-b54259e9f8d8n%40googlegroups.com.

[austinfria...@googlemail.com]

the above log was using the URL:   https://www.youtube.com/watch?v=SkyQUqSi0Xc

I have that video as being  "category":"Entertainment"

[austinfria...@googlemail.com]

I did have " Strip query terms from the filtered URLs. If enabled, query terms are stripped from the full URL before saving it into the access log. One possible disadvantage is that YouTube videos and Google search reports will be empty. Size of the logs are greatly reduced though. Enabled by default.  " enabled.  When I removed this, the log is a little larger:

1611069316.078    127 10.1.5.130 TCP_MISS/200 1215996 GET https://r3---sn-bvvbax-ac5e.googlevideo.com/videoplayback?expire=1611090912&ei=gPcGYM_cKc2m1waR2LrACQ&ip=xxxxxxxxxx&id=o-AHllq3rBAEn9I2qOCCmHkcEmhVt4MOoRH23SgqxlDH_g&itag=397&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C394%2C395%2C396%2C397%2C398%2C399&source=youtube&requiressl=yes&mh=Pc&mm=31%2C29&mn=sn-bvvbax-ac5e%2Csn-aigzrner&ms=au%2Crdu&mv=m&mvi=3&pl=21&initcwndbps=1411250&vprv=1&mime=video%2Fmp4&ns=yipeyN19exJ0hlZcMe3hY60F&gir=yes&clen=19233302&dur=340.298&lmt=1610944668199604&mt=1611068678&fvip=3&keepalive=yes&c=WEB&txp=5531432&n=8aJASynApdN8UA&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cns%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhALHwYgKroJRkeKUgH-V88CfZLSANLm65Xv7_F6Y5pXdzAiBBBl9ZEFBmQDXWOlBTKVPUNBLhyhJaUs9ibcs2oMTSPg%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIgfoqfMhtlJdfskRpFIYW1USkYWWwePt3YkJaVdtg4KO0CIQDsH2IFcn472vVO9bpX2ViDYxE3NMxmO_S0PODoa1GCxQ%3D%3D&alr=yes&cpn=psqQaH-yrv2iHI8S&cver=2.20210114.08.00&range=1277222-2491990&rn=8&rbuf=17925 xxxxxxxxxxxxxxxxxx HIER_DIRECT/208.117.250.20 video/mp4 "ws-iid=60468" "ws-mac=00:00:00:00:00:00" "ws-duration=391" "ws-timing=0" "ws-mtime=0" "ws-scanflags=63" "ws-categories=2251799813685248" "ws-trusted=0" "ws-level=2" "ws-verdict=0" "ws-policy=pupils_(non_senior_school)" "ws-member=Pupils" "ws-module=2" "ws-msgtype=3" "ws-param1=None" "ws-param2=None" "ws-debug=None" "squid-gt-st=1597"

I stripped out the username (it has their email address) and our external IP.  This is from the raw squid access.log and using the above URL.  It seems to be using the correct policy but again, nothing is blocked.

[austinfria...@googlemail.com]

Oddly enough I seem to have fixed it.  taking the tick out of the logging feature has now produced a web safety block page scanned by "youtube_guard" with the correct "video SkyQUqSu0Xc belongs to the blocked category entertainment"

I had originally enabled that setting to cut down on log file sizes