+1 for the issue that was annoying for so long now Yes we do use GitHub caching, advocated for it to appear, and hope it stays So I set out digging in the data, and found that cached HTTP-404s in *.0 files correlate with very short *.1 files (the compact error message from Github REST API), so selecting those to look deeper: ```` :; find . -name '.1' -size 1 | sed -e 's,^./,,' -e s',.1$,,' | while read F ; do egrep 'HTTP.*404' "$F.0" >&2 && echo "=== $F" && head -1 "$F.0" && ls -la "$F" ; done ```` Due to reasons unknown however, the cached response for some of the URLs is HTTP/404 even with a valid JSON in the (gzipped) `hashstring.1` file: ```` {"message":"No commit found for the ref refs/heads/4.2.0-FTY","documentation_url":"https://developer.github.com/v3/repos/contents/"} ```` and the corresponding `hashstring.0` file looks like: ```` :; cat fbe7227813e6f1a6bbb2f1e5202a84a2.0 https://api.github.com/repos/42ity/libzmq/contents/?ref=refs%2Fheads%2F4.2.0-FTY GET 1 Authorization: Basic NDJpdHktY2k6NjA5MDk2YTVmNzNhNTc1YzE1OWYxZjI3NDJlZmI1YjhiMTQzZmIzMw== HTTP/1.1 404 Not Found 31 X-OAuth-Scopes: admin:repo_hook, public_repo, repo:status, repo_deployment X-Accepted-OAuth-Scopes: X-GitHub-Media-Type: github.v3; format=json Content-Encoding: gzip Transfer-Encoding: chunked Connection: keep-alive Content-Type: application/octet-stream X-Cache: MISS from thunderbolt.localdomain X-Cache-Lookup: MISS from thunderbolt.localdomain:8080 Via: 1.1 thunderbolt.localdomain (squid/3.4.4) Server: GitHub.com Date: Thu, 28 Nov 2019 00:41:22 GMT Status: 304 Not Modified X-RateLimit-Limit: 5000 X-RateLimit-Remaining: 5000 X-RateLimit-Reset: 1574905280 Cache-Control: private, max-age=60, s-maxage=60 Vary: Accept, Authorization, Cookie, X-GitHub-OTP ETag: "2513f4bbc2abb8b63adbec8336a82810a4fb5dc5" Last-Modified: Wed, 05 Dec 2018 10:54:24 GMT Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type Access-Control-Allow-Origin: * Strict-Transport-Security: max-age=31536000; includeSubdomains; preload X-Frame-Options: deny X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin Content-Security-Policy: default-src 'none' X-GitHub-Request-Id: F066:2FC4:FE494:24E933:5DDF17B1 OkHttp-Sent-Millis: 1574901681913 OkHttp-Received-Millis: 1574901682110 TLS_RSA_WITH_AES_128_GCM_SHA256 2 MIIECDCCAvCgAwIBAgIUEG8XFkmTLxiL4iPSXqLddY7e6AswDQYJKoZIhvcNAQEFBQAwga0xCzAJBgNVBAYTAkNaMRcwFQYDVQQIDA5QcmFndWUgc3VidXJiczEQMA4GA1UEBwwHUm96dG9reTENMAsGA1UECgwERUVJQzERMA8GA1UECwwIQklPUyBMQUIxJDAiBgNVBAMMG3RodW5kZXJib2x0LnJvei5sYWIuZXRuLmNvbTErMCkGCSqGSIb3DQEJARYcRWF0b25JUENPcGVuc291cmNlQEVhdG9uLmNvbTAeFw0xOTA3MDgwMDAwMDBaFw0yMDA3MTYxMjAwMDBaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2NvMRUwEwYDVQQKEwxHaXRIdWIsIEluYy4xFTATBgNVBAMMDCouZ2l0aHViLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKIFH+JTppW1pvbrqnLU1SCYOsFsI6vdoL66M/497v413h1TOEwGWEo1wvZq3YhD65VSlxrsEj7xGd+ZUy2/mzRh2XmGRolJUWd/XKCQ+lJukRLX3BYhRBXfGK9Njv/afR1OIs96A4dTZA7PpPwC5Gvk34iTcJe4gilud//3UqD55A0jk+uEwQqosAImeGQg4Ayqo3K5rR+NhF8NnR7kXT1Cijk6jySbgX5Lhu8FPu7LdiPntxjuvFNJNaRy+6t4PxHJ1iRRlDdsVHyZMcZGb8klafrKsr7kLBWSMKiVaXTdlNc26bUOctH+LySlZB6Q7LgSec3MBqXZBFk0AzfwxPcCAwEAAaNkMGIwIwYDVR0RBBwwGoIMKi5naXRodWIuY29tggpnaXRodWIuY29tMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUFAAOCAQEAocIF+SVNlLFzWv0A/OUu4TG+aRBdzplMrF6Gy8JxwBSp22SB1PD2H71R5bi4U7UA3vgnpLbyg283XhZndNern1rIf49XXTqFbPC1xcZi85NcYc6xE18pnO0GQRaVgple2MOZXrn32FPgV2Zn/5XxGlQU1eL8leLc8tvMZkokmuBWRkuvCkx7xM5YMSAo4lRsL6zqzio/RLTOqWP1d6qSsGsf3Zc4HJ5RUTeA2QnyO1TRVvO+8bo5rQUHBOVmYhc006zs35LsjaUhG/6R1POZW2OS55U8ArQgLE/dZZV9mNJsTdd2hefv3v0+/whB+Y3stiO7zDMVFIOoHEd0+cUfGg== MIIELzCCAxegAwIBAgIJAOz23xAU+F0TMA0GCSqGSIb3DQEBCwUAMIGtMQswCQYDVQQGEwJDWjEXMBUGA1UECAwOUHJhZ3VlIHN1YnVyYnMxEDAOBgNVBAcMB1JvenRva3kxDTALBgNVBAoMBEVFSUMxETAPBgNVBAsMCEJJT1MgTEFCMSQwIgYDVQQDDBt0aHVuZGVyYm9sdC5yb3oubGFiLmV0bi5jb20xKzApBgkqhkiG9w0BCQEWHEVhdG9uSVBDT3BlbnNvdXJjZUBFYXRvbi5jb20wHhcNMTgwNDAzMTIxNzU2WhcNMjgwMzMxMTIxNzU2WjCBrTELMAkGA1UEBhMCQ1oxFzAVBgNVBAgMDlByYWd1ZSBzdWJ1cmJzMRAwDgYDVQQHDAdSb3p0b2t5MQ0wCwYDVQQKDARFRUlDMREwDwYDVQQLDAhCSU9TIExBQjEkMCIGA1UEAwwbdGh1bmRlcmJvbHQucm96LmxhYi5ldG4uY29tMSswKQYJKoZIhvcNAQkBFhxFYXRvbklQQ09wZW5zb3VyY2VARWF0b24uY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAogUf4lOmlbWm9uuqctTVIJg6wWwjq92gvroz/j3u/jXeHVM4TAZYSjXC9mrdiEPrlVKXGuwSPvEZ35lTLb+bNGHZeYZGiUlRZ39coJD6Um6REtfcFiFEFd8Yr02O/9p9HU4iz3oDh1NkDs+k/ALka+TfiJNwl7iCKW53//dSoPnkDSOT64TBCqiwAiZ4ZCDgDKqjcrmtH42EXw2dHuRdPUKKOTqPJJuBfkuG7wU+7st2I+e3GO68U0k1pHL7q3g/EcnWJFGUN2xUfJkxxkZvySVp+sqyvuQsFZIwqJVpdN2U1zbptQ5y0f4vJKVkHpDsuBJ5zcwGpdkEWTQDN/DE9wIDAQABo1AwTjAdBgNVHQ4EFgQUAf/vfDxEB9kv3Cfo9fb3ikvyWNswHwYDVR0jBBgwFoAUAf/vfDxEB9kv3Cfo9fb3ikvyWNswDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAlwBAM+b+mxtzgP+Q5AFWzLqj2TwSWXERGNnZQFDVeoZXb2y7UqaAf+Dz8WvTOrn51/fE5jsyqYHUCBXucbFJIuFx4G7vhsspcraIgenTGoP5N4L2UamrEkrqBl1CkYVhP2aykdA9G2Tu/61/rHMNycuLCf/CrZA54QlVQ8M8KtAQo+CEKcGeDBabP4TOtWvPO7ScM9kj5vRTiwy0DaVIL2VaNWLsdqT9tQ8e01wB1CRtjBFb1lhr3zMT0wXF8gAA9zcL6h1/1yiD5lNFKYUTKtsAuLpNb51lUq1k8eshyqiCHMrSm9/nj4L1WcWSiiR4MxvU2DTGUmwrKJ6Z3tf1Xw== 0 ```` It seems that a large portion of such files appeared Jul 22 between 15:45-16:30 UTC so maybe there was an outage of GitHub at that time... there were a few this year. The few other short files apparently point to scans/builds of recently merged PRs so the ephemeral branch is really not there. For reasons unknown, the "Date:" timestamp in the .0 header file is fresh, probably from the last scan; the result and content on-disk remain unchanged. Manually submitted requests through same proxy do return expected contents of the Git branch (wrapped into GitHub's REST API JSON markup). Previously tried forcing the job configs to be not-disabled (via on-disk XMLs and reload of Jenkins configuration), this got the jobs not-marked with gray balls in the dashboard... but then they were re-marked probably due to this cache issue. For our OrgFolders making MultiBranch pipelines, the half-successful magic looked like this: ```` :; for D in /var/lib/jenkins/jobs//jobs ; do ( cd "$D" && for F in */branches//config.xml ; do sed 's,<disabled>true</disabled>,<disabled>false</disabled>,' -i "$F" ; done ); done ```` |