Meeting Purpose
Troubleshoot broken link detection on
interlisp.org.
Key Takeaways - Problem: curl link-checking fails on ~33% of
bibliography links, generating false positives. - Cause:
Websites use JavaScript challenges (e.g., Anubi) to block
non-browser traffic, including simple curl requests. - Solution
Path: Use a browser automation tool (e.g., Selenium) for offline
checks, or validate the hypothesis that receiving a challenge
implies a live site.
Topics
Link Checker Failure - A curl script to check
interlisp.org
links failed on ~33% of bibliography links. - Cause: Websites
block automated traffic to counter threats like ad fraud and
DDoS attacks. - Mechanism: They issue JavaScript challenges
(e.g., Anubi) that require a browser to execute a cryptographic
proof-of-work. - curl Limitation: As a command-line tool, it
cannot execute JavaScript and thus fails these challenges.
Proposed Solutions - 1. Browser Automation: Use a tool like
Selenium to automate a real browser, which can execute
JavaScript. - Rationale: This would mimic human traffic and
bypass the challenges. - Implementation: Run as an offline job
(e.g., weekly) in a Docker container, not as part of the build
process. - 2. Validate Challenge-as-Proof Hypothesis: -
Hypothesis: Receiving a JavaScript challenge response from a URL
is sufficient proof that the site is live. - Validation Test:
Send a curl request to a known-bad URL on a challenge-protected
site to see if it still triggers a challenge. - Significance: A
positive result would simplify the check to a simple header
inspection, avoiding the complexity of browser automation.
Next Steps - Bill Stumbo: Test the "challenge-as-proof"
hypothesis by requesting a known-bad URL on a
challenge-protected site. - Team: If the hypothesis fails,
research browser automation tools (e.g., Selenium) for an
offline link-checking job.
