fix: update USER_AGENT to mimic browser to bypass publisher WAFs

[vedant1711]

Updates the spider's USER_AGENT to a standard Chrome browser string to bypass publisher Bot Protections and CloudFront WAFs that were blocking PDF downloads for open access articles.

[boyechko]

The branch carries the 4 commits from the OSTI spider work (14e200a, 56afb6c, b66ac78, 7076e75), which already landed on main via PR #107 that was squashed and merged. They're harmless, but make it hard to see what this PR is actually doing.

If you want to clean that up before we merge, this drops them and keeps just your commit:

git fetch origin
git rebase --onto origin/main 7076e75
git push --force-with-lease

[boyechko]

Regarding commit dc9a6c0:

I'm concerned that changing the base setting for USER_AGENT to conceal who we are goes against a P1 functional requirement that "System shall obey robots.txt and other scraping etiquette, such as limiting the number of parallel requests to the same domain."

If we do want to circumvent that for OSTI's unreasonable settings, it would be better to do it via custom_settings the way you did for ROBOTSTXT_OBEY in 7076e75.

Moreover, if I am understanding correctly, setting USER_AGENT and ROBOTSTXT_OBEY in base.py to Chrome UA would only work reliably if ENVIRONMENT=development, since production.py resets USER_AGENT to the self-identifying value:

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = "open_ire (+https://lib.uw.edu/)"

[vedant1711]

Thanks @boyechko. Both points well taken, and honestly you've made me want to question this change rather than defend it.

Branch hygiene: done. I rebased with git rebase --onto origin/main 7076e75, so the four already-merged OSTI commits are gone and this is now a single commit.

The change itself: you're right on all three counts. base.py made concealment the global default (against our P1 etiquette requirement), was the wrong scope, and wouldn't even work in production since production.py resets USER_AGENT. What I've pushed instead keeps base.py honest and scopes a UA to just the OSTI spider via custom_settings. As a bonus that also fixes the production problem: custom_settings is 'spider' priority, which outranks production.py, so it actually applies everywhere.

"USER_AGENT": (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 "
    "(open_ire; +https://lib.uw.edu/; contact [email protected])"
)

The idea: browser-shaped (to slip past WAFs that block anything "script-like") but still naming the project plus a contact address, so we're not hiding who we are. That said, I'm not convinced we should do this at all, and I'd rather decide together:

1. Does it even help? A WAF doing real bot-detection won't be fooled by a UA string, and a strict allowlist won't care about the contact note. The only cases it works are dumb substring filters. Worth it?
2. Is my distinction real? These are open-access PDFs and the block looks indiscriminate rather than a deliberate "no bots" policy, which feels different from evading an unwilling host. But I might be rationalizing. Where would you draw the line against the P1 requirement?
3. Or just don't? Prefer the front door (publisher APIs, or asking to be allowlisted) and accept we don't get every PDF.

I left a NOTE: in the code pointing here so it isn't silently merged. My weak lean is toward option 3 unless we can confirm the scoped UA reliably helps on real sources. Happy to write this up as explicit project policy too.