Support a web browser-looking user agent in requests

[johnthagen]

Some websites and CDNs are increasingly disallowing connections from non-browser-looking entities. This results in some false positive from URL checking as 403 is returned to htmlproofer, but the link is valid if opened from a browser.

I've had success adding a user-agent in the past like this:

        req = urllib.request.Request(
            url,
            headers={
                # Mimic a web browser to bypass certain CDN 403 blocking.
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36"
            },
        )

[johnthagen]

It looks like here we are specifically identifying as a bot:

mkdocs-htmlproofer-plugin/htmlproofer/plugin.py

Line 25 in 0ebf10e

_URL_BOT_ID = f'Bot {uuid.uuid4()}'

While technically true, I don't think users of htmlproofer are going to be massively scraping sites. Perhaps we should allow this to be configurable? Or statically set a web browser-looking user agent to avoid false positives?

[johnthagen]

As a test, I changed the line to

_URL_BOT_ID = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36"

And it fixed one of the false positives I had.

[manuzhang]

As always, feel free to open a PR.