Should we add a slight delay in between requests? If we do too much, we get a 429 like on your example :/

Random brainstorming, don't know how easy or difficult they are to implement in current code:

• delay as you described, if doable throttle per domain but not between different domains
• retry failed URLs after some time (e.g., 10s and print on console that URLs are retried so user knows why it's taking extra time)
• cache remote link check and only re-check after Cache-Control or X minutes/hours/days if successful (local links as part of Zola generation I'd still check every time)
• randomize link check order and ignore failed URLs if clearly identifiable as 429-alike errors

If throttling works that might already be it; but I don't know how site rate limiting heuristics usually trigger.

About the check command: I have an example run where this error is reported

Failed to check the site
Error: Dead link in /Users/maxfire/projects/web/maxild.github.io/content/component-based-css.md to https://gist.github.com/necolas/1024797#file-style-css-L196-L222: Anchor `#file-style-css-L196-L222` not found on page

But the link is valid.

Note: I can open a new issue (tomorrow), but reporting this in a hurry because I am out of time for an important appointment.

That link is valid but does not exist in the HTML since it's handled in JS. You want to skip those anchor checks with the skip_anchor_prefixes option: https://www.getzola.org/documentation/getting-started/configuration/

@Keats Thanks, but writing this in config.toml

# Skip anchor checking for external URLs that start with these prefixes
skip_anchor_prefixes = [
    # This link is valid but does not exist in the HTML since it's handled in JS
    #"https://gist.github.com/",
    "https://gist.github.com/necolas/1024797",
]

The check command still shows the same error?

@maxild can you try with:

[link_checker]
skip_anchor_prefixes = [
    "https://gist.github.com/necolas/1024797",
]

Place it just above the [extra] section.
It seems to work on my side. I think there is a pitfall in the documentation, the [link_checker] section is missing.
If you confirm it is ok on your side, I'll update the documentation.

It is there in the docs, it's just that TOML is not super obvious about sections...

@Keats yes you are right, the docs is ok. I think in such case we are focussing on the required keys and just forget about the section. But yes the issue is definitively between the chair and the keyboard. :)

I just stumbled across this overeagerness of the link checker today. On reasonably busy blogs like https://github.com/rust-embedded/blog with several hundred links into GH the limit is reached within seconds. Also the deduplication done in the linkchecker doesn't do much at all in most cases since the HashMap of the results is only updated when the result comes back from the server which means that if the requests end up in the same "batch" they'll be fired off in parallel anyway.

I've been thinking about ways to address that but a throttling mechanism really doesn't help too much here because it will just make everything slow without providing any real benefit. I think instead there should be a way to restrict link checking to new content, e.g. based on file modification times. No need to check (and potentially flag) hundreds of years old links over and over again in CI when adding new content...

Also the number of parallel threads should really be configurable instead of being a hardcoded number...

No need to check (and potentially flag) hundreds of years old links over and over again in CI when adding new content...

Content disappears all the time :/ I would say it's more likely that some link died in older content than recent content.

I'm wondering if we can group the links to check per domain as mentioned at the beginning of the issue and have some light (and maybe configurable) throttling for domains. All the links from the same domains would get sent to the same thread. Maybe reducing the number of threads would do the trick without having to do any throttling. The rust-embedded blog would be a good test case for that.

Content disappears all the time :/ I would say it's more likely that some link died in older content than recent content.

Sure. However I'm more worried about defunct links in new content rather than old link targets disappearing. For use in CI we certainly wouldn't want to put the onus on the contributors to fix broken links in old content, do we? I could imagine having a monthly complete check to flag broken links and CI only checking the links in files changed in the last 2 weeks.

I agree, maybe a url:last_check_timestamp dot file loaded from the root of the site?