Go to file
2023-01-26 09:36:05 -06:00
migrations Initial 2023-01-26 09:36:05 -06:00
src Initial 2023-01-26 09:36:05 -06:00
.env Initial 2023-01-26 09:36:05 -06:00
.envrc Initial 2023-01-26 09:36:05 -06:00
.gitignore Initial 2023-01-26 09:36:05 -06:00
Cargo.lock Initial 2023-01-26 09:36:05 -06:00
Cargo.toml Initial 2023-01-26 09:36:05 -06:00
default.nix Initial 2023-01-26 09:36:05 -06:00
docker-compose.yml Initial 2023-01-26 09:36:05 -06:00
flake.lock Initial 2023-01-26 09:36:05 -06:00
flake.nix Initial 2023-01-26 09:36:05 -06:00
README.md Initial 2023-01-26 09:36:05 -06:00
rustfmt.toml Initial 2023-01-26 09:36:05 -06:00

Crawling requirements

  • You must serve a robots.txt to describe the terms of crawling. Otherwise, it will be ignored by this bot.

  • You must serve an atom feed.

RFC References

  • RFC4287: The Atom Syndication Format
  • RFC9309: Proposed Robots Exclusion Protocol