michael/searchinblog
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity
searchinblog/README.md
Michael Zhang 71f42e1d88 Initial
2023-01-26 09:36:05 -06:00

386 B
Raw Blame History

Crawling requirements

  • You must serve a robots.txt to describe the terms of crawling. Otherwise, it will be ignored by this bot.

  • You must serve an atom feed.

RFC References

  • RFC4287: The Atom Syndication Format
  • RFC9309: Proposed Robots Exclusion Protocol