14 lines
386 B
Markdown
14 lines
386 B
Markdown
## Crawling requirements
|
|
|
|
- You must serve a robots.txt to describe the terms of crawling. Otherwise, it
|
|
will be ignored by this bot.
|
|
|
|
- You must serve an atom feed.
|
|
|
|
## RFC References
|
|
|
|
- **[RFC4287]**: The Atom Syndication Format
|
|
- **[RFC9309]**: Proposed Robots Exclusion Protocol
|
|
|
|
[rfc4287]: https://www.rfc-editor.org/rfc/rfc4287
|
|
[rfc9309]: https://www.rfc-editor.org/rfc/rfc9309
|