Feat: @astrojs/rss package! (#3271)

* feat: introduce @astrojs/rss package!

* feat: add config "site" to env variable

* docs: add @astrojs/rss readme

* chore: changeset

* fix: testing script

* deps: add mocha, chai, chai-promises

* tests: add rss test!

* feat: add canonicalUrl arg

* chore: remove console.log

* fix: remove null check on env (breaks build)

* docs: stray `

* chore: update error message to doc link

* chore: remove getStylesheet

* docs: update stylesheet reference
This commit is contained in:
Ben Holmes 2022-05-03 18:26:13 -04:00 committed by GitHub
parent e2a037be94
commit fbfb6190ab
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
9 changed files with 538 additions and 0 deletions

View file

@ -0,0 +1,6 @@
---
'astro': patch
'@astrojs/rss': patch
---
Introduce new @astrojs/rss package for RSS feed generation! This also adds a new global env variable for your project's configured "site": import.meta.env.SITE. This is consumed by the RSS feed helper to generate the correct canonical URL.

View file

@ -0,0 +1,147 @@
# @astrojs/rss 📖
This package brings fast RSS feed generation to blogs and other content sites built with [Astro](https://astro.build/). For more information about RSS feeds in general, see [aboutfeeds.com](https://aboutfeeds.com/).
## Installation
Install the `@astrojs/rss` package into any Astro project using your preferred package manager:
```bash
# npm
npm i @astrojs/rss
# yarn
yarn add @astrojs/rss
# pnpm
pnpm i @astrojs/rss
```
## Example usage
The `@astrojs/rss` package provides helpers for generating RSS feeds within [Astro endpoints][astro-endpoints]. This unlocks both static builds _and_ on-demand generation when using an [SSR adapter](https://docs.astro.build/en/guides/server-side-rendering/#enabling-ssr-in-your-project).
For instance, say you need to generate an RSS feed for all posts under `src/pages/blog/`. Start by [adding a `site` to your project's `astro.config` for link generation](https://docs.astro.build/en/reference/configuration-reference/#site). Then, create an `rss.xml.js` file under your project's `src/pages/` directory, and use [Vite's `import.meta.glob` helper](https://vitejs.dev/guide/features.html#glob-import) like so:
```js
// src/pages/rss.xml.js
import rss from '@astrojs/rss';
export const get = () => rss({
title: 'Buzzs Blog',
description: 'A humble Astronauts guide to the stars',
items: import.meta.glob('./blog/**/*.md'),
});
```
Read **[Astro's RSS docs][astro-rss]** for full usage examples.
## `rss()` configuration options
The `rss` default export offers a number of configuration options. Here's a quick reference:
```js
rss({
// `<title>` field in output xml
title: 'Buzzs Blog',
// `<description>` field in output xml
description: 'A humble Astronauts guide to the stars',
// list of `<item>`s in output xml
items: import.meta.glob('./**/*.md'),
// (optional) absolute path to XSL stylesheet in your project
stylesheet: '/rss-styles.xsl',
// (optional) inject custom xml
customData: '<language>en-us</language>',
// (optional) add arbitrary metadata to opening <rss> tag
xmlns: { h: 'http://www.w3.org/TR/html4/' },
// (optional) provide a canonical URL
// defaults to the "site" configured in your project's astro.config
canonicalUrl: 'https://stargazers.club',
});
```
### title
Type: `string (required)`
The `<title>` attribute of your RSS feed's output xml.
### description
Type: `string (required)`
The `<description>` attribute of your RSS feed's output xml.
### items
Type: `RSSFeedItem[] | GlobResult (required)`
Either a list of formatted RSS feed items or the result of [Vite's `import.meta.glob` helper](https://vitejs.dev/guide/features.html#glob-import). See [Astro's RSS items documentation](https://docs.astro.build/en/guides/rss/#generating-items) for usage examples to choose the best option for you.
When providing a formatted RSS item list, see the `RSSFeedItem` type reference below:
```ts
type RSSFeedItem = {
/** Link to item */
link: string;
/** Title of item */
title: string;
/** Publication date of item */
pubDate: Date;
/** Item description */
description?: string;
/** Append some other XML-valid data to this item */
customData?: string;
};
```
### stylesheet
Type: `string (optional)`
An absolute path to an XSL stylesheet in your project. If you dont have an RSS stylesheet in mind, we recommend the [Pretty Feed v3 default stylesheet](https://github.com/genmon/aboutfeeds/blob/main/tools/pretty-feed-v3.xsl), which you can download from GitHub and save into your project's `public/` directory.
### customData
Type: `string (optional)`
A string of valid XML to be injected between your feed's `<description>` and `<item>` tags. This is commonly used to set a language for your feed:
```js
import rss from '@astrojs/rss';
export const get = () => rss({
...
customData: '<language>en-us</language>',
});
```
### xmlns
Type: `Record<string, string> (optional)`
An object mapping a set of `xmlns` suffixes to strings of metadata on the opening `<rss>` tag.
For example, this object:
```js
rss({
...
xmlns: { h: 'http://www.w3.org/TR/html4/' },
})
```
Will inject the following XML:
```xml
<rss xmlns:h="http://www.w3.org/TR/html4/"...
```
### canonicalUrl
Type: `string (optional)`
The base URL to use when generating RSS item links. This defaults to the [`site` configured in your project's `astro.config`](https://docs.astro.build/en/reference/configuration-reference/#site). We recommend using `site` instead of `canonicalUrl`, though we provide this option if an override is necessary.
For more on building with Astro, [visit the Astro docs][astro-rss].
[astro-rss]: https://docs.astro.build/en/guides/rss/#using-astrojsrss-recommended
[astro-endpoints]: https://docs.astro.build/en/core-concepts/astro-pages/#non-html-pages

View file

@ -0,0 +1,39 @@
{
"name": "@astrojs/rss",
"description": "Add RSS feeds to your Astro projects",
"version": "0.1.0",
"type": "module",
"types": "./dist/index.d.ts",
"author": "withastro",
"license": "MIT",
"repository": {
"type": "git",
"url": "https://github.com/withastro/astro.git",
"directory": "packages/astro-rss"
},
"bugs": "https://github.com/withastro/astro/issues",
"homepage": "https://astro.build",
"exports": {
".": "./dist/index.js",
"./package.json": "./package.json"
},
"scripts": {
"build": "astro-scripts build \"src/**/*.ts\" && tsc",
"build:ci": "astro-scripts build \"src/**/*.ts\"",
"dev": "astro-scripts dev \"src/**/*.ts\"",
"test": "mocha --exit --timeout 20000"
},
"devDependencies": {
"@types/chai": "^4.3.1",
"@types/chai-as-promised": "^7.1.5",
"@types/mocha": "^9.1.1",
"astro": "workspace:*",
"astro-scripts": "workspace:*",
"chai": "^4.3.6",
"chai-as-promised": "^7.1.1",
"mocha": "^9.2.2"
},
"dependencies": {
"fast-xml-parser": "^4.0.7"
}
}

View file

@ -0,0 +1,154 @@
import { XMLValidator } from 'fast-xml-parser';
import { createCanonicalURL, isValidURL } from './util.js';
type GlobResult = Record<string, () => Promise<{ [key: string]: any }>>;
type RSSOptions = {
/** (required) Title of the RSS Feed */
title: string;
/** (required) Description of the RSS Feed */
description: string;
/**
* List of RSS feed items to render. Accepts either:
* a) list of RSSFeedItems
* b) import.meta.glob result. You can only glob ".md" files within src/pages/ when using this method!
*/
items: RSSFeedItem[] | GlobResult;
/** Specify arbitrary metadata on opening <xml> tag */
xmlns?: Record<string, string>;
/**
* Specifies a local custom XSL stylesheet. Ex. '/public/custom-feed.xsl'
*/
stylesheet?: string | boolean;
/** Specify custom data in opening of file */
customData?: string;
/**
* Specify the base URL to use for RSS feed links.
* Defaults to "site" in your project's astro.config
*/
canonicalUrl?: string;
};
type RSSFeedItem = {
/** Link to item */
link: string;
/** Title of item */
title: string;
/** Publication date of item */
pubDate: Date;
/** Item description */
description?: string;
/** Append some other XML-valid data to this item */
customData?: string;
};
type GenerateRSSArgs = {
site: string;
rssOptions: RSSOptions;
items: RSSFeedItem[];
};
function isGlobResult(items: RSSOptions['items']): items is GlobResult {
return typeof items === 'object' && !items.length;
}
function mapGlobResult(items: GlobResult): Promise<RSSFeedItem[]> {
return Promise.all(
Object.values(items).map(async (getInfo) => {
const { url, frontmatter } = await getInfo();
if (!Boolean(url)) {
throw new Error(
`[RSS] When passing an import.meta.glob result directly, you can only glob ".md" files within /pages! Consider mapping the result to an array of RSSFeedItems. See the RSS docs for usage examples: https://docs.astro.build/en/guides/rss/#2-list-of-rss-feed-objects`
);
}
if (!Boolean(frontmatter.title) || !Boolean(frontmatter.pubDate)) {
throw new Error(`[RSS] "${url}" is missing a "title" and/or "pubDate" in its frontmatter.`);
}
return {
link: url,
title: frontmatter.title,
pubDate: frontmatter.pubDate,
description: frontmatter.description,
customData: frontmatter.customData,
};
})
);
}
export default async function getRSS(rssOptions: RSSOptions) {
const site = rssOptions.canonicalUrl ?? (import.meta as any).env.SITE;
if (!site) {
throw new Error(
`RSS requires a canonical URL. Either add a "site" to your project's astro.config, or supply the canonicalUrl argument.`
);
}
let { items } = rssOptions;
if (isGlobResult(items)) {
items = await mapGlobResult(items);
}
return {
body: await generateRSS({
site,
rssOptions,
items,
}),
};
}
/** Generate RSS 2.0 feed */
export async function generateRSS({ site, rssOptions, items }: GenerateRSSArgs): Promise<string> {
let xml = `<?xml version="1.0" encoding="UTF-8"?>`;
if (typeof rssOptions.stylesheet === 'string') {
xml += `<?xml-stylesheet href="${rssOptions.stylesheet}" type="text/xsl"?>`;
}
xml += `<rss version="2.0"`;
// xmlns
if (rssOptions.xmlns) {
for (const [k, v] of Object.entries(rssOptions.xmlns)) {
xml += ` xmlns:${k}="${v}"`;
}
}
xml += `>`;
xml += `<channel>`;
// title, description, customData
xml += `<title><![CDATA[${rssOptions.title}]]></title>`;
xml += `<description><![CDATA[${rssOptions.description}]]></description>`;
xml += `<link>${createCanonicalURL(site).href}</link>`;
if (typeof rssOptions.customData === 'string') xml += rssOptions.customData;
// items
for (const result of items) {
xml += `<item>`;
xml += `<title><![CDATA[${result.title}]]></title>`;
// If the item's link is already a valid URL, don't mess with it.
const itemLink = isValidURL(result.link)
? result.link
: createCanonicalURL(result.link, site).href;
xml += `<link>${itemLink}</link>`;
xml += `<guid>${itemLink}</guid>`;
if (result.description) xml += `<description><![CDATA[${result.description}]]></description>`;
if (result.pubDate) {
// note: this should be a Date, but if user provided a string or number, we can work with that, too.
if (typeof result.pubDate === 'number' || typeof result.pubDate === 'string') {
result.pubDate = new Date(result.pubDate);
} else if (result.pubDate instanceof Date === false) {
throw new Error('[${filename}] rss.item().pubDate must be a Date');
}
xml += `<pubDate>${result.pubDate.toUTCString()}</pubDate>`;
}
if (typeof result.customData === 'string') xml += result.customData;
xml += `</item>`;
}
xml += `</channel></rss>`;
// validate users inputs to see if its valid XML
const isValid = XMLValidator.validate(xml);
if (isValid !== true) {
// If valid XML, isValid will be `true`. Otherwise, this will be an error object. Throw.
throw new Error(isValid as any);
}
return xml;
}

View file

@ -0,0 +1,19 @@
import npath from 'path-browserify';
/** Normalize URL to its canonical form */
export function createCanonicalURL(url: string, base?: string): URL {
let pathname = url.replace(/\/index.html$/, ''); // index.html is not canonical
pathname = pathname.replace(/\/1\/?$/, ''); // neither is a trailing /1/ (impl. detail of collections)
if (!npath.extname(pathname)) pathname = pathname.replace(/(\/+)?$/, '/'); // add trailing slash if theres no extension
pathname = pathname.replace(/\/+/g, '/'); // remove duplicate slashes (URL() wont)
return new URL(pathname, base);
}
/** Check if a URL is already valid */
export function isValidURL(url: string): boolean {
try {
new URL(url);
return true;
} catch (e) {}
return false;
}

View file

@ -0,0 +1,122 @@
import rss from '../dist/index.js';
import chai from 'chai';
import chaiPromises from 'chai-as-promised';
chai.use(chaiPromises);
const title = 'My RSS feed';
const description = 'This sure is a nice RSS feed';
const canonicalUrl = 'https://example.com';
const phpFeedItem = {
link: '/php',
title: 'Remember PHP?',
pubDate: '1994-05-03',
description: 'PHP is a general-purpose scripting language geared toward web development. It was originally created by Danish-Canadian programmer Rasmus Lerdorf in 1994.',
};
const web1FeedItem = {
link: '/web1',
title: 'Web 1.0',
pubDate: '1997-05-03',
description: 'Web 1.0 is the term used for the earliest version of the Internet as it emerged from its origins with Defense Advanced Research Projects Agency (DARPA) and became, for the first time, a global network representing the future of digital communications.',
};
// note: I spent 30 minutes looking for a nice node-based snapshot tool
// ...and I gave up. Enjoy big strings!
const validXmlResult = `<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"><channel><title><![CDATA[My RSS feed]]></title><description><![CDATA[This sure is a nice RSS feed]]></description><link>https://example.com/</link><item><title><![CDATA[Remember PHP?]]></title><link>https://example.com/php/</link><guid>https://example.com/php/</guid><description><![CDATA[PHP is a general-purpose scripting language geared toward web development. It was originally created by Danish-Canadian programmer Rasmus Lerdorf in 1994.]]></description><pubDate>Tue, 03 May 1994 00:00:00 GMT</pubDate></item><item><title><![CDATA[Web 1.0]]></title><link>https://example.com/web1/</link><guid>https://example.com/web1/</guid><description><![CDATA[Web 1.0 is the term used for the earliest version of the Internet as it emerged from its origins with Defense Advanced Research Projects Agency (DARPA) and became, for the first time, a global network representing the future of digital communications.]]></description><pubDate>Sat, 03 May 1997 00:00:00 GMT</pubDate></item></channel></rss>`;
describe('rss', () => {
it('should fail on missing "site" and/or "canonicalUrl"', () => {
return chai.expect(
rss({
title,
description,
items: [],
})
).to.be.rejected;
});
it('should generate on valid RSSFeedItem array', async () => {
const { body } = await rss({
title,
description,
items: [phpFeedItem, web1FeedItem],
canonicalUrl,
});
chai.expect(body).to.equal(validXmlResult);
})
describe('glob result', () => {
it('should generate on valid result', async () => {
const globResult = {
'./posts/php.md': () => new Promise(resolve => resolve({
url: phpFeedItem.link,
frontmatter: {
title: phpFeedItem.title,
pubDate: phpFeedItem.pubDate,
description: phpFeedItem.description,
},
})),
'./posts/nested/web1.md': () => new Promise(resolve => resolve({
url: web1FeedItem.link,
frontmatter: {
title: web1FeedItem.title,
pubDate: web1FeedItem.pubDate,
description: web1FeedItem.description,
},
})),
};
const { body } = await rss({
title,
description,
items: globResult,
canonicalUrl,
});
chai.expect(body).to.equal(validXmlResult);
});
it('should fail on missing "title" key', () => {
const globResult = {
'./posts/php.md': () => new Promise(resolve => resolve({
url: phpFeedItem.link,
frontmatter: {
pubDate: phpFeedItem.pubDate,
description: phpFeedItem.description,
},
})),
};
return chai.expect(
rss({
title,
description,
items: globResult,
canonicalUrl,
})
).to.be.rejected;
});
it('should fail on missing "pubDate" key', () => {
const globResult = {
'./posts/php.md': () => new Promise(resolve => resolve({
url: phpFeedItem.link,
frontmatter: {
title: phpFeedItem.title,
description: phpFeedItem.description,
},
})),
};
return chai.expect(
rss({
title,
description,
items: globResult,
canonicalUrl,
})
).to.be.rejected;
});
});
})

View file

@ -0,0 +1,10 @@
{
"extends": "../../tsconfig.base.json",
"include": ["src"],
"compilerOptions": {
"allowJs": true,
"module": "ES2020",
"outDir": "./dist",
"target": "ES2020"
}
}

View file

@ -71,6 +71,9 @@ export async function createVite(
publicDir: fileURLToPath(astroConfig.publicDir),
root: fileURLToPath(astroConfig.root),
envPrefix: 'PUBLIC_',
define: {
'import.meta.env.SITE': astroConfig.site ? `'${astroConfig.site}'` : 'undefined',
},
server: {
force: true, // force dependency rebuild (TODO: enabled only while next is unstable; eventually only call in "production" mode?)
hmr:

View file

@ -634,6 +634,29 @@ importers:
devDependencies:
prismjs: 1.28.0
packages/astro-rss:
specifiers:
'@types/chai': ^4.3.1
'@types/chai-as-promised': ^7.1.5
'@types/mocha': ^9.1.1
astro: workspace:*
astro-scripts: workspace:*
chai: ^4.3.6
chai-as-promised: ^7.1.1
fast-xml-parser: ^4.0.7
mocha: ^9.2.2
dependencies:
fast-xml-parser: 4.0.7
devDependencies:
'@types/chai': 4.3.1
'@types/chai-as-promised': 7.1.5
'@types/mocha': 9.1.1
astro: link:../astro
astro-scripts: link:../../scripts
chai: 4.3.6
chai-as-promised: 7.1.1_chai@4.3.6
mocha: 9.2.2
packages/astro/test/fixtures/0-css:
specifiers:
'@astrojs/react': workspace:*
@ -3883,6 +3906,12 @@ packages:
'@babel/types': 7.17.0
dev: true
/@types/chai-as-promised/7.1.5:
resolution: {integrity: sha512-jStwss93SITGBwt/niYrkf2C+/1KTeZCZl1LaeezTlqppAKeoQC7jxyqYuP72sxBGKCIbw7oHgbYssIRzT5FCQ==}
dependencies:
'@types/chai': 4.3.1
dev: true
/@types/chai/4.3.1:
resolution: {integrity: sha512-/zPMqDkzSZ8t3VtxOa4KPq7uzzW978M9Tvh+j7GHKuo6k6GTLxPJ4J5gE5cjfJ26pnXst0N5Hax8Sr0T2Mi9zQ==}
dev: true
@ -4976,6 +5005,15 @@ packages:
resolution: {integrity: sha512-eyrF0jiFpY+3drT6383f1qhkbGsLSifNAjA61IUjZjmLCWjItY6LB9ft9YhoDgwfmclB2zhu51Lc7+95b8NRAg==}
dev: false
/chai-as-promised/7.1.1_chai@4.3.6:
resolution: {integrity: sha512-azL6xMoi+uxu6z4rhWQ1jbdUhOMhis2PvscD/xjLqNMkv3BPPp2JyyuTHOrf9BOosGpNQ11v6BKv/g57RXbiaA==}
peerDependencies:
chai: '>= 2.1.2 < 5'
dependencies:
chai: 4.3.6
check-error: 1.0.2
dev: true
/chai/4.3.6:
resolution: {integrity: sha512-bbcp3YfHCUzMOvKqsztczerVgBKSsEijCySNlHHbX3VG1nskvqjz5Rfso1gGwD6w6oOV3eI60pKuMOV5MV7p3Q==}
engines: {node: '>=4'}