michael/astro
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Fix complex MDX parsing (#50)

Browse source 
* Fix complex MDX parsing

This allows fully MDX support using the micromark MDX extension. One caveat is that if you do something like use the less than sign, you need to escape it because the parser expects these to be tags otherwise.

* Move micromark definition
This commit is contained in:
Matthew Phillips 2021-04-01 16:34:11 -04:00 committed by GitHub
parent 397b7145cc
commit 54ba9f5ee1
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
16 changed files with 136 additions and 36 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
examples/snowpack/astro/pages/concepts/dev-server.md
View file

@ -6,6 +6,6 @@ description: Snowpack's dev server is fast because it only rebuilds the files yo
![dev command output example](/img/snowpack-dev-startup-2.png) ![dev command output example](/img/snowpack-dev-startup-2.png)
`snowpack dev` - Snowpack's dev server is an instant dev environment for [unbundled development.](/concepts/how-snowpack-works) The dev server will build a file only when it's requested by the browser. That means that Snowpack can start up instantly (usually in **<50ms**) and scale to infinitely large projects without slowing down. In contrast, it's common to see 30+ second dev startup times when building large apps with a traditional bundler. `snowpack dev` - Snowpack's dev server is an instant dev environment for [unbundled development.](/concepts/how-snowpack-works) The dev server will build a file only when it's requested by the browser. That means that Snowpack can start up instantly (usually in **\<50ms**) and scale to infinitely large projects without slowing down. In contrast, it's common to see 30+ second dev startup times when building large apps with a traditional bundler.
Snowpack supports JSX & TypeScript source code by default. You can extend your build even further with [custom plugins](/plugins) that connect Snowpack with your favorite build tools: TypeScript, Babel, Vue, Svelte, PostCSS, Sass... go wild! Snowpack supports JSX & TypeScript source code by default. You can extend your build even further with [custom plugins](/plugins) that connect Snowpack with your favorite build tools: TypeScript, Babel, Vue, Svelte, PostCSS, Sass... go wild!

20
examples/snowpack/astro/pages/guides/web-worker.md
View file

@ -28,22 +28,4 @@ const worker = new Worker(new URL('./esm-worker.js', import.meta.url), {
name: 'my-worker', name: 'my-worker',
type: 'module', type: 'module',
}); });
``` ```
<!--
TO REPLACE THE PREVIOUS PARAGRAPH ON v3.0.0 LAUNCH DAY:
Modern browsers have begun to support ESM syntax (`import`/`export`) inside of Web Workers. However, some notable exceptions still exist. To use ESM syntax inside of a web worker, consult [caniuse.com](https://caniuse.com/mdn-api_worker_worker_ecmascript_modules) and choose a supported browser for your local development. When you build for production, choose a bundler that will bundle your Web Worker to remove ESM import/export syntax. Currently, Snowpack's builtin bundler and @snowpack/plugin-webpack both support automatic Web Worker bundling to remove ESM syntax from web workers.
```js
const worker = new Worker(
new URL('./esm-worker.js', import.meta.url),
{
name: 'my-worker',
type: import.meta.env.MODE === 'development' ? "module" : "classic"
}
);
```
-->

2
examples/snowpack/astro/pages/posts/2020-05-26-snowpack-2-0-release.md
View file

@ -10,7 +10,7 @@ date: 2020-05-26
After 40+ beta versions & release candidates we are very excited to introduce **Snowpack 2.0: A build system for the modern web.** After 40+ beta versions & release candidates we are very excited to introduce **Snowpack 2.0: A build system for the modern web.**
- Starts up in <50ms and stays fast in large projects. - Starts up in \<50ms and stays fast in large projects.
- Bundle-free development with bundled production builds. - Bundle-free development with bundled production builds.
- Built-in support for TypeScript, JSX, CSS Modules and more. - Built-in support for TypeScript, JSX, CSS Modules and more.
- Works with React, Preact, Vue, Svelte, and all your favorite libraries. - Works with React, Preact, Vue, Svelte, and all your favorite libraries.

4
examples/snowpack/astro/pages/reference/configuration.md
View file

@ -84,8 +84,6 @@ Example:
You can further customize this the build behavior for any mounted directory by using the expanded object notation: You can further customize this the build behavior for any mounted directory by using the expanded object notation:
<!-- snowpack/src/config.ts -->
```js ```js
// snowpack.config.js // snowpack.config.js
// Example: expanded object notation "mount" usage // Example: expanded object notation "mount" usage
@ -441,7 +439,7 @@ Run Snowpack's build pipeline through a file watcher. This option works best for
Toggles whether HTML fragments are transformed like full HTML pages. Toggles whether HTML fragments are transformed like full HTML pages.
HTML fragments are HTML files not starting with "<!doctype html>". HTML fragments are HTML files not starting with "\<!doctype html\>".
### buildOptions.jsxFactory ### buildOptions.jsxFactory

2
examples/snowpack/astro/pages/tutorials/getting-started.md
View file

@ -48,7 +48,7 @@ npm install --save-dev snowpack
## Snowpack's development server ## Snowpack's development server
Adding a basic HTML file allows us to run Snowpack's development server, an instant development environment for unbundled development. The development server builds a file only when it's requested by the browser. That means that Snowpack can start up instantly (usually in **<50 ms**) and scale to infinitely large projects without slowing down. In contrast, it's common to see 30+ second development startup times when building large apps with a traditional bundler. Adding a basic HTML file allows us to run Snowpack's development server, an instant development environment for unbundled development. The development server builds a file only when it's requested by the browser. That means that Snowpack can start up instantly (usually in **\<50 ms**) and scale to infinitely large projects without slowing down. In contrast, it's common to see 30+ second development startup times when building large apps with a traditional bundler.
Create an `index.html` in your project with the following contents: Create an `index.html` in your project with the following contents:

47
package-lock.json generated
View file

@ -354,6 +354,11 @@
"resolved": "https://registry.npmjs.org/@types/scheduler/-/scheduler-0.16.1.tgz", "resolved": "https://registry.npmjs.org/@types/scheduler/-/scheduler-0.16.1.tgz",
"integrity": "sha512-EaCxbanVeyxDRTQBkdLb3Bvl/HK7PBK6UJjsSixB0iHKoWxE5uu2Q/DgtpOhPIojN0Zl1whvOd7PoHs2P0s5eA==" "integrity": "sha512-EaCxbanVeyxDRTQBkdLb3Bvl/HK7PBK6UJjsSixB0iHKoWxE5uu2Q/DgtpOhPIojN0Zl1whvOd7PoHs2P0s5eA=="
}, },
"@types/unist": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/@types/unist/-/unist-2.0.3.tgz",
"integrity": "sha512-FvUupuM3rlRsRtCN+fDudtmytGO6iHJuuRKS1Ss0pG5z8oX0diNEw94UEL7hgDbpN94rgaK5R7sWm6RrSkZuAQ=="
},
"@types/yargs-parser": { "@types/yargs-parser": {
"version": "20.2.0", "version": "20.2.0",
"resolved": "https://registry.npmjs.org/@types/yargs-parser/-/yargs-parser-20.2.0.tgz", "resolved": "https://registry.npmjs.org/@types/yargs-parser/-/yargs-parser-20.2.0.tgz",
@ -1641,6 +1646,11 @@
"integrity": "sha512-39nnKffWz8xN1BU/2c79n9nB9HDzo0niYUqx6xyqUnyoAnQyyWpOTdZEeiCch8BBu515t4wp9ZmgVfVhn9EBpw==", "integrity": "sha512-39nnKffWz8xN1BU/2c79n9nB9HDzo0niYUqx6xyqUnyoAnQyyWpOTdZEeiCch8BBu515t4wp9ZmgVfVhn9EBpw==",
"dev": true "dev": true
}, },
"estree-util-is-identifier-name": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/estree-util-is-identifier-name/-/estree-util-is-identifier-name-1.1.0.tgz",
"integrity": "sha512-OVJZ3fGGt9By77Ix9NhaRbzfbDV/2rx9EP7YIDJTmsZSEc5kYn2vWcNccYyahJL2uAQZK2a5Or2i0wtIKTPoRQ=="
},
"estree-walker": { "estree-walker": {
"version": "3.0.0", "version": "3.0.0",
"resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.0.tgz", "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.0.tgz",
@ -2534,6 +2544,26 @@
"micromark": "~2.11.0" "micromark": "~2.11.0"
} }
}, },
"micromark-extension-mdx-expression": {
"version": "0.3.2",
"resolved": "https://registry.npmjs.org/micromark-extension-mdx-expression/-/micromark-extension-mdx-expression-0.3.2.tgz",
"integrity": "sha512-Sh8YHLSAlbm/7TZkVKEC4wDcJE8XhVpZ9hUXBue1TcAicrrzs/oXu7PHH3NcyMemjGyMkiVS34Y0AHC5KG3y4A==",
"requires": {
"micromark": "~2.11.0",
"vfile-message": "^2.0.0"
}
},
"micromark-extension-mdx-jsx": {
"version": "0.3.3",
"resolved": "https://registry.npmjs.org/micromark-extension-mdx-jsx/-/micromark-extension-mdx-jsx-0.3.3.tgz",
"integrity": "sha512-kG3VwaJlzAPdtIVDznfDfBfNGMTIzsHqKpTmMlew/iPnUCDRNkX+48ElpaOzXAtK5axtpFKE3Hu3VBriZDnRTQ==",
"requires": {
"estree-util-is-identifier-name": "^1.0.0",
"micromark": "~2.11.0",
"micromark-extension-mdx-expression": "^0.3.2",
"vfile-message": "^2.0.0"
}
},
"micromatch": { "micromatch": {
"version": "4.0.2", "version": "4.0.2",
"resolved": "https://registry.npmjs.org/micromatch/-/micromatch-4.0.2.tgz", "resolved": "https://registry.npmjs.org/micromatch/-/micromatch-4.0.2.tgz",
@ -3798,6 +3828,14 @@
"crypto-random-string": "^2.0.0" "crypto-random-string": "^2.0.0"
} }
}, },
"unist-util-stringify-position": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/unist-util-stringify-position/-/unist-util-stringify-position-2.0.3.tgz",
"integrity": "sha512-3faScn5I+hy9VleOq/qNbAd6pAx7iH5jYBMS9I1HgQVijz/4mv5Bvw5iw1sC/90CODiKo81G/ps8AJrISn687g==",
"requires": {
"@types/unist": "^2.0.2"
}
},
"untildify": { "untildify": {
"version": "4.0.0", "version": "4.0.0",
"resolved": "https://registry.npmjs.org/untildify/-/untildify-4.0.0.tgz", "resolved": "https://registry.npmjs.org/untildify/-/untildify-4.0.0.tgz",
@ -3877,6 +3915,15 @@
"spdx-expression-parse": "^3.0.0" "spdx-expression-parse": "^3.0.0"
} }
}, },
"vfile-message": {
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/vfile-message/-/vfile-message-2.0.4.tgz",
"integrity": "sha512-DjssxRGkMvifUOJre00juHoP9DPWuzjxKuMDrhNbk2TdaYYBNMStsNhEOt3idrtI12VQYM/1+iM0KOzXi4pxwQ==",
"requires": {
"@types/unist": "^2.0.0",
"unist-util-stringify-position": "^2.0.0"
}
},
"vue": { "vue": {
"version": "3.0.7", "version": "3.0.7",
"resolved": "https://registry.npmjs.org/vue/-/vue-3.0.7.tgz", "resolved": "https://registry.npmjs.org/vue/-/vue-3.0.7.tgz",

2
package.json
View file

@ -54,6 +54,8 @@
"magic-string": "^0.25.3", "magic-string": "^0.25.3",
"micromark": "^2.11.4", "micromark": "^2.11.4",
"micromark-extension-gfm": "^0.3.3", "micromark-extension-gfm": "^0.3.3",
"micromark-extension-mdx-expression": "^0.3.2",
"micromark-extension-mdx-jsx": "^0.3.3",
"node-fetch": "^2.6.1", "node-fetch": "^2.6.1",
"postcss": "^8.2.8", "postcss": "^8.2.8",
"react": "^17.0.1", "react": "^17.0.1",

12
src/@types/micromark.ts Normal file
View file

@ -0,0 +1,12 @@
export interface MicromarkExtensionContext {
sliceSerialize(node: any): string;
raw(value: string): void;
}
export type MicromarkExtensionCallback = (this: MicromarkExtensionContext, node: any) => void;
export interface MicromarkExtension {
enter?: Record<string, MicromarkExtensionCallback>;
exit?: Record<string, MicromarkExtensionCallback>;
}

10
src/compiler/index.ts
View file

@ -8,8 +8,9 @@ import matter from 'gray-matter';
import gfmHtml from 'micromark-extension-gfm/html.js'; import gfmHtml from 'micromark-extension-gfm/html.js';
import { parse } from '../parser/index.js'; import { parse } from '../parser/index.js';
import { createMarkdownHeadersCollector } from '../micromark-collect-headers.js'; import { createMarkdownHeadersCollector } from './markdown/micromark-collect-headers.js';
import { encodeMarkdown } from '../micromark-encode.js'; import { encodeMarkdown } from './markdown/micromark-encode.js';
import { encodeAstroMdx } from './markdown/micromark-mdx-astro.js';
import { optimize } from './optimize/index.js'; import { optimize } from './optimize/index.js';
import { codegen } from './codegen.js'; import { codegen } from './codegen.js';
@ -56,10 +57,11 @@ async function convertMdToJsx(
): Promise<TransformResult> { ): Promise<TransformResult> {
const { data: frontmatterData, content } = matter(contents); const { data: frontmatterData, content } = matter(contents);
const { headers, headersExtension } = createMarkdownHeadersCollector(); const { headers, headersExtension } = createMarkdownHeadersCollector();
const { htmlAstro, mdAstro } = encodeAstroMdx();
const mdHtml = micromark(content, { const mdHtml = micromark(content, {
allowDangerousHtml: true, allowDangerousHtml: true,
extensions: [gfmSyntax()], extensions: [gfmSyntax(), ...htmlAstro],
htmlExtensions: [gfmHtml, encodeMarkdown, headersExtension], htmlExtensions: [gfmHtml, encodeMarkdown, headersExtension, mdAstro],
}); });
// TODO: Warn if reserved word is used in "frontmatterData" // TODO: Warn if reserved word is used in "frontmatterData"

0
src/micromark-collect-headers.ts → src/compiler/markdown/micromark-collect-headers.ts
View file

13
src/micromark-encode.ts → src/compiler/markdown/micromark-encode.ts
View file

@ -1,4 +1,5 @@
import type { HtmlExtension, Token, Tokenize } from 'micromark/dist/shared-types'; import type { Token } from 'micromark/dist/shared-types';
import type { MicromarkExtension, MicromarkExtensionContext } from '../../@types/micromark';
const characterReferences = { const characterReferences = {
'"': 'quot', '"': 'quot',
@ -19,15 +20,13 @@ function encode(value: string): string {
} }
/** Encode Markdown node */ /** Encode Markdown node */
function encodeToken(this: Record<string, () => void>) { function encodeToken(this: MicromarkExtensionContext) {
const token: Token = arguments[0]; const token: Token = arguments[0];
const serialize = (this.sliceSerialize as unknown) as (t: Token) => string; const value = this.sliceSerialize(token);
const raw = (this.raw as unknown) as (s: string) => void; this.raw(encode(value));
const value = serialize(token);
raw(encode(value));
} }
const plugin: HtmlExtension = { const plugin: MicromarkExtension = {
exit: { exit: {
codeTextData: encodeToken, codeTextData: encodeToken,
codeFlowValue: encodeToken, codeFlowValue: encodeToken,

23
src/compiler/markdown/micromark-mdx-astro.ts Normal file
View file

@ -0,0 +1,23 @@
import type { MicromarkExtension } from '../../@types/micromark';
import mdxExpression from 'micromark-extension-mdx-expression';
import mdxJsx from 'micromark-extension-mdx-jsx';
/**
* Keep MDX.
*/
export function encodeAstroMdx() {
const extension: MicromarkExtension = {
enter: {
mdxJsxFlowTag(node: any) {
const mdx = this.sliceSerialize(node);
this.raw(mdx);
}
}
};
return {
htmlAstro: [mdxExpression(), mdxJsx()],
mdAstro: extension
};
}

11
src/compiler/markdown/micromark.d.ts vendored Normal file
View file

@ -0,0 +1,11 @@
declare module 'micromark-extension-mdx-expression' {
import type { HtmlExtension } from 'micromark/dist/shared-types';
export default function(): HtmlExtension;
}
declare module 'micromark-extension-mdx-jsx' {
import type { HtmlExtension } from 'micromark/dist/shared-types';
export default function(): HtmlExtension;
}

8
test/astro-markdown.test.js
View file

@ -42,4 +42,12 @@ Markdown('Can load markdown pages with hmx', async () => {
assert.ok($('#test').length, 'There is a div added via a component from markdown'); assert.ok($('#test').length, 'There is a div added via a component from markdown');
}); });
Markdown('Can load more complex jsxy stuff', async () => {
const result = await runtime.load('/complex');
const $ = doc(result.contents);
const $el = $('#test');
assert.equal($el.text(), 'Hello world');
});
Markdown.run(); Markdown.run();

5
test/fixtures/astro-markdown/astro/components/Hello.jsx vendored Normal file
View file

@ -0,0 +1,5 @@
import { h } from 'preact';
export default function({ name }) {
return <div id="test">Hello {name}</div>
}

11
test/fixtures/astro-markdown/astro/pages/complex.md vendored Normal file
View file

@ -0,0 +1,11 @@
---
layout: ../layouts/content.astro
title: My Blog Post
description: This is a post about some stuff.
import:
Hello: '../components/Hello.jsx'
---
## Interesting Topic
<Hello name={`world`} />