Parsing and serialization of multiple code files in Markdown for LLMs
npm install llm-code-formatFrom source code to Markdown to LLMs and back again
A TypeScript library for parsing and serializing multiple code files in Markdown format, specifically designed for Large Language Model (LLM) interactions. The idea of this comes from the fact that when you ask an LLM to write code for you with multiple files, different models have different "preferences" when it comes to representing code. This library aims to provide a consistent way to parse and serialize code files in Markdown format, regardless of the format used by the LLM. It may also be useful for other applications that need to work with multiple code files in Markdown format, such as being able to extract code files from a Markdown document or generate a Markdown document from code files.
See also editcodewithai, a higher level code editing tool built on llm-code-format.
- Parse code blocks from various Markdown formats including:
- Bold Format (filename.js)
- Backtick-Heading Format (### filename.js`)### filename.js
- Standard Heading Format ()filename.js:
- Colon Format ()# filename.js
- Hash Format ()### File: filename.js
- File Bold Format ()### filename.js
- Heading Bold Format ()1. filename.js
- Numbered Bold Format ()### 1.
- Numbered Backtick Format (filename.js)
- Streaming parser for real-time code editing views
- Serialize code files back to Markdown in a consistent format
- TypeScript support with full type definitions
- Zero dependencies
- Comprehensive test coverage
`bash`
npm install llm-code-format
`typescript
import { parseMarkdownFiles, formatMarkdownFiles } from "llm-code-format";
// Parse Markdown content containing code blocks
const markdownString =
index.html
\\\html\Hello World
\\
styles.css
\\\css\
body { color: blue; }
\\;
const { files, format } = parseMarkdownFiles(markdownString);
console.log(format); // "Bold Format"
console.log(files); // Object with filenames as keys and contents as values
// Serialize files back to Markdown
const files = {
"index.html": "
const markdown = formatMarkdownFiles(files);
`
This "recipe" shows how to extract example code from blog posts such that they can be run (works with Astro):
`js
import { parseMarkdownFiles } from "llm-code-format";
import fs from "fs";
import path from "path";
const blogDir = "src/pages/blog";
const publicDir = "public/examples";
// Read all markdown files from blog directory
fs.readdirSync(blogDir)
.filter((fileName) => fileName.endsWith(".md"))
.forEach((fileName) => {
const postName = fileName.replace(".md", "");
const filePath = path.join(blogDir, fileName);
const markdownString = fs.readFileSync(filePath, "utf8");
// Use llm-code-format to extract code blocks from markdown!
const { files } = parseMarkdownFiles(markdownString);
if (Object.keys(files).length === 0) return;
const outputDirectory = path.join(publicDir, postName);
if (fs.existsSync(outputDirectory)) {
fs.rmSync(outputDirectory, { recursive: true });
}
fs.mkdirSync(outputDirectory);
Object.entries(files).forEach(([name, content]) => {
fs.writeFileSync(path.join(outputDirectory, name), content);
});
});
`
Parses a Markdown string containing code blocks and returns an object with:
- files: Object with filenames as keys and file contents as valuesformat
- : String indicating the detected format
Optional format parameter to specify a particular format to parse:
- 'backtick-heading'
- 'file-bold'
- 'numbered-backtick'
- 'heading-bold'
- 'standard-heading'
- 'colon'
- 'bold'
- 'hash'
- 'numbered-bold'
Converts a FileCollection object into a Markdown string using the Bold Format.
A streaming parser that processes Markdown content chunk by chunk, emitting callbacks when file names or code blocks are detected. This is particularly useful for real-time code editing views where users can see LLMs edit specific files as they generate content.
``typescript
import {
StreamingMarkdownParser,
StreamingParserCallbacks,
} from "llm-code-format";
// Define callbacks for file name changes, code lines, and non-code lines
const callbacks: StreamingParserCallbacks = {
onFileNameChange: async (fileName, format) => {
console.log(File changed to: ${fileName} (${format}));Code line: ${line}
// Can now perform async operations like database saves
await saveFileMetadata(fileName, format);
},
onCodeLine: async (line) => {
console.log();Comment/text: ${line}
// Can format code asynchronously
await formatAndSaveCodeLine(line);
},
onNonCodeLine: async (line) => {
console.log();File marked for deletion: ${fileName}
// Can process documentation asynchronously
await processDocumentationLine(line);
},
onFileDelete: async (fileName) => {
console.log();
// Handle file deletion when empty code blocks are detected
await deleteFile(fileName);
},
};
// Create a new parser instance with the callbacks
const parser = new StreamingMarkdownParser(callbacks);
// Process chunks as they arrive (e.g., from an LLM streaming response)
await parser.processChunk("index.html\n");
await parser.processChunk("`html\n");
await parser.processChunk("Hello World
\n");
await parser.processChunk("