marked-token-position

Add position field for each token.

``ts
interface Position {
/**
* Positions for each line of the token. LinePositions will not include the newline character for the line.
*/
lines: LinePosition[]
/**
* Position at the beginning of token
*/
start: PositionFields;
/**
* Position at the end of token
*/
end: PositionFields;
}

interface LinePosition {
/**
* Position at the beginning of line
*/
start: PositionFields;
/**
* Position at the end of line. Will not include the newline character.
*/
end: PositionFields;
}

interface PositionFields {
/**
* Number of characters from the beginning of the markdown string
*/
offset: number;
/**
* Line number of the token. Starts at line 0.
*/
line: number;
/**
* Column number of the token. Starts at column 0.
*/
column: number;
}
`

Usage

Extension

`js
import {Marked} from "marked";
import markedTokenPosition from "marked-token-position";

// or UMD script
//
//
// const Marked = marked.Marked;

const marked = new Marked();

function anotherExtension {
return {
walkTokens(token) {
// token has position field
}
hooks: {
processAllTokens(tokens) {
// tokens have position field
}
}
};
}

marked.use(anotherExtension(), markedTokenPosition());

marked.parse("# example markdown");
`

The position field will be added to the tokens so any other extension can
use the position field in a walkTokens function or processAllTokens hook.

> [!CAUTION]
> The processAllTokens hook is used by this extension so any other extension
> using processAllTokens that requires the position field must be added
> before this extension because marked calls the processAllTokens hooks in
> reverse order.

The tokens will look like:

`json
[
{
"type": "heading",
"raw": "# example markdown",
"depth": 1,
"text": "example markdown",
"tokens": [
{
"type": "text",
"raw": "example markdown",
"text": "example markdown",
"escaped": false,
"position": {
"start": {
"offset": 2,
"line": 0,
"column": 2
},
"end": {
"offset": 18,
"line": 0,
"column": 18
}
}
}
],
"position": {
"start": {
"offset": 0,
"line": 0,
"column": 0
},
"end": {
"offset": 18,
"line": 0,
"column": 18
}
}
}
]
`

addTokenPositions

Calling marked.lexer() will not add the position field with the extension
since the extension is only called on marked.parse() and marked.parseInline().

An addTokenPositions function is exported to add the position field to the
tokens returned by marked.lexer().

`js
import {Marked} from "marked";
import {addTokenPositions} from "marked-token-position";

// or UMD script
//
//
// const Marked = marked.Marked;
// const addTokenPositions = markedTokenPosition.addTokenPositions;


const marked = new Marked();
const tokens = marked.lexer("# example markdown");

addTokenPositions(tokens);

// tokens now have a position field
``