annotatedtext


A lightweight JavaScript library for converting markup documents into an
annotated text format consumable by LanguageTool as
AnnotatedText.

Usage

See API below for details.

Motivation

Provide an easier way of running LanguageTool on documents by separating the
text from the markup.

Installation

npm:

``sh
npm install annotatedtext
`

API

$3

Returns Annotated Text as described by LanguageTool's API:

`json
{
"annotation": [
{ "text": "A " },
{ "markup": "" },
{ "text": "test" },
{ "markup": "" }
]
}
`

Run the object through JSON.stringfy() to get a string suitable for passing to
LanguageTool's data parameter.

This is the main function you'll use in implementing for different parsers.

`js
import { compose } from "annotatedtext";
import { parse } from "remark-parse";
import { unified } from "unified";

const text = "This is a sentence.";
const processor = unified()
.use(remarkparse, options.remarkoptions)
.use(frontmatter, ["yaml", "toml"])
.parse(text);

const annotatedtext = compose(text, parse);
JSON.stringify(annotatedtext);
`

- text: The text from the markup document in its original form.
- node: The root AST node that conforms to INode.
- _options_: (optional) See defaults.

$3

_Deprecated_

Use compose instead.

Returns Annotated Text as described by LanguageTool's API:

`json
{
"annotation": [
{ "text": "A " },
{ "markup": "" },
{ "text": "test" },
{ "markup": "" }
]
}
`

Run the object through JSON.stringfy() to get a string suitable for passing to
LanguageTool's data parameter.

This is the main function you'll use in implementing for different parsers.

`js
import * as builder from ‘annotatedtext’;
const processor = unified().use(mark, { commonmark: true });
var annotatedtext = builder.build(text, processor.parse);
JSON.stringify(annotatedtext);
`

- text: The text from the markup document in its original form.
- parse: A function that parses a markup document and returns an abstract
syntax tree.
- _options_: (optional) See defaults.

$3

Returns an array of annotated text nodes>) used in
the final annotated text object.

- ast: An abstract syntax tree.
- _options_: (optional) See defaults.

$3

- text: The text from the markup document in its original form.
- annotatedtextnodes: An array of an array of
annotated text nodes>) such as produced by
collecttextnodes>).
- _options_: (optional) See defaults.

$3

annotatedtext comes with the following default functions used throughout.

`js
const defaults = {
children: function (node) {
return node.children;
},
annotatetextnode: function (node) {
if (node.type === "text") {
return {
text: node.value,
offset: {
start: node.position.start.offset,
end: node.position.end.offset,
},
};
} else {
return null;
}
},
interpretmarkup: function (text = "") {
return "";
},
};
`

Functions can be overriden by making a copy and assigning a new function. For
example, the tests use markdown and need to interpret new lines in the markup as
new lines. The interpretmarkup function is overriden as:

`js
var options = builder.defaults;
options.interpretmarkup = function (text) {
let count = (text.match(/\n/g) || []).length;
return "\n".repeat(count);
};
`

#### children(node)

Expected to return an array of child nodes.

#### annotatetextnode(node)

Expected to return a struture for a text ast node with at least the following:

- text is the natural language text from the node, devoid of all markup.
- offset contains offsets used to extract markup text from the original
document.
- start is the offset start of the text
- end is the offset end of the text

`json
{
"text": "A snippet of the natural language text from the document.",
"offset": {
"start": 1,
"end": 57
}
}
`

If the node is not a text node, it must return null;

#### interpretmarkup(node)

Used to make sure LanguageTool knows when markup represents some form of
whitespace. As mentioned above, the tests override this function to ensure new
lines captured as markup are also visible to LanguageTool.

`js
var options = builder.defaults;
options.interpretmarkup = function (text) {
let count = (text.match(/\n/g) || []).length;
return "\n".repeat(count);
};
`

Tests

Unit tests are also run via npm:

`sh
npm test
`

Implemented Parsers

The following packages wrap annotatedtext for specific parsers:

- annotatedtext-remark
for markdown using
remark-parse.
- annotatedtext-rehype`
for html using
rehype-parse.

License

MIT © David L. Day