@albertocruzluis/lexer

Lexer
=========

A small library for parse code in tokens

Table of Contents

* Installation
* Usage
* Docs
* License

Installation

$3

``bash
npm i -g @albertocruzluis/lexer
`

$3

`bash
npm i @albertocruzluis/lexer --save
`

Usage

Import of the lexer module
`js
import buildLexer from '@albertocruzluis/lexer'
`

we create the tokens that our parser will be able to recognise
`js
const SPACE = /(?\s+|\/\/.*)/;
const RESERVEDWORD = /(?\b(const|let)\b)/;
const ID = /(?\b([a-z_]\w*))\b/;
const STRING = /(?"([^\\"]|\\.")*")/;
const OP = /(?[+*\/=-])/;

const tokens = [
['SPACE', SPACE, true], ['RESERVEDWORD', RESERVEDWORD], ['ID', ID],
['STRING', STRING], ['OP', OP]
];

const lexer = buildLexer(tokens);
`

Example generate tokens of a code

`js
const str = 'const varName = "value"';
const result = lexer(str);
console.log(result);
/*
[
{ type: 'RESERVEDWORD', value: 'const' },
{ type: 'ID', value: 'varName' },
{ type: 'OP', value: '=' },
{ type: 'STRING', value: '"value"' }
]
*/
``

Docs

See documentation for more details.

License

MIT