tokenizer of source code for jscpd
npm install @jscpd/tokenizer@jscpd/tokenizer> tokenizer is package from @jscpd used for convert programming code to list of tokens
``
npm install @jscpd/tokenizer --save
`
import {IOptions, ITokensMap} from '@jscpd/core';
import {Tokenizer} from '@jscpd/tokenizer';
const tokenizer = new Tokenizer();
const options: IOptions = {};
const maps: ITokensMap[] = tokenizer.generateMaps('source_id', 'let a = "11"', 'javascript', options);
``
MIT © Andrey Kucherenko