string-punctuation-tokenizer

Small library that provides functions to tokenize a string into an array of words with or without punctuation

Setup

npm install string-punctuation-tokenizer

Usage

var stringTokenizer = require('string-punctuation-tokenizer');

or ES6

import {tokenize} from 'string-punctuation-tokenizer';

#### Tokenize with punctuation
``js
import {tokenize} from './src/tokenizers'; // use the import from above instead of this
let words = tokenize({text: 'Hello world, my name is Manny!', includePunctuation: true});
// words = ["Hello", "world", ",", "my", "name", "is", "Manny", "!"]
`
#### Tokenize without punctuation
`js
import {tokenize} from './src/tokenizers'; // use the import from above instead of this
let words = tokenize({text: 'Hello world, my name is Manny!'});
// words = ["Hello", "world", "my", "name", "is", "Manny"]
``

$3

See detailed documentation and live WYSIWYG playground here: https://string-punctuation-tokenizer.netlify.app/#/Tokenize