A typescript implementation of the Rapid Automated Keyword Extraction (RAKE) algorithm with Part-of-Speech post-processing
npm install rake-posThe goal of this library was to create a Typescript translation of the
javacsript implementation,
which in itself was a translation of the python implementation.
Extraction defaults to processing the input text as if it is written in the English language. Stop
words are provided by default using stopwords-iso
In addition, Part-of-Speech post-processing is used to further filter keywords. This is done by
using an implementation of brill tagging. For usage details,
see the list of brill tag descriptions
``typescript
let keywords: string[] = [];
console.log(keywords);
> []
const input: string = 'I have some apples and bananas here for the table';
keywords = extractWithRakePos({ text: input });
console.log(keywords);
> ['table', 'bananas', 'apples']
const stop: Set
keywords = extractWithRakePos({ text: input, additionalStopWordSet: stop });
console.log(keywords);
> ['table', 'bananas']
``
---
Differences in regular expressions and stopword lists have big impacts on this algorithm and
sticking close to the python means that the code was easy to compare to ensure
that it was in the ballpark.
This algorithm is described in Text Mining: Applications and
Theory
and also in this excellent blog
post by Alyona
Medelyan.
It operates using only the text you give it and produces surprisingly good
results. There are likely better results
possible
but these mostly seem to involve a combination of Python, Machine Learning and
a corpus of data.
The appeal of RAKE is of the "bang for the buck" variety.
Currently this library produces subtly different results than either the paper
or the original Python implementation. While the results (especially the top
scoring ones) line up nicely, these little deviations represent something to
understand and resolve.
After hammering out differences in the results, plans are to focus on
* Fully embracing JS idioms (Promises/ES201X)
* Explore ways to improve the results as described
here
* Options to control result format (number, result|result+rank, etc)
* Include default stopword list.
* Improve handling of special characters and italics
* Deal with sentences that have been split over multiple lines (sentence now ends with -)
The stopword list used by the python version is here.
It has a comment as the first line which might break the world...
Links to other stopword lists can be found here
Any file with one word per line should be fine.