Felig Toolkit

A toolset for Amharic Language pre-processing 🔧

Felig Toolkit Web​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

Now with Typescript support!

---

What is felig-toolkit?

It is a toolset for Amharic Language pre-processing. It includes an Amharic Stemmer, Amharic Transliterator, Amharic Stopword remover, Amharic Lexical analyzer, Amharic Corpus indexer and Term weighter.

$3

Breaks down Amharic language corpus and returns tokens by removing any whitespace, expanding abbreviations(አ.አ -> አዲስ አበበ), removing numbers, breaking up hyphenated words, and removing punctuation (፡ ። ! ? ...).

$3

Removes commonly occuring words that have no contribution to the semantics of the corpus. Eg: እና ፡ ስለዚህ ፡ በመሆኑም...

$3

Changes Unicode Amharic characters to ASCII. Exmaple: ልጆች -> ልጅኦች -> ljoc. This tool implements two types of Amharic transliteration lookup tables.

- SERA (System for Ethiopic Representation in ASCII) - This system maps alphabets with similar sounds separately. Eg: (ሀ፣ሐ፡ኀ)፣(ሰ፡ሠ)፡(ጸ፡ፀ)፡(ዐ፡አ). However, in practice, these alphabets are used interchangeably and use of SERA would greatly decrease recall. NOT RECOMMENDED!

- Felig - Normalizes the redundant symbols into a common symbol. RECOMMENDED!

$3

Reduces the different morphological (e.g. inflectional or
derivational) variations of Amharic word forms by taking an Amharic word and returning the stem through affix-removal with longest match.

Exmaple:
ልጆች -> ልጅኦች -> ljoc -> lj -> ልጅ

$3

Produces an index file for the stemmed words in a corpus and relates them with the files they are found in. It also stores their frequencies per file.

$3

Calculates the weight of words from the index file using product of their length normalized Term frequency and Inverse document frequency (tf*idf).

Installation

Felig Toolkit is available as a package on NPM for use in a Node application:

``bash


NPM

`

`bash


YARN

`

`bash


PNPM

`

$3

note: this package uses es-modules

`javaScript
import felig_toolkit from 'felig-toolkit'

`

What's Included

- felig_transliterate(word,lang): takes a single word and its' language (am/en) and returns felig-transliterated string

- sera_transliterate(word,lang): takes a single word and its' language (am/en) and returns SERA-transliterated string.

- rmvStopwrd(corpus): takes an Amharic corpus text (sentence/paragraph/multiple-paragraphs) and removes stop wprds

- lexAnalyze(corpus): takes an Amharic corpus text returns a string of tokens

- stem(word): takes an Amharic word string and returns the stem as a string (async)

- indexer(filesArray, outputIndexFilePath, type): takes an array of files and produces an index (.json) file. (type= "doc" | "query")

- weigh_terms(indexFilePath, outputWeightedTermsPath, typeOfIndex): takes an index file and produces a file (.json) with weighted terms. (typeOfIndex= "doc" | "query")

How to use in Web apps

> Felig toolkit does not work in the browser (requires node.js enviroment).

> Use felig-toolkit on your server.

Exmaple:
If you are using Next.js, you can use felig-toolkit in a Next server route handler (/api/felig/route.ts) and pass the results.

Contributions

felig-toolkit is open to contributions, but it is recommend to create an issue or reply in a comment to let others know what you are working on first.

How to run locally

$3

- nodejs

1. Clone the repository
1. Run npm install
1. Run node index.js` on the root directory

Attribution

To prepare the following tools, these academic papers were used

- Girma Neshir Alemneh. “Amharic Light Stemmer”. ResearchGate. Sep 2020.
- Genet Mezemir Fikremariam. ”Automatic Stemming for Amharic text: An experiment using successor variety approach”. AAU. Jan 2009.
- Tessema Mindaye Mengistu. “Design and Implementation of Amharic Search Engine”. ResearchGate. August 2007.
- Yitna Firdyiwek and Daniel Yaqob. “The System for Ethiopic Representation in ASCII”. ResearchGate. Jan 1997.