Amharic Language Pre-processor toolkit
npm install felig-toolkit

A toolset for Amharic Language pre-processing π§
Felig Toolkit Webββββββββββββββββββββββββββββββββββββ
Now with Typescript support!
---
It is a toolset for Amharic Language pre-processing. It includes an Amharic Stemmer, Amharic Transliterator, Amharic Stopword remover, Amharic Lexical analyzer, Amharic Corpus indexer and Term weighter.
Breaks down Amharic language corpus and returns tokens by removing any whitespace, expanding abbreviations(α .α -> α α²α΅ α α α ), removing numbers, breaking up hyphenated words, and removing punctuation (α‘ α’ ! ? ...).
Removes commonly occuring words that have no contribution to the semantics of the corpus. Eg: α₯α α‘ α΅ααα
α‘ α αααα...
Changes Unicode Amharic characters to ASCII. Exmaple: ααα½ -> αα
α¦α½ -> ljoc. This tool implements two types of Amharic transliteration lookup tables.
- SERA (System for Ethiopic Representation in ASCII) - This system maps alphabets with similar sounds separately. Eg: (αα£αα‘α)α£(α°α‘α )α‘(αΈα‘α)α‘(αα‘α ). However, in practice, these alphabets are used interchangeably and use of SERA would greatly decrease recall. NOT RECOMMENDED!
- Felig - Normalizes the redundant symbols into a common symbol. RECOMMENDED!
Reduces the different morphological (e.g. inflectional or
derivational) variations of Amharic word forms by taking an Amharic word and returning the stem through affix-removal with longest match.
Exmaple:ααα½ -> αα
α¦α½ -> ljoc -> lj -> αα
Produces an index file for the stemmed words in a corpus and relates them with the files they are found in. It also stores their frequencies per file.
Calculates the weight of words from the index file using product of their length normalized Term frequency and Inverse document frequency (tf*idf).
Felig Toolkit is available as a package on NPM for use in a Node application:
``bashNPM
npm install felig-toolkit
`
`bashYARN
yarn add felig-toolkit
`
`bashPNPM
pnpm install felig-toolkit
`
note: this package uses es-modules
`javaScript
import felig_toolkit from 'felig-toolkit'
`
- felig_transliterate(word,lang): takes a single word and its' language (am/en) and returns felig-transliterated string
- sera_transliterate(word,lang): takes a single word and its' language (am/en) and returns SERA-transliterated string.
- rmvStopwrd(corpus): takes an Amharic corpus text (sentence/paragraph/multiple-paragraphs) and removes stop wprds
- lexAnalyze(corpus): takes an Amharic corpus text returns a string of tokens
- stem(word): takes an Amharic word string and returns the stem as a string (async)
- indexer(filesArray, outputIndexFilePath, type): takes an array of files and produces an index (.json) file. (type= "doc" | "query")
- weigh_terms(indexFilePath, outputWeightedTermsPath, typeOfIndex): takes an index file and produces a file (.json) with weighted terms. (typeOfIndex= "doc" | "query")
> Felig toolkit does not work in the browser (requires node.js enviroment).
> Use felig-toolkit on your server.
Exmaple:
If you are using Next.js, you can use felig-toolkit in a Next server route handler (/api/felig/route.ts) and pass the results.
felig-toolkit is open to contributions, but it is recommend to create an issue or reply in a comment to let others know what you are working on first.
- nodejs
1. Clone the repository
1. Run npm installnode index.js` on the root directory
1. Run
To prepare the following tools, these academic papers were used
- Girma Neshir Alemneh. βAmharic Light Stemmerβ. ResearchGate. Sep 2020.
- Genet Mezemir Fikremariam. βAutomatic Stemming for Amharic text: An experiment using successor variety approachβ. AAU. Jan 2009.
- Tessema Mindaye Mengistu. βDesign and Implementation of Amharic Search Engineβ. ResearchGate. August 2007.
- Yitna Firdyiwek and Daniel Yaqob. βThe System for Ethiopic Representation in ASCIIβ. ResearchGate. Jan 1997.