This is an NPM module that uses the FME gSheets object interface to have a ghseet setup to rip through a list of RSS sites and produce a databse of article for processing
npm install fme-rss
mongo.MongoClient.connect(mongoURL, async (err, db) => {
if (err) {
L.error("MongoDB not running? Requires it");
throw err;
}
L.info("database connected:",mongoURL);
var rss = new FmeRss(db,gsheetConfig);
L.info("Have my rss object, step 1 is to set a stream function");
rss.stream = stream;
L.info("now init the Rss feeds");
await rss.init();
L.info("the number of feeds processes is",await rss.feedsDb.count())
L.info("the number of posts in the posts db is", await rss.fmePostsDb.count());
L.info("our post history count is",rss.postsHistory.length);
L.info("set the update rates to faster values for testing and debuggin");
rss.setFeedsDbPollInterval(1);
rss.setRssPollInterval(1);
L.info("set logging level to debug");
rss.setLogLevel("debug");
})
var stream = (posts:FmePost[]) => {
for (var i in posts) {
L.info("New post",posts[i].title,posts[i].pubdate);
}
}
`
Format of gsheetConfig for the FeedsDb to read and find the feeds;
if you don't know how to get this, read the google-edit-sheets NPM
{
"oauthEmail" : "BIG LONG KEY@developer.gserviceaccount.com",
"oauthKeyFile" : "Path to GDOCS Pem file",
"sheetId" : "unique key if for page",
"debugLevel" : "info"
}
Motivation
To aggreate RSS feeds based on topics for various websites;
Installation
`
npm install --save fme-rss
`
API Reference
`
See Code example above.
``