ASIMOV Bright Data Module


[ASIMOV] module for data import powered by the [Bright Data] web data platform.

✨ Features

- Imports structured data from Airbnb, Amazon, Crunchbase, eBay, Facebook,
Google, Indeed, Instagram, LinkedIn, Walmart, X (aka Twitter), Yahoo, and
YouTube.
- Collects the raw JSON data via the Bright Data API (requires an API key).
- Constructs a semantic knowledge graph based on the [KNOW] ontology.
- Supports plain JSON output as well as [RDF] output in the form of JSON-LD.
- Distributed as a standalone static binary with zero runtime dependencies.

🛠️ Prerequisites

- [Rust] 1.85+ (2024 edition) if building from source code

⬇️ Installation

$3

``bash
pip install -U asimov-brightdata-module
`

$3

`bash
gem install asimov-brightdata-module
`

$3

`bash
npm install -g asimov-brightdata-module
`

$3

`bash
cargo install asimov-brightdata-module
`

👉 Examples

`bash
export BRIGHTDATA_API_KEY="..."
`

$3

`bash
asimov-brightdata-fetcher https://x.com/bright_init # JSON
asimov-brightdata-importer https://x.com/bright_init # JSON-LD
`

$3

`bash
asimov-brightdata-fetcher https://www.linkedin.com/in/orlenchner/
asimov-brightdata-fetcher https://www.linkedin.com/company/bright-data/
`

$3

`bash
asimov-brightdata-fetcher https://www.crunchbase.com/organization/brightdata
`

$3

`bash
asimov-brightdata-fetcher https://www.amazon.com/Master-Algorithm-Ultimate-Learning-Machine/dp/0465094279
`

⚙ Configuration

$3

- BRIGHTDATA_API_KEY: (required) the [Bright Data API key] to use

📚 Reference

$3

- asimov-brightdata-cataloger: discovers entities via the Bright Data API
_(not implemented yet)_
- asimov-brightdata-fetcher: collects JSON data from the Bright Data API
- asimov-brightdata-importer: collects and transforms JSON into JSON-LD
_(not implemented yet)_

$3

Dataset | URL Prefix | JSON | RDF
:------ | :--------- | :--: | :--:
Airbnb | https://www.airbnb.com/rooms/ | ✅ | 🚧
Amazon | https://www.amazon.com/ | ✅ | 🚧
  | https://www.amazon.com/sp?seller= | ✅ | 🚧
Crunchbase | https://www.crunchbase.com/organization/ | ✅ | 🚧
eBay | https://www.ebay.com/itm/ | ✅ | 🚧
Facebook | https://www.facebook.com/events/ | ✅ | 🚧
  | https://www.facebook.com/groups/ | ✅ | 🚧
  | https://www.facebook.com/marketplace/item/ | ✅ | 🚧
  | https://www.facebook.com/share/p/ | ✅ | 🚧
Google | https://www.google.com/shopping/product/ | ✅ | 🚧
Indeed | https://www.indeed.com/cmp/ | ✅ | 🚧
Instagram | https://www.instagram.com/ | ✅ | 🚧
  | https://www.instagram.com/p/ | ✅ | 🚧
  | https://www.instagram.com/reel/ | ✅ | 🚧
LinkedIn | https://www.linkedin.com/company/ | ✅ | 🚧
  | https://www.linkedin.com/in/ | ✅ | 🚧
  | https://www.linkedin.com/jobs/ | ✅ | 🚧
  | https://www.linkedin.com/posts/ | ✅ | 🚧
  | https://www.linkedin.com/pulse/ | ✅ | 🚧
Walmart | https://www.walmart.com/global/seller/ | ✅ | 🚧
  | https://www.walmart.com/ip/ | ✅ | 🚧
X (Twitter) | https://x.com/ | ✅ | ✅
Yahoo | https://finance.yahoo.com/quote/ | ✅ | 🚧
YouTube | https://www.youtube.com/@ | ✅ | 🚧
  | https://www.youtube.com/watch?v= | ✅ | 🚧
| | |

👨‍💻 Development

`bash
git clone https://github.com/asimov-modules/asimov-brightdata-module.git
``

---


[ASIMOV]: https://github.com/asimov-platform
[Bright Data]: https://brightdata.com/products/web-scraper
[Bright Data API key]: https://docs.brightdata.com/general/account/api-token
[JSON-LD]: https://json-ld.org
[KNOW]: https://github.com/know-ontology
[NPM]: https:/npmjs.org
[Python]: https://python.org
[RDF]: https://github.com/rust-rdf
[Ruby]: https://ruby-lang.org
[Rust]: https://rust-lang.org